Subway vehicle body defect detection method and system based on multi-source image data fusion
Through multi-source image data fusion and convolutional neural network, a subway body defect detection model is built, which solves the problems of low detection accuracy and difficulty in local position detection in the existing technology, realizes an efficient and automated maintenance process, and improves the detection quality and security.
Patent Information
- Application Number
- CN202410927501.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-11
- Publication Date
- 2024-11-08
AI Technical Summary
The existing 360° appearance image fault detection system cannot effectively detect the side, roof and bottom of the vehicle under the limitations of train speed and detection scenarios, resulting in low detection accuracy and the inability to conduct secondary detection of local positions.
Using a method based on multi-source image data fusion, the subway appearance image data and detection data are collected, preprocessed and feature extraction are carried out, and the subway body defect detection model is constructed in combination with the convolutional neural network algorithm to realize quick inspection and full inspection detection, and use The inspection robot conducts fixed-point inspection and re-inspection.
It improves the accuracy and efficiency of inspection, meets the maintenance needs in different scenarios, realizes the automation and flexibility of the inspection process, and ensures the stability and safe operation of the subway system.
Smart Images

Figure CN118918423A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of rail transit technology, and in particular to a subway body defect detection method and system based on multi-source image data fusion. Background Art
[0002] With the rapid development of my country's rail transit industry, the current subway train fault detection and prevention are mostly done by manual inspection, which is in a mode of regular maintenance and fault maintenance. The complex structure and many parts of the trains result in low efficiency of manual inspection, and the inspection results rely on the subjective feelings and personal experience of the inspectors, which cannot guarantee the quality of maintenance.
[0003] With the increasing density of lines and the number of trains, the traditional subway operation mode can no longer meet the urgent needs of the subway industry. The application of machine vision technology to train fault detection has become a new research direction. The subway train 360° appearance image fault detection system is aimed at the important items of manual daily inspection / four-day inspection safety and maintenance of subway trains. It collects high-definition images of key components including current collection, running gear, traction, brake air supply, couplers, etc., to form a variety of vehicle appearance image perception data. Through pattern recognition, computer vision and data-driven hierarchical warning technologies, machine vision, pattern recognition and other technologies are applied to subway train fault detection to realize fault diagnosis and analysis of key components of trains, realize the transformation from "human inspection" to "machine inspection", reduce operation and maintenance costs, and lay the foundation for comprehensive automated inspection of trains.
[0004] Most existing 360° exterior image fault detection systems are limited by train speed, and have limited effectiveness in detecting the sides, roofs and bottoms of vehicles. At the same time, due to detection scene limitations, 360° systems cannot perform secondary detection of specific local locations, making it difficult to improve detection accuracy.
[0005] Currently, no effective solution has been proposed for the problems in the related technologies. Summary of the invention
[0006] In view of the problems in the related art, the present invention proposes a subway body defect detection method and system based on multi-source image data fusion to overcome the above-mentioned technical problems existing in the existing related art.
[0007] To this end, the specific technical solution adopted by the present invention is as follows: According to one aspect of the present invention, a method for detecting subway body defects based on multi-source image data fusion is provided, the method comprising the following steps: S1, collecting subway appearance image data, and inputting the subway appearance image data into a pre-built appearance image fault monitoring model to obtain subway quick inspection results; S2, fusing the collected subway detection data with the subway appearance image data, and extracting fusion features from the fused data; S3. Construct a subway body defect detection model based on the fusion features, and input the fusion data into the subway body defect detection model to obtain the subway full inspection result; S4. Based on the results of the subway quick inspection and the subway full inspection, conduct fixed-point inspections on the locations where the subway needs maintenance, and re-inspect the local locations after maintenance.
[0008] Furthermore, collecting subway appearance image data and inputting the subway appearance image data into a pre-built appearance image fault monitoring model to obtain subway quick inspection results includes the following steps: S11, using a gantry to collect images of the subway appearance to obtain subway appearance image data; S12, preprocessing the acquired subway appearance image data; S13. Using the preprocessed subway appearance image data as input and combining it with a pre-built appearance image fault monitoring model, obtain subway quick inspection results, wherein the appearance image fault monitoring model is based on the preprocessed subway appearance image data and is trained using a neural network model.
[0009] Furthermore, preprocessing the acquired subway appearance image data includes the following steps: S121, converting the color image in the acquired subway appearance image data into a grayscale image to obtain a grayscale image; S122: Pair the obtained grayscale images, and adjust the pixels of the subway appearance image data based on the pairing results.
[0010] Furthermore, the pixels of the subway appearance image data are adjusted to 256*256.
[0011] Further, fusing the collected subway detection data with the subway appearance image data and extracting fusion features from the fused data includes the following steps: S21. Use the inspection robot to inspect the subway, obtain subway inspection data, and pre-process the subway inspection data; S22, fusing the preprocessed subway detection data with the subway appearance image data to obtain a fused data set; S23. Use image processing technology to process the fused data set, and combine the convolutional neural network algorithm to extract fusion features from the processed fused data set.
[0012] Furthermore, the subway inspection data includes subway vehicle side inspection data and subway vehicle bottom inspection data.
[0013] Furthermore, a subway body defect detection model is constructed based on the fusion features, and the fusion data is input into the subway body defect detection model to obtain the subway full inspection result, which includes the following steps: S31. Combining the extracted fusion features with the neural network algorithm, a subway body defect detection model is constructed; S32, input the fused data into the constructed subway body defect detection model to obtain the subway full inspection result.
[0014] Furthermore, combining the extracted fusion features with the neural network algorithm, constructing a subway body defect detection model includes the following steps: S321, based on the extracted fusion features, selecting a basic network as a neural network main body, the basic network including several downsampling and several upsampling; S322, combining the features in the downsampling with the features in the upsampling by using a splicing technique, and embedding the sinusoidal position encoding information corresponding to the diffusion step length into the neural network before sampling at each layer; S323, introducing a linear attention mechanism into the middle layer of the neural network after the sinusoidal position encoding information is embedded, and fusing the output of the linear attention mechanism with the features of the current layer; S324. Construct a subway body defect detection model based on the fusion features, sinusoidal position encoding information and features processed by the linear attention mechanism; S325. Use the processed fusion data set to train and evaluate the subway body defect detection model to obtain the final subway body defect detection model.
[0015] Furthermore, the processed fusion data set is used to train and evaluate the subway body defect detection model, and obtaining the final subway body defect detection model includes the following steps: S3251, dividing the processed fusion data into a training set and a test set; S3252. Based on the divided training set, a subway body defect detection model is trained in combination with an N-fold cross validation algorithm; S3253. Use the test set to verify the trained subway body defect detection model, evaluate the subway body defect detection model, and obtain the final subway body defect detection model.
[0016] According to another aspect of the present invention, a subway body defect detection system based on multi-source image data fusion is also provided, the system comprising a subway quick inspection module, a fusion feature extraction module, a subway full inspection module and a subway re-inspection module; The subway quick inspection module, the fusion feature extraction module, the subway full inspection module and the subway re-inspection module are connected in sequence; The subway quick inspection module is used to collect subway appearance image data and input the subway appearance image data into a pre-built appearance image fault monitoring model to obtain subway quick inspection results; A fusion feature extraction module is used to fuse the collected subway detection data with the subway appearance image data and extract fusion features from the fused data; The subway full inspection module is used to build a subway body defect detection model based on fusion features, and input the fusion data into the subway body defect detection model to obtain the subway full inspection results; The subway re-inspection module is used to combine the subway quick inspection results with the subway full inspection results, conduct fixed-point inspections on the locations that require subway maintenance, and re-inspect local locations after maintenance.
[0017] The beneficial effects of the present invention are: 1. The present invention can identify potential problems and hidden dangers by integrating subway detection data with subway appearance image data, effectively prevent the occurrence of failures, provide solid support for the safe operation of the subway, ensure the stability and reliability of the subway system, and thus create a more comfortable and safe travel environment for passengers.
[0018] 2. The present invention can cover a wider range of maintenance scenarios and meet maintenance needs in different scenarios by setting two modes: quick inspection and full inspection. It not only provides more diverse choices, but also significantly improves the efficiency of maintenance work in actual operation.
[0019] 3. The present invention realizes fixed-point detection of local maintenance positions by utilizing inspection robots, which not only realizes the automation of the detection process, but also significantly enhances the flexibility and adaptability of the detection, making maintenance detection more efficient, thereby improving the quality and accuracy of the overall detection work. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0021] Figure 1 is a flow chart of a subway body defect detection method based on multi-source image data fusion according to an embodiment of the present invention; Figure 2 It is a schematic diagram of a detection robot inspecting the side and bottom of a subway car in a subway car body defect detection method based on multi-source image data fusion according to an embodiment of the present invention; Figure 3It is a principle block diagram of a subway body defect detection system based on multi-source image data fusion according to an embodiment of the present invention.
[0022] In the figure: 1. Subway quick inspection module; 2. Fusion feature extraction module; 3. Subway full inspection module; 4. Subway re-inspection module. DETAILED DESCRIPTION
[0023] To further illustrate each embodiment, the present invention provides drawings, which are part of the disclosure of the present invention and are mainly used to illustrate the embodiments. They can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, ordinary technicians in this field should be able to understand other possible implementation methods and advantages of the present invention.
[0024] According to an embodiment of the present invention, a method and system for detecting subway body defects based on multi-source image data fusion are provided.
[0025] The present invention is further described with reference to the accompanying drawings and specific embodiments. Figure 1-Figure 2 As shown, according to the subway body defect detection method based on multi-source image data fusion according to an embodiment of the present invention, the method includes the following steps: S1. Collect subway appearance image data, and input the subway appearance image data into a pre-built appearance image fault monitoring model to obtain subway quick inspection results.
[0026] Specifically, collecting subway appearance image data and inputting the subway appearance image data into a pre-built appearance image fault monitoring model to obtain subway quick inspection results includes the following steps: S11. Collect images of the subway exterior using a gantry to obtain subway exterior image data.
[0027] It should be noted that after the subway returns to the base, the gantry is used to collect images of the 360° exterior image fault detection system. The roof, running gear, and bottom of the vehicle are captured by 4K high-definition cameras, and the windows are captured by 2K high-definition cameras. The image resolution can reach 1 mm / pixel, and multi-level magnification is supported for small parts, so that bolt marks can be clearly viewed.
[0028] S12, pre-processing the acquired subway appearance image data.
[0029] Specifically, preprocessing the acquired subway appearance image data includes the following steps: S121, converting the color image in the acquired subway appearance image data into a grayscale image to obtain a grayscale image; S122: Pair the obtained grayscale images, and adjust the pixels of the subway appearance image data based on the pairing results.
[0030] Specifically, the pixels of the subway appearance image data after adjustment are 256*256.
[0031] It should be noted that 720 pairs of paired image data with a pixel size of 256*256 are obtained.
[0032] S13. Using the preprocessed subway appearance image data as input and combining it with a pre-built appearance image fault monitoring model, obtain subway quick inspection results, wherein the appearance image fault monitoring model is based on the preprocessed subway appearance image data and is trained using a neural network model.
[0033] It should be noted that the pre-built appearance image fault monitoring model is mainly trained based on the subway appearance image data obtained by the gantry image acquisition device; the pre-processed grayscale image pixels are used as input data, and a neural network model with an architecture including but not limited to CNN, RNN or EfficientNet is used for training to preliminarily identify the faulty parts in the subway appearance image.
[0034] S2. Fuse the collected subway detection data with the subway appearance image data, and extract fusion features from the fused data.
[0035] Specifically, fusing the collected subway detection data with the subway appearance image data and extracting fusion features from the fused data includes the following steps: S21. Use the inspection robot to inspect the subway, obtain subway inspection data, and pre-process the subway inspection data.
[0036] Specifically, the subway inspection data includes subway vehicle side inspection data and subway vehicle bottom inspection data.
[0037] S22, fusing the preprocessed subway detection data with the subway appearance image data to obtain a fused data set.
[0038] S23. Use image processing technology to process the fused data set, and combine the convolutional neural network algorithm to extract fusion features from the processed fused data set.
[0039] It should be noted that image processing technology includes image enhancement, image reconstruction, and image reconstruction for blurred image scenes.
[0040] S3. Build a subway body defect detection model based on the fusion features, and input the fusion data into the subway body defect detection model to obtain the subway full inspection results.
[0041] Specifically, building a subway body defect detection model based on fusion features, and inputting fusion data into the subway body defect detection model to obtain the subway full inspection results includes the following steps: S31. Combining the extracted fusion features with the neural network algorithm, a subway body defect detection model is constructed.
[0042] Specifically, combining the extracted fusion features with the neural network algorithm, building a subway body defect detection model includes the following steps: S311. Based on the extracted fusion features, a basic network is selected as the main body of the neural network, and the basic network includes several downsamplings and several upsamplings.
[0043] It should be noted that the basic network is the Unet basic network structure, which includes three downsampling and three upsampling to obtain the characteristics of the data at different scales.
[0044] S312. Use the splicing technology to combine the features in the downsampling with the features in the upsampling, and before sampling each layer, embed the sinusoidal position encoding information corresponding to the diffusion step into the neural network.
[0045] S313. A linear attention mechanism is introduced into the middle layer of the neural network after the sinusoidal position coding information is embedded, and the output of the linear attention mechanism is integrated with the features of the current layer.
[0046] S314. Based on the fusion features, sinusoidal position encoding information and features processed by the linear attention mechanism, a subway body defect detection model is constructed.
[0047] S315. Use the processed fusion data set to train and evaluate the subway body defect detection model to obtain the final subway body defect detection model.
[0048] Specifically, the subway body defect detection model is trained and evaluated using the processed fusion data set to obtain the final subway body defect detection model, including the following steps: S3151. Divide the processed fusion data into a training set and a test set.
[0049] It should be noted that the processed fusion data is divided into training set and test set in a ratio of 9:1.
[0050] S3152. Based on the divided training set, the subway body defect detection model is trained in combination with the N-fold cross-validation algorithm.
[0051] S3153. Use the test set to verify the trained subway body defect detection model, evaluate the subway body defect detection model, and obtain the final subway body defect detection model.
[0052] S32, input the fused data into the constructed subway body defect detection model to obtain the subway full inspection result.
[0053] S4. Based on the results of the subway quick inspection and the subway full inspection, conduct fixed-point inspections on the locations where the subway needs maintenance, and re-inspect the local locations after maintenance.
[0054] It should be noted that after completing the subway quick inspection or the subway full inspection, maintenance personnel can carry out repairs on local locations based on the subway quick inspection results and the subway full inspection results; after the maintenance is completed, robots can be used to re-inspect the local locations after the maintenance.
[0055] In addition, the collected subway detection data and subway appearance image data are used for fusion analysis to propose an overall detection scheme for trackside vision; wherein, the overall detection scheme for trackside vision includes: sensor module, including a variety of sensors including visual sensors, used to detect the status of the train chassis; positioning module, used to determine whether the train stop position and the position of the train chassis intelligent detection device meet the predefined conditions; displacement module, used to drive the sensor module to move to the specified position; communication module, used to communicate with the central control module and the ground control device; central control module, which is connected to the sensor module, positioning module, displacement module and communication module, and is used to control the operation of the above modules. After analyzing the two types of image data, through strategy optimization, the detection efficiency is improved and the integrated analysis and utilization of data is achieved.
[0056] Subway maintenance is divided into daily inspection, biweekly inspection, three-month inspection, scheduled maintenance, frame maintenance and major maintenance; among them, the quick inspection mode is mainly for daily inspection scenarios, and the full inspection mode is mainly for scenarios with longer maintenance cycles such as biweekly and three-month inspections.
[0057] like Figure 3 As shown, according to another embodiment of the present invention, a subway body defect detection system based on multi-source image data fusion is also provided, the system comprising a subway quick inspection module 1, a fusion feature extraction module 2, a subway full inspection module 3 and a subway re-inspection module 4; The subway quick inspection module 1, the fusion feature extraction module 2, the subway full inspection module 3 and the subway re-inspection module 4 are connected in sequence; The subway quick inspection module 1 is used to collect subway appearance image data and input the subway appearance image data into a pre-built appearance image fault monitoring model to obtain subway quick inspection results; The fusion feature extraction module 2 is used to fuse the collected subway detection data with the subway appearance image data, and extract fusion features from the fused data; The subway full inspection module 3 is used to construct a subway body defect detection model based on the fusion features, and input the fusion data into the subway body defect detection model to obtain the subway full inspection results; The subway re-inspection module 4 is used to combine the subway quick inspection results with the subway full inspection results, conduct fixed-point inspections on the locations that require maintenance on the subway, and re-inspect the local locations after maintenance.
[0058] In summary, with the help of the above technical solution of the present invention, through the fusion of subway inspection data and subway appearance image data, potential problems and hidden dangers can be identified, and the occurrence of failures can be effectively prevented, which provides solid support for the safe operation of the subway, ensures the stability and reliability of the subway system, and creates a more comfortable and safe travel environment for passengers. By setting the two modes of quick inspection and full inspection, a wider range of maintenance scenarios can be covered to meet the maintenance needs in different scenarios, which not only provides more diverse choices, but also significantly improves the efficiency of maintenance work in actual operation. By using inspection robots to realize fixed-point detection of local maintenance locations, not only the automation of the detection process is realized, but also the flexibility and adaptability of the detection are significantly enhanced, making maintenance detection more efficient, thereby improving the quality and accuracy of the overall detection work.
[0059] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A subway body defect detection method and system based on multi-source image data fusion, characterized in that: The method comprises the following steps: S1, collecting subway appearance image data, and inputting the subway appearance image data into a pre-built appearance image fault monitoring model to obtain subway quick inspection results; S2, fusing the collected subway detection data with the subway appearance image data, and extracting fusion features from the fused data; S3. Construct a subway body defect detection model based on the fusion features, and input the fusion data into the subway body defect detection model to obtain the subway full inspection result; S4. Based on the results of the subway quick inspection and the subway full inspection, conduct fixed-point inspections on the locations where the subway needs maintenance, and re-inspect the local locations after maintenance.
2. The subway body defect detection method based on multi-source image data fusion according to claim 1 is characterized in that: The collecting of subway appearance image data and inputting the subway appearance image data into a pre-built appearance image fault monitoring model to obtain subway quick inspection results comprises the following steps: S11, using a gantry to collect images of the subway appearance to obtain subway appearance image data; S12, preprocessing the acquired subway appearance image data; S13. Using the preprocessed subway appearance image data as input and combining it with a pre-built appearance image fault monitoring model, obtain subway quick inspection results, wherein the appearance image fault monitoring model is based on the preprocessed subway appearance image data and is trained using a neural network model.
3. The subway body defect detection method based on multi-source image data fusion according to claim 2 is characterized in that: The preprocessing of the acquired subway appearance image data comprises the following steps: S121, converting the color image in the acquired subway appearance image data into a grayscale image to obtain a grayscale image; S122: Pair the obtained grayscale images, and adjust the pixels of the subway appearance image data based on the pairing results.
4. The subway body defect detection method based on multi-source image data fusion according to claim 3 is characterized in that: The pixels of the adjusted subway appearance image data are 256*256.
5. The method for detecting subway body defects based on multi-source image data fusion according to claim 1 is characterized in that: The step of fusing the collected subway detection data with the subway appearance image data and extracting fusion features from the fused data includes the following steps: S21. Use the inspection robot to inspect the subway, obtain subway inspection data, and pre-process the subway inspection data; S22, fusing the preprocessed subway detection data with the subway appearance image data to obtain a fused data set; S23. Use image processing technology to process the fused data set, and combine the convolutional neural network algorithm to extract fusion features from the processed fused data set.
6. The method for detecting subway body defects based on multi-source image data fusion according to claim 5 is characterized in that: The subway detection data includes subway vehicle side detection data and subway vehicle bottom detection data.
7. The method for detecting subway body defects based on multi-source image data fusion according to claim 1 is characterized in that: The method of constructing a subway body defect detection model based on fusion features and inputting fusion data into the subway body defect detection model to obtain a subway full inspection result includes the following steps: S31. Combining the extracted fusion features with the neural network algorithm, a subway body defect detection model is constructed; S32, input the fused data into the constructed subway body defect detection model to obtain the subway full inspection result.
8. The method for detecting subway body defects based on multi-source image data fusion according to claim 7 is characterized in that: The method of combining the extracted fusion features with the neural network algorithm to construct a subway body defect detection model includes the following steps: S311, based on the extracted fusion features, selecting a basic network as a neural network main body, wherein the basic network includes several downsamplings and several upsamplings; S312, combining the features in the downsampling with the features in the upsampling by using a splicing technique, and embedding the sinusoidal position encoding information corresponding to the diffusion step length into the neural network before sampling at each layer; S313, introducing a linear attention mechanism into the middle layer of the neural network after the sinusoidal position encoding information is embedded, and fusing the output of the linear attention mechanism with the features of the current layer; S314. Construct a subway body defect detection model based on the fusion features, sinusoidal position encoding information and features processed by the linear attention mechanism; S315. Use the processed fusion data set to train and evaluate the subway body defect detection model to obtain the final subway body defect detection model.
9. The method for detecting subway body defects based on multi-source image data fusion according to claim 8 is characterized in that: The method of training and evaluating the subway body defect detection model using the processed fusion data set to obtain the final subway body defect detection model includes the following steps: S3151, dividing the processed fusion data into a training set and a test set; S3152. Based on the divided training set, the subway body defect detection model is trained in combination with the N-fold cross validation algorithm; S3153. Use the test set to verify the trained subway body defect detection model, evaluate the subway body defect detection model, and obtain the final subway body defect detection model.
10. A subway body defect detection system based on multi-source image data fusion, used to implement the subway body defect detection method based on multi-source image data fusion as described in any one of claims 1 to 9, characterized in that: The system includes a subway quick inspection module, a fusion feature extraction module, a subway full inspection module and a subway re-inspection module; The subway quick inspection module, the fusion feature extraction module, the subway full inspection module and the subway re-inspection module are connected in sequence; The subway quick inspection module is used to collect subway appearance image data and input the subway appearance image data into a pre-built appearance image fault monitoring model to obtain subway quick inspection results; The fusion feature extraction module is used to fuse the collected subway detection data with the subway appearance image data, and extract fusion features from the fused data; The subway full inspection module is used to construct a subway body defect detection model based on the fusion features, and input the fusion data into the subway body defect detection model to obtain the subway full inspection result; The subway re-inspection module is used to combine the subway quick inspection results with the subway full inspection results, conduct fixed-point inspections on the locations where the subway needs to be repaired, and re-inspect the local locations after maintenance.
