A train inspection method, system and storage medium based on multimodal signals

Abstract

The present invention relates to the field of train anomaly detection, and specifically to a train inspection method, system and storage medium based on multimodal signals, including data collection and processing steps, anomaly classification, data analysis and verification, anomaly analysis and fault output. The method performs anomaly detection on trains by using multimodal data fusion such as images, sounds, temperature and lasers, thereby reducing the limitations of single-signal train anomaly detection. By covering multiple key parts and features of trains with multimodal signals, a comprehensive assessment of the overall state of the train is achieved, thereby improving the comprehensiveness and accuracy of train anomaly inspection.

Description

Technical Field

[0001] The present invention relates to the field of train anomaly detection, and in particular to a train inspection method, system and storage medium based on multimodal signals. Background Art

[0002] During train operation and maintenance, anomaly detection is an important part of ensuring safe train operation. However, most existing train anomaly detection methods rely on a single type of signal, such as image, temperature, sound wave and light signal for detection, which has obvious limitations. Single signal detection methods are easily affected by factors such as environmental noise, signal blind spots or covering foreign materials, resulting in large errors and low precision in detection results, which cannot meet the high standards and requirements of train anomaly detection. In terms of foreign body detection, it is difficult to accurately determine whether there are tiny cracks or imperceptible foreign bodies on the surface of the train based on image information alone; in terms of oil level detection, a single visual inspection is easily affected by factors such as light and oil stains, resulting in inaccurate oil level judgment; and in motor detection, relying solely on temperature signals may make it difficult to fully evaluate the health of the motor due to the complex internal structure of the motor.

[0003] In order to solve the limitations of the above-mentioned prior art, the present application proposes a train inspection method, system and storage medium based on multimodal signals. By using multimodal signal fusion detection to detect train anomalies, the limitation of single signal in the prior art for anomaly detection is solved. By multimodal signal fusion and mutual verification, the singleness and limitations of single signal or two signal verification can be avoided, achieving the beneficial effect of improving the accuracy of foreign object detection, oil level detection and motor detection in train anomaly detection. Summary of the invention

[0004] In order to solve the above-mentioned problems in the prior art, the first purpose of the present invention is to provide a train inspection method based on multimodal signals, which solves the problems of large errors and low precision in the prior art based on a single type of signal detection, and achieves the effect of improving the accuracy of abnormality detection in train inspection.

[0005] In order to solve the above problems in the prior art, the second object of the present invention is to provide a train inspection system based on multimodal signals, which is used to provide a control system of a train inspection method based on multimodal signals;

[0006] In order to solve the above-mentioned problems in the prior art, the third object of the present invention is to provide a train inspection storage medium based on multimodal signals, which is used to store execution instructions and programs of a train inspection method based on multimodal signals.

[0007] In order to achieve the above object, the present invention provides a solution: a train inspection method based on multimodal signals, characterized in that it includes the following steps:

[0008] The data acquisition and processing step performs sensor pre-processing, fuses the signals of the multimodal sensors, and synchronizes the time and space information of the data collected by the multimodal sensors, obtains image data, sound wave data, light reflectivity data, and temperature data, and combines them to obtain a data set to be tested; the data set to be tested is used in the subsequent anomaly detection step, and the four types of data in the data set to be tested are grouped in the form of combination, and different groups of data sets to be tested are called in different types of detection.

[0009] In the abnormal classification step, the sensor data in the data set to be tested under the current position to be tested are sorted by reliability according to the data set to be tested corresponding to the position to be tested, and the verification data set is obtained. The interference factor in the sensor data is determined by the interference analysis strategy according to the position to be tested; by sorting the data combined in the data set to be tested by reliability, the data with the lowest reliability can be removed as redundant data and not used as reference data for abnormal detection, which reduces the complexity of data processing and improves the data processing efficiency in multimodal signal analysis. By judging the interference factors between the sensor data to be tested, the fusion correlation between the multimodal sensor data can be obtained, thereby increasing the accuracy of multimodal fusion detection.

[0010] The data fusion verification step verifies the sensor data by using a fusion verification strategy based on the sensor data in the verification data set combined with interference factors, and obtains an anomaly analysis data set based on the verification results; the sensor data set is fused and verified by using the fusion verification strategy, and train anomaly detection is performed through the correlation between multimodal signals, and the accuracy of anomaly detection is verified.

[0011] In the anomaly analysis step, the anomaly analysis strategy is used to determine whether there is an anomaly based on the anomaly analysis data set. If it is anomaly, the position to be inspected is marked as an anomaly position. By determining the anomaly position and comparing it with historical data, the anomaly detection type can be confirmed, and the accuracy of the reliability ranking of the position to be inspected can be verified.

[0012] The fault output step outputs the corresponding abnormal results according to the abnormal position, including: foreign body abnormality, body scratch defect, oil level abnormality and motor abnormality.

[0013] As a further improvement of the present invention, when the position to be inspected is consistent with the standard foreign body detection position in the database, the verification data set includes image data, sound wave data and light reflectivity data; the verification data set package is compared with the standard data set package in the database, and the comparison result outputs an abnormal instruction or a normal instruction. By comparing the position to be inspected with the corresponding type of detection position in the database, the detection data type required for the position to be inspected can be accurately selected, which increases the accuracy of reliability sorting of data in the data set. By comparing the verification data set of the position to be inspected with the standard data set, the difference between the data of the position to be inspected and the standard data can be obtained, thereby performing abnormal detection on the train, and the detection accuracy is increased through multimodal data fusion.

[0014] As a further improvement of the present invention, the comparison result of the image data and the comparison result of the acoustic wave data are verified for consistency, and the verification of the comparison result of the image data and the comparison result of the light reflectivity data, as well as the verification of the comparison result of the acoustic wave data and the comparison result of the light reflectivity data are output based on the verification result. Then, based on the two verification results, whether the light reflectivity data is abnormal is output. Since there is an interference factor between the image data and the light reflectivity data, that is, when an abnormality occurs in the train in the image data, the light reflectivity has two results, abnormal and normal. Therefore, when there is a difference between the comparison result of the image data and the comparison result of the acoustic wave data, the accuracy of the light reflectivity data cannot be verified, and the accuracy of the above two types of data cannot be judged. Therefore, by comparing the two types of data, the acoustic wave data and the image data, with the light reflectivity, the accuracy of the light reflectivity data can be verified.

[0015] As a further improvement of the present invention, when the light reflectivity data is abnormal, the abnormal ratio of the data at several different positions to be inspected is analyzed, and the ratio value is compared with the threshold value to determine whether the light reflectivity data needs to be replaced. Since the light reflectivity is the data type with the lowest reliability among the data types used in the foreign body detection process, the light reflectivity data is judged to be suitable for the detection by verifying the historical performance of the light reflectivity data, that is, the frequency of abnormalities at different positions to be inspected. When the light reflectivity is often inaccurate in this type of abnormal detection, that is, the light reflectivity exceeds the preset threshold in this type of abnormal detection, the reliability ranking of the light reflectivity data in this type of detection is reduced, and it is replaced with other types of sensor data to re-perform multimodal signal fusion detection.

[0016] As a further improvement of the present invention, the interference analysis strategy performs data association on the sensor data in the verification data set. When the sensor data therein affects the accuracy of other sensor data, it is obtained that there is an interference factor therebetween. By judging the factors that influence each other between the sensor data, the interference factors between the sensor data are obtained, which can reduce the situation where subsequent sensor data are abnormal and inaccurate, and verify that multimodal fusion data is more accurate in detecting the same anomaly through the interference factors.

[0017] As a further improvement of the present invention, the anomaly analysis strategy, when the verification results of all the sensor data in the anomaly analysis data set output abnormal instructions, the position to be inspected in the anomaly analysis step is abnormal, and the abnormal position is output. The same output of the same anomaly detection by multi-modal sensor signals is used as the standard for abnormal output, thereby increasing the reliability and accuracy of anomaly detection and avoiding the problem of inaccurate anomaly detection but unable to be verified due to interference from other factors in the anomaly detection process of a single modal signal.

[0018] As a further improvement of the present invention, when the output result of the light reflectance data is normal, the image data is verified for accuracy, feature points in the image data are extracted, feature descriptors are generated using the feature points, the similarity between the feature descriptors and the feature descriptors of standard image data in the database is calculated, and a comparison result is obtained based on the ratio of the similarity to the threshold value, and then the image data is output based on the comparison result to determine whether it is accurate. When there is a difference in the detection of the same anomaly detection by multimodal sensor data during the anomaly detection process, the image data ranked first in reliability is verified for accuracy again, and its reliability is re-ranked, thereby increasing the accuracy of the multimodal sensor signal in train anomaly detection, improving the reliability and detection quality of train anomaly detection, and reducing the risk of increased detection costs due to multiple retests due to inaccurate anomaly detection.

[0019] As a further improvement of the present invention, when the output image data of the accuracy verification of the image data is abnormal, the sensor for collecting the image data is calibrated.

[0020] As a further improvement of the present invention, the acoustic wave data is configured with a comparison acoustic wave function:

[0021] ;

[0022] ;

[0023] ;

[0024] Among them, I is the comparison sound wave function, I 0is the initial sound wave intensity emitted by the sound source, d is the distance between the sound source and the position to be measured, n is the constant for the sound wave to propagate in the environment, is the effect of environmental noise on the sound wave function as the distance changes, α is the environmental impact factor, is the influence function of environmental noise on the acoustic wave function as the moving speed of the train inspection robot changes, v is the moving speed of the train inspection robot, k is the speed adjustment factor, and v ref It is the standard speed value.

[0025] The present invention also proposes a train inspection system based on multimodal signals, and provides a control system for the above-mentioned train inspection method based on multimodal signals, comprising:

[0026] A data acquisition module, including a plurality of sensors, for acquiring the image data, sound wave data, light reflectivity data and temperature data;

[0027] The data processing module includes an abnormal classification submodule, a data fusion submodule and an abnormal analysis submodule, wherein the abnormal classification submodule is used to screen the data set to be inspected at the position to be inspected, output the verification data set, and obtain the interference factor;

[0028] The data fusion submodule is used to perform anomaly analysis on the verification data set and output an anomaly analysis data set;

[0029] The abnormality analysis submodule is used to perform abnormality analysis on the abnormality analysis data set and output whether the position to be inspected is abnormal;

[0030] The data output module is used to determine and output the abnormal results corresponding to the abnormal positions.

[0031] By using this system, the accuracy of the train inspection method based on multimodal signal fusion is guaranteed.

[0032] The present invention also proposes a train inspection storage medium based on multimodal signals, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps of the above-mentioned train inspection method based on multimodal signals are implemented.

[0033] The beneficial effects of the present invention are:

[0034] (1) By using multiple types of sensor data such as images, sound, temperature and laser to detect train anomalies, the limitations of single-signal train anomaly detection are reduced. In addition, multi-modal signals are used to cover multiple key parts and features of the train to detect the train, achieving a comprehensive assessment of the overall status of the train and improving the comprehensiveness of train anomaly inspections.

[0035] (2) The accuracy of the detection results is improved by using multimodal signal fusion to detect train abnormalities. The fusion processing of multimodal signals solves the false alarm problem caused by data interference in the existing single signal detection, while reducing the underreporting caused by signal blind spots or insufficient coverage, thereby enhancing the accuracy and reliability of train abnormality detection as a whole.

[0036] (3) By using multimodal signal train inspection robots to inspect train anomalies, the time for intervention and judgment in conventional manual inspections is reduced. By using efficient data analysis and calculation, large amounts of data can be quickly processed, thereby improving the efficiency of train anomaly inspections. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a flow chart of a train inspection method based on multimodal signals of the present invention;

[0038] Figure 2 It is a block diagram of a train inspection system based on multimodal signals of the present invention;

[0039] Figure 3 is a multi-modal signal fusion analysis diagram during foreign body detection in the first embodiment of the present invention;

[0040] Figure 4 It is a schematic diagram of the train inspection robot in the present invention. DETAILED DESCRIPTION

[0041] The specific embodiments of the present invention will be described in detail below. It should be noted that the embodiments described here are only part of the embodiments of the present invention, not all of the embodiments. Based on the following embodiments, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] The first embodiment of the present invention proposes a train inspection method based on multimodal signals and a system and storage medium applicable thereto, the method comprising:

[0043] The data acquisition and processing steps include sensor pre-processing, fusion processing of the signals of the multimodal sensors, and synchronization of the time and space information of the data collected by the multimodal sensors to ensure that each sensor is in normal working state and its sampling space calibration point and timestamp are synchronized.

[0044] The data of different modes collected by sensors are collected and organized, and preprocessed, including data cleaning, format conversion and normalization, to make the data of different modes consistent and comparable; feature extraction is performed on the data of each mode to capture the key information, and dimensionality reduction is performed after feature extraction to eliminate redundant information, and data of different modes are aligned in time and space, and the features of different modes are subsequently fused by a hybrid fusion of feature fusion and decision-level fusion.

[0045] Image data, acoustic wave data, light reflectivity data and temperature data are acquired through a number of sensors and fused and then combined to obtain the data set to be inspected. The sensors include cameras, acoustic wave sensors, laser sensors and temperature sensors.

[0046] The abnormality classification step is to determine the type of train abnormality detection corresponding to the position to be inspected according to the position to be inspected, and obtain the data set to be inspected. The accuracy of the data set to be inspected is sorted by reliability sorting. The reliability is verified by the accuracy performance of this type of sensor data in the historical data, and the sensor data with the highest reliability are arranged in descending order. According to the different types of train abnormality detection, the sensor data in the data set to be inspected at the current position to be inspected are screened respectively, and the three data with the highest reliability are retained for abnormality detection. The data with lower reliability are removed to obtain the verification data set. According to the position to be inspected, the interference factor in the sensor data is determined by the interference analysis strategy. The interference factor is the interference factor between several sensor data with mutual influence in the verification data set, that is, the change of one sensor data affects the corresponding change of other sensor signals, which is called the interference factor.

[0047] Data fusion verification step: based on the sensor data in the verification data set combined with the interference factor, the sensor data is verified through the fusion verification strategy. The fusion verification strategy verifies the sensor data in pairs, and verifies the accuracy of one sensor data through other types of sensor data. Based on the verification results, an abnormal analysis data set is obtained, and the abnormal analysis data set contains the abnormal detection results of this type of train;

[0048] The abnormality analysis step determines whether it is abnormal according to the abnormality analysis data set through the abnormality analysis strategy. When every data in the abnormality analysis data set is abnormal, the abnormality is output. At this time, the position to be inspected is marked as an abnormal position for matching with the abnormality detection type of the position to be inspected in the historical data;

[0049] The fault output step outputs the corresponding abnormal results according to the abnormal position, including: foreign body abnormality, body scratch defect, oil level abnormality and motor abnormality.

[0050] In this embodiment, refer to Figures 1 to 4As shown, when the position to be inspected is consistent with the standard foreign body detection position in the database, it is judged that in the data set to be inspected at the position to be inspected, the most reliable sensing data type is image data, followed by acoustic wave data, then light reflectivity data, and finally temperature data. At this time, the temperature data is eliminated to form a verification data set, including image data, acoustic wave data and light reflectivity data; the verification data set package is compared with the standard data set package in the database, that is, the image data, acoustic wave data and light reflectivity data are compared with the standard image data, acoustic wave data and light reflectivity data in the database without abnormal conditions, and a comparison result is obtained. The comparison result outputs an abnormal instruction or a normal instruction, corresponding to the presence or absence of foreign matter at the train position to be inspected.

[0051] By constructing an unsupervised foreign object detection model, foreign objects in the image are detected. When the image data of the input position to be inspected detects the presence of foreign objects, the output result is combined with the sound wave data and light reflectivity data, and the output train detection is abnormal.

[0052] An acoustic wave sensor installed on the train inspection robot sends acoustic waves to the position to be inspected on the train, receives reflected acoustic waves, obtains the corresponding acoustic wave function in the acoustic wave data, performs environmental noise denoising on the corresponding acoustic wave function, obtains a comparison acoustic wave function, converts the comparison acoustic wave function into an acoustic wave comparison graph, compares the acoustic wave comparison graph with the standard acoustic wave graph at the corresponding position in the database, and determines whether the acoustic wave comparison graph and the standard acoustic wave graph are completely fitted, that is, whether the acoustic wave amplitudes are the same. If not, the result is output in combination with the image data and the light reflectivity data, and an abnormality in the train detection is output.

[0053] The comparison sound wave function is configured as:

[0054] ;

[0055] ;

[0056] ;

[0057] Among them, I is the comparison sound wave function, I 0 is the initial sound wave intensity emitted by the sound source, d is the distance between the sound source and the position to be tested, and n is the constant for the propagation of sound waves in the environment, where n=2. is the effect of environmental noise on the acoustic wave function as the distance changes, α is the environmental impact factor, and the environmental impact factor is the degree of influence of noise on the acoustic wave function in the current detection environment. is the effect of the environmental noise on the acoustic wave function as the moving speed of the train inspection robot changes, v is the moving speed of the train inspection robot, k is the speed adjustment factor, and the speed adjustment factor determines the excitation variable according to the ratio of the moving speed of the inspection robot to the standard speed value, v ref By using the comparison sound wave function, the noise change caused by the moving speed of the train inspection robot and the influence of the distance change between the signal receiving position and the position to be inspected on the noise can be directly reflected in the calculation of the collected sound wave data, thereby increasing the calculation accuracy of the influence of artificial noise in the process of sound wave data collection during the train inspection process and improving the accuracy of train abnormality detection.

[0058] The laser sensor carried by the train inspection robot emits laser to the surface of the position to be inspected, receives the laser reflection signal, converts the laser reflection signal into an electrical signal, and then processes the data to obtain light reflectivity data. By comparing the light reflectivity data of the position to be inspected with the standard light reflectivity data in the database, if they are not exactly the same, the result is output in combination with the image data and the sound wave data, and an abnormality in the train detection is output.

[0059] In this embodiment, refer to Figures 1 to 4 As shown, the comparison result of the image data and the comparison result of the acoustic wave data are verified for consistency, and the consistency verification includes obtaining the comparison result of the image data as abnormal or normal for train detection, and obtaining the comparison result of the acoustic wave data as abnormal or normal for train detection. When the two results are consistent, the verification result output is: consistent, and when the two results are inconsistent, the verification result output is inconsistent. According to the verification result, the verification of the image data comparison result and the light reflectivity data comparison result and the verification of the acoustic wave data comparison result and the light reflectivity data comparison result are output, and then according to the verification result, whether the light reflectivity data is abnormal is output. When the consistency verification result of the image data comparison result and the light reflectivity data comparison result is inconsistent after the verification of the acoustic wave data comparison result and the light reflectivity comparison result, it is proved that the light reflectivity comparison result is abnormal at this time.

[0060] In this embodiment, refer to Figures 1 to 4 As shown, when the light reflectivity data is abnormal, the abnormal ratio of the data at several different positions to be inspected is analyzed, and the ratio value is compared with the threshold value to determine whether the light reflectivity data needs to be replaced.

[0061] In this embodiment, refer to Figures 1 to 4 As shown, the interference analysis strategy performs data association on the sensor data in the verification data set. When the sensor data in the verification data set affects the accuracy of other sensor data, it is obtained that there is an interference factor therebetween.

[0062] During the foreign body detection process, when the image data output result indicates that there is a foreign body abnormality on the train, at this time, due to the difference in the material of the foreign body itself, the effect of its change on the light reflectivity is also different. Therefore, the light reflectivity data may have two results, abnormal and normal, when performing abnormal detection. At this time, the light reflectivity data is verified by image data, which is convenient for timely correction of the reliability of light reflectivity in foreign body detection. If the foreign body is made of a material with almost the same light reflectivity as the rail vehicle to be inspected, the light reflectivity data will appear normal when performing abnormal detection, but the fusion of multimodal signals such as image data and sound wave data can tell that there is a detection abnormality in the rail vehicle detection at this time, which means that the comparison result of the light reflectivity data is inaccurate and needs to be calibrated.

[0063] In this embodiment, refer to Figures 1 to 4 As shown, the anomaly analysis strategy, when the verification results of all the sensor data in the anomaly analysis data set output abnormal instructions, the position to be inspected in the anomaly analysis step is abnormal, and the abnormal position is output. The same output of the same anomaly detection by multi-modal sensor signals is used as the standard for abnormal output, which increases the reliability and accuracy of anomaly detection and avoids the problem of inaccurate anomaly detection but unable to be verified due to interference from other factors in the anomaly detection process of a single modal signal.

[0064] In this embodiment, refer to Figures 1 to 4 As shown, when the output result of the light reflectance data is normal, the image data is verified for accuracy, feature points in the image data are extracted, feature descriptors are generated using the feature points, the similarity between the feature descriptors and the feature descriptors of standard image data in the database is calculated, a comparison result is obtained based on the ratio of the similarity to a threshold, and then a check is performed based on the comparison result to determine whether the image data is accurate.

[0065] A first consistency verification is performed on the comparison result of the image data and the comparison result of the acoustic wave data. When the first verification result is inconsistent, a second consistency verification is performed on the comparison result of the image data and the comparison result of the light reflectance data. When the second verification result is consistent, a third consistency verification is performed on the light reflectance data and the acoustic wave data to determine the sensor data in which errors occur among the three types of sensor data. When the second verification result is inconsistent, a third consistency verification is performed on the comparison result of the light reflectance data and the comparison result of the acoustic wave data. When the third verification result is consistent, the image data is verified for accuracy. When the accuracy verification result is inaccurate, the reliability of the image data is re-verified and calculated.

[0066] When there are differences in the detection of the same anomaly by multimodal sensor data during the anomaly detection process, the accuracy of the image data ranked first in reliability is verified again and its reliability is re-ranked, thereby increasing the accuracy of the multimodal sensor signal in train anomaly detection, improving the reliability and detection quality of train anomaly detection, and reducing the risk of increased detection costs due to multiple re-measurements due to inaccurate anomaly detection.

[0067] The system includes:

[0068] A data acquisition module, including a plurality of sensors, for collecting a plurality of sensor data;

[0069] The data processing module includes an abnormal classification submodule, a data fusion submodule and an abnormal analysis submodule, wherein the abnormal classification submodule is used to screen the data set to be inspected at the position to be inspected, output the verification data set, and obtain the interference factor;

[0070] The data fusion submodule is used to perform anomaly analysis on the verification data set and output an anomaly analysis data set;

[0071] The abnormality analysis submodule is used to perform abnormality analysis on the abnormality analysis data set and output whether the position to be inspected is abnormal;

[0072] The data output module is used to determine and output the abnormal results corresponding to the abnormal positions.

[0073] The storage medium stores a computer program / instruction, which, when executed by a processor, implements the steps of the train inspection method based on multimodal signals.

[0074] A second embodiment of the present invention provides a method for detecting oil level abnormality in a railway train based on a multimodal signal, comprising:

[0075] Its verification data set includes: image data, light reflectivity data and sound wave data. The verification data set package is compared with the standard data set package in the database, that is, the image data, light reflectivity data and sound wave data are respectively compared with the standard image data, light reflectivity data and sound wave data in the database without abnormal conditions, and the comparison result is obtained. The comparison result outputs an abnormal instruction or a normal instruction, corresponding to the train to be inspected position at this time. The oil level is abnormal or the oil level is normal. In the oil level detection process, when the oil level collected by the image data is different from the oil level in the standard image, the specific position of the oil level is determined in combination with the determination of the light reflectivity mutation point and the sound wave reflectivity mutation point in the sound wave data, and it is judged whether the oil level is abnormal, and the fuel color is determined by the image. When the fuel appears milky white or dark black, it can be directly output that the fuel is abnormal and the fuel needs to be replaced.

[0076] The third embodiment of the present invention provides a method for detecting abnormality of a railway train motor based on multimodal signals, comprising:

[0077] Its verification data set includes: temperature data, light reflectivity data and sound wave data. The verification data set package is compared with the standard data set package in the database, that is, the temperature data, light reflectivity data and sound wave data are respectively compared with the standard temperature data, light reflectivity data and sound wave data in the database without abnormal conditions, and the comparison result is obtained. The comparison result outputs an abnormal instruction or a normal instruction, corresponding to the motor abnormality or normal motor at the position to be inspected in the train. In the process of motor detection, when the temperature data collects the motor temperature and the standard temperature data, it is too high or too low, combined with the light reflectivity of the position to be inspected and the sound wave reflectivity in the sound wave data, to determine whether the motor has temperature abnormality and motor structure damage, and to judge whether the motor has abnormality.

[0078] The present invention proposes a train inspection method, system and medium based on multimodal signals, which detect train anomalies by fusing multiple types of sensor data such as images, sounds, temperature and lasers, thereby reducing the limitations of single-signal train anomaly detection, and detects trains by covering multiple key parts and features of trains with multimodal signals, thereby achieving a comprehensive evaluation of the overall state of the train and improving the comprehensiveness of train anomaly inspection; by using multimodal signal fusion to detect train anomalies, the accuracy of the detection results is improved, and through the fusion processing of multimodal signals, the false alarm caused by interference in data in the existing single signal detection is solved, while the omission caused by signal blind spots or insufficient coverage is reduced, thereby enhancing the accuracy and reliability of train anomaly detection as a whole; by using a train inspection robot using multimodal signals to inspect train anomalies, the time for intervention and judgment in the conventional manual inspection process is reduced, and by using efficient data analysis and calculation, rapid processing of large amounts of data is achieved, thereby improving the efficiency of train anomaly inspection.

[0079] The above shows and describes the basic features, principles and advantages of the present invention. It should be noted that the present invention is not limited to the above embodiments, which are only partial embodiments. Under the premise of not departing from the spirit and scope of the present invention, several improvements and supplements made are considered as the protection scope of the present invention.

Claims

1. A train inspection method based on multimodal signals, characterized in that: The following steps are involved: The data acquisition and processing step is to perform sensor pre-processing, synchronously collect the time and space information of the data, obtain image data, sound wave data, light reflectivity data and temperature data through a number of sensors and combine them to obtain the data set to be tested. The sound wave data is configured with a comparison sound wave function: ； ； ； Among them, I is the comparison sound wave function, I0 is the initial sound wave intensity emitted by the sound source, d is the distance between the sound source and the position to be measured, and n is the constant for the propagation of sound waves in the environment. is the effect of environmental noise on the sound wave function as the distance changes, α is the environmental impact factor, is the influence function of environmental noise on the acoustic wave function as the moving speed of the train inspection robot changes, v is the moving speed of the train inspection robot, k is the speed adjustment factor, and v ref is the standard speed value; In the abnormal classification step, the sensor data in the data set to be inspected at the current position to be inspected are sorted by reliability according to the data set to be inspected corresponding to the position to be inspected, and the verification data set is obtained. The interference factor in the sensor data is determined according to the position to be inspected through the interference analysis strategy; Data fusion verification step: verify the sensor data by combining the sensor data in the verification data set with the interference factors through the fusion verification strategy, and obtain the abnormal analysis data set according to the verification results; Anomaly analysis step: judging whether it is abnormal or not by using anomaly analysis strategy according to the anomaly analysis data set; if abnormal, marking the position to be inspected as an abnormal position; The fault output step outputs the corresponding abnormal result according to the abnormal position, including: foreign body abnormality, oil level abnormality and motor abnormality.

2. A train inspection method based on multimodal signals according to claim 1, characterized in that: When the position to be inspected is consistent with the standard foreign body detection position in the database, the verification data set includes image data, sound wave data and light reflectivity data; the verification data set package is compared with the standard data set package in the database, and the comparison result outputs an abnormal instruction or a normal instruction.

3. A train inspection method based on multimodal signals according to claim 2, characterized in that: The comparison result of the image data and the comparison result of the sound wave data are verified for consistency, and the verification of the comparison result of the image data and the comparison result of the light reflectance data and the verification of the comparison result of the sound wave data and the comparison result of the light reflectance data are output according to the verification result, and then whether the light reflectance data is abnormal is output according to the verification result.

4. A train inspection method based on multimodal signals according to claim 3, characterized in that: When the light reflectivity data is abnormal, the abnormal ratio of the data at several different positions to be inspected is analyzed, and the ratio value is compared with a threshold value to determine whether the light reflectivity data needs to be replaced.

5. A train inspection method based on multimodal signals according to claim 1, characterized in that: The interference analysis strategy performs data association on the sensor data in the verification data set, and when the sensor data therein affects the accuracy of other sensor data, it is obtained that there is an interference factor therebetween.

6. A train inspection method based on multimodal signals according to claim 1, characterized in that: The abnormal analysis strategy is that when the verification results of all the sensor data in the abnormal analysis data set output abnormal instructions, the position to be inspected in the abnormal analysis step is abnormal and the abnormal position is output.

7. A train inspection method based on multimodal signals according to claim 3, characterized in that: When the output result of the light reflectance data is normal, the image data is verified for accuracy, feature points in the image data are extracted, feature descriptors are generated using the feature points, the similarity between the feature descriptors and the feature descriptors of standard image data in the database is calculated, a comparison result is obtained based on the ratio of the similarity to a threshold, and then a check is performed to determine whether the image data is accurate based on the comparison result.

8. A train inspection system based on multimodal signals, providing a control system of a train inspection method based on multimodal signals as claimed in any one of claims 1 to 7, characterized in that: include: A data acquisition module, including a plurality of sensors, for acquiring the image data, sound wave data, light reflectivity data and temperature data; The data processing module includes an abnormal classification submodule, a data fusion submodule and an abnormal analysis submodule, wherein the abnormal classification submodule is used to screen the data set to be inspected at the position to be inspected, output the verification data set, and obtain the interference factor; The data fusion submodule is used to perform anomaly analysis on the verification data set and output an anomaly analysis data set; The abnormality analysis submodule is used to perform abnormality analysis on the abnormality analysis data set and output whether the position to be inspected is abnormal; The data output module is used to determine and output the abnormal results corresponding to the abnormal positions.

9. A train inspection storage medium based on multimodal signals, on which a computer program / instruction is stored, characterized in that: When the computer program / instruction is executed by a processor, the steps of the train inspection method based on multimodal signals described in any one of claims 1 to 7 are implemented.

Images