Intelligent railway inspection method and system based on multi-mode data fusion

Abstract

The present invention provides an intelligent railway inspection method and system WALLET (rall W A Yinte L Ligent insp Ection method and sysTem) based on multi-mode data fusion, comprising the following steps: step one: fusion expression of multi-modal data of inspection scenes; step two: accurately characterizing the characteristics of railway equipment hidden in multi-modal data through multi-modal object feature extraction and description, and realizing multi-modal data analysis; step three: constructing an end-edge-cloud fusion architecture consisting of an intelligent inspection instrument, an inspection edge server and an inspection analysis platform to realize dynamic transmission and calculation; step four: oriented to intelligent railway inspection applications, dynamically constructing a distributed data flow graph to realize parallel acceleration based on data flow.

Description

Technical Field

[0001] The present invention relates to an intelligent railway inspection method and system WALLET (railway inspection method and system) based on multi-mode data fusion, which relates to the fields of big data processing, machine learning, cloud computing, etc. Background Art

[0002] Railway inspection is an important guarantee for driving safety. The rapid development of my country's railway mileage and transportation tasks has put forward higher technical requirements for the railway inspection system. In order to maintain the safe operation of trains, track inspectors must regularly inspect the tracks (railway subgrade, tracks, switches, signal lights, railway bridges, facilities along the railway, and communication lines) and the status of locomotives to promptly discover and eliminate hidden dangers along the way.

[0003] The existing railway inspection scheme mainly adopts human defense and physical defense. The former mainly relies on manual work, including line inspection before delivery, vehicle inspection for picking up vehicles, and loading inspection of freight trains. The inspection of each work post and the handover of each type of work are all realized through manual methods such as radio reporting, marking of the first and last vehicles, record signing, and counterpart handover, which increases the difficulty of real-time decision-making for railway operators. At the same time, human defense relies on the expert experience of inspectors, and the inspection cost is high and the scope is limited; the latter collects information through fixed-point installed cameras and sensors, with a single monitoring object, limited scope, poor environmental adaptability, and high false alarm rate. In recent years, the concept of railway safety technical defense through intelligent analysis has been proposed one after another, but the relevant technology is still far from the actual needs. In short, the existing schemes generally have problems such as targeting specific data, low analysis accuracy, difficulty in automation, and low timeliness.

[0004] After searching the existing technical literature, it was found that Jin X et al. (JIN X, WANG Y, ZHANG H, et al. DM-RIS: Deep Multimodel Rail Inspection System With Improved MRF-GMM and CNN. IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT, 2020, 69(41): 1051-1065.) established a deep multimodal model for surface defects, proposed a fast and robust spatially constrained Gaussian mixture model for segmentation proposals, and used Faster RCNN for target localization in a parallel structure. However, this method cannot achieve effective multimodal feature extraction and form sparse representation. Zhao Yang (Zhao Yang. Fault diagnosis of onboard equipment of high-speed railway signal system based on text mining. Beijing Jiaotong University, 2015.) used topic model to analyze fault text data and obtained the expression of text in topic space, so as to reduce dimension and extract features. On this basis, in view of the uncertainty of onboard equipment fault diagnosis, Bayesian network was used as fault diagnosis method, and a Bayesian structure learning algorithm suitable for onboard equipment was proposed by fully combining field data and domain knowledge. However, this method is highly dependent on text data and cannot be applied to multimodal data in railway inspection. Li Jun et al. (Li Jun, Zhong Zhinong et al. Multi-stage collaborative task planning method for aerospace resources under heterogeneous MAS structure. Acta Aeronautica Sinica, 2013, 34(7):1682-1697) analyzed the resource heterogeneity in resource collaborative planning and proposed a multi-stage monitoring task decomposition method. They also established a multi-stage collaborative task planning model for heterogeneous multi-agent system (MAS) based on resource status and task dependency, and built a collaborative task planning service platform, making full use of the complementary advantages of heterogeneous resources. However, this method is limited to periodic satellite systems and is difficult to be directly applied to complex railway inspection systems. Suettlerlein et al. (Suettlerlein J, Zuckerman S, Gao G R. An implementation of the codelet model. European Conference on Parallel Processing. Springer, Berlin, Heidelberg, 2013: 633-644) proposed a codelet model based on data flow to implement data-driven parallel scheduling of tasks on heterogeneous computing resources. However, this method does not take into account the edge-cloud collaborative inspection system architecture, and it is difficult to give full play to the advantages of heterogeneous resources. Summary of the invention

[0005] The purpose of the present invention is to overcome the deficiencies in existing methods and systems, and to provide an intelligent railway inspection method and system based on multi-mode data fusion. For the multi-mode data in the inspection scene, full use is made of the knowledge in the railway and inspection fields to realize the fusion expression and intelligent analysis of multi-modal data. An edge-cloud collaborative inspection system is constructed to realize the collection of data while transmitting and calculating, thereby reducing the analysis delay. A data flow graph is constructed from a data-driven perspective to realize parallel computing.

[0006] An intelligent railway inspection method based on multi-mode data fusion provided by the present invention comprises the following steps:

[0007] Step 1: Use sparse autoencoder network to extract features of contact network;

[0008] Step 2: Obtain the feature representation of the performance index in a low-dimensional space;

[0009] Step 3: Further feature fusion of features extracted from multiple performance indicators based on manual intervention;

[0010] Step 4: Implicit feature characterization of railway equipment in multimodal data;

[0011] Step 5: Mining the mapping relationship between local features and global features;

[0012] Step 6: Analyze and predict potential failures of the railway system;

[0013] Step 7: Build an edge-cloud fusion architecture consisting of intelligent inspection instruments, inspection edge servers, and inspection analysis platforms;

[0014] Step 8: Improve computing localization through data transmission and computing driven by inspection services, and reduce the delay of inspection service analysis and the load of cloud computing centers;

[0015] Step 9: For intelligent railway inspection applications, dynamically build a distributed data flow graph and add a description of streaming input and output and its storage access characteristics;

[0016] Step 10: Implement distributed parallel execution of data flow graphs through efficient segmentation of data flow graph subgraphs and segmentation of subgraphs adapted to heterogeneous computing resource capabilities.

[0017] Preferably, the step 1 comprises the following steps:

[0018] Step 1.1: Extract the features of the contact network morphology by using a sparse autoencoder network;

[0019] Step 1.2: Extract features of contact network usage time by using sparse autoencoder network;

[0020] Step 1.2: Extract the characteristics of contact network usage by using sparse autoencoder network;

[0021] Preferably, the step 1.1 comprises the following steps:

[0022] Step 1.1.1: Construct the geometric topological structure formed by different types of features;

[0023] Step 1.1.2: Analyze the unique embedded structure of multimodal features;

[0024] Step 1.1.3: Feature selection and representation through multimodal complex heterogeneous feature sparseness;

[0025] Preferably, step 2 comprises the following steps:

[0026] Step 2.1: Extract features from multiple input features respectively;

[0027] Step 2.2: Perform Fourier encoding on the extracted features;

[0028] Step 2.3: Splice the encoded features into the autoencoder neural network;

[0029] Step 2.4: Get the common features after feature compression;

[0030] Step 2.5: Add human intervention to reduce the error of feature compression process;

[0031] Preferably, step 6 comprises the following steps:

[0032] Step 6.1: Use a sparse autoencoder network to extract features from historical sample data such as contact network morphology, contact network usage time and characteristics;

[0033] Step 6.2: Obtain the characteristic representation of the performance index;

[0034] Step 6.3: Use the sparse autoencoder network to further fuse the features extracted from multiple performance indicators;

[0035] Step 6.4: Get the common feature representation

[0036] Step 6.5: Add human intervention and add the possibility of human-predicted failures at certain time points on the timeline;

[0037] Step 6.6: Take the fusion features and manual intervention features as input, conduct supervised learning with the neural network, analyze the fault location, and predict the fault probability;

[0038] Preferably, step 8 comprises the following steps:

[0039] Step 8.1: Use the triple T =<ID,Op,Cstr> To describe the atomic task, ID represents the atomic task serial number, Op represents the set of operations of the atomic task on the data, and Cstr represents the set of constraints of the atomic task, including the dependent data set and the maximum execution time tolerated.

[0040] Step 8.2: Decompose the tasks based on inspection domain knowledge and expert system;

[0041] Step 8.3: Use quadruple to describe the computation task;

[0042] Step 8.4: Perform calculation planning based on cooperative game;

[0043] Preferably, the step 8.2 comprises the following steps:

[0044] Step 8.2.1: Decompose the tasks based on inspection domain knowledge and expert system;

[0045] Step 8.2.2: Construct task decomposition graph DG = (V, E, {or, join});

[0046] Step 8.2.3: The nodes v∈V in the graph represent atomic tasks, and the edges e∈E represent the dependencies between atomic tasks;

[0047] Step 8.2.4: The connectors or and join are used in pairs to represent different ways of decomposing subtasks. You only need to select one of them to execute.

[0048] Preferably, the step 8.3 comprises the following steps:

[0049] Step 8.3.1: Construct a quad<S,{DG},Action,Gain> Describes the computing task, S represents the status of system resources, S 0 Indicates the initial state of the system. {DG} = {(V, E, {split, join})} is the task decomposition diagram of all services booked in the system. At each time t, the node can choose to perform a subtask of a certain service, forming a decision set:

[0050] Action(t) = {A i},A i ∈V

[0051] Among them, A i Represents the set of subtasks executed by point i.

[0052] Step 8.3.2: Construct a decision transfer function The overall benefits of the system are affected by resources and decisions. The calculation of the value reflects the process of cooperative game among computing nodes, that is, the computing overhead caused by the node executing the subtask and the final overall transmission benefit of the system. The computational cost is evaluated based on the computational complexity of the subtask, the amount of data to be processed, and the resource conditions of the node.

[0053] Step 8.3.3: Evaluate the transfer benefit based on the difference in the amount of data after the subtask is executed. Define the decision transfer function Transfer as follows:

[0054]

[0055] Action1 and Action2 represent the corresponding decision sets after i and j exchange strategies respectively.

[0056] Preferably, the step 8.4 comprises the following steps:

[0057] Step 8.4.1: With the goal of maximizing the overall benefit of the system, a propagation planning problem is constructed with the constraint condition of ensuring that a complete path in each task decomposition graph is covered by the decision set;

[0058] Step 8.4.2: Based on the iterative negotiation mechanism, complete the solution of the transmission planning cooperative game process.

[0059] Preferably, the step 8.4.2 comprises the following steps:

[0060] Step 8.4.2.1: First find a feasible solution Action(t 0 ), and calculate the overall system benefit

[0061] Step 8.4.2.2: Within the range of iterations, select any two nodes to execute the decision transfer function and choose the decision solution that meets the transfer condition (i.e., the transfer benefit is greater than zero);

[0062] Step 8.4.2.3: At the end of the iteration, the optimal decision solution is obtained and the next round of iterative negotiation begins;

[0063] Preferably, step 9 comprises the following steps:

[0064] Step 9.1: Expand the data flow programming model and abstract a set of operations without data dependencies into data flow nodes. The weight of the node is the processing size of the intermediate data per unit time. The nodes are connected by directed edges, and the weight of the edge represents the size of the data migration dependency between tasks.

[0065] Step 9.2: To express various business characteristics, add two types of time attributes t and d to the edge. The former represents the shortest time interval for data arrival, and the latter represents the latest completion time of the corresponding operation of the node;

[0066] Preferably, the step 10 comprises the following steps:

[0067] Step 10.1: Perform sequence analysis and time-space estimation on the execution time of the data flow graph;

[0068] Step 10.2: To optimize the layout of the distributed data flow graph, the dynamic data flow graph is parallelized to heterogeneous computing resources. The computing nodes where the tasks are finally placed are:

[0069]

[0070] Preferably, the step 10.1 comprises the following steps:

[0071] Step 10.1.1: Sort each task in the data flow graph in descending order of the total amount of dependent data, starting with the first task (denoted as t i ), determine whether the ratio of computing node load exceeds k times;

[0072] Step 10.1.2: If it exceeds, select a computing node with lower load to place the task;

[0073] Step 10.1.3: If there are no dependent tasks, place the current task directly on the computing node with lower load;

[0074] Step 10.1.4: If t i There is a dependent branch, and the computing node locations of other tasks in the dependent branch are known. i Place it on the computing node with the least amount of data migration;

[0075] Step 10.1.5: Let the set of other tasks in the dependent branch be T = {t j}, and t j The placed computing node is denoted as c j , t j For t i The amount of input data generated is d ij , then t i Placed in c j The amount of data migration brought about by the above is:

[0076]

[0077] According to the present invention, an intelligent railway inspection system based on multi-mode data fusion is provided, and the intelligent railway inspection system based on multi-mode data fusion uses the above-mentioned intelligent railway inspection method based on multi-mode data fusion. The system is based on data fusion expression and includes a multi-mode data fusion module, an intelligent analysis module, a dynamic transmission module and a parallel acceleration module.

[0078] This system includes a graphical user interface for railway operators, through which interactive operations such as multi-mode data fusion, parallel acceleration, and inspection result analysis can be realized. When using this system, first open the graphical user interface, then select the target inspection railway section and inspection time period according to the needs of the railway operator, if intelligent analysis is to be performed, select the intelligent analysis module, if dynamic calculation is to be performed, select the dynamic calculation module, and if parallel acceleration is to be performed, select the parallel acceleration module.

[0079] The multi-mode data fusion module runs on the inspection analysis platform as a background program. For inspection data such as rail deformation, brush anomaly, wheel wear, and foreign body intrusion, two types of interfaces are provided: query and monitoring. The query interface specifies the storage location of the inspection data (such as edge storage media or cloud computing center storage media) and the fault category (contact network, wheel, rail or foreign body). The monitoring interface can specify the monitoring frequency, time interval range, and data sampling point interval. After obtaining the original inspection data through the query and monitoring interfaces, the multi-modal feature extraction function can be selected to characterize the mapping mechanism between the power supply equipment's own features, local features, and global features hidden in the multi-modal data. At the same time, the sparse representation and selection method of multi-modal complex heterogeneous features can be selected to construct the geometric topological structure formed by different types of attributes and reveal the unique embedded structure of multi-modal attributes.

[0080] After selecting the intelligent analysis module, the fusion data expression technology uses a sparse autoencoder network to extract features from historical sample data such as the contact network morphology, contact network usage time and characteristics, and obtains the feature representation of these performance indicators in low-dimensional space. During the monitoring process, each performance indicator is monitored separately, and these indicators are intrinsically related. We use multi-mode data values ​​such as contact network morphology (video, image) and contact network usage time and characteristics (text) to jointly predict the possibility of contact network failure. The first step of the model is to extract features from multiple input features, then encode them separately and splice them into the autoencoder neural network. The common features of the two features after compression can be obtained. A simple test can be used to see whether the shared features are well representative. The minimum mean square error (MSE) method is used to measure the reconstruction criterion. In the actual application process, considering that people can predict many things that computers cannot predict, adding human intervention can achieve better prediction results. Next, the prediction model based on neural network is studied. First, feature extraction is performed on historical sample data such as contact network morphology, contact network usage time and characteristics. The extraction method uses a sparse autoencoder network. The feature representations of these performance indicators are obtained separately, and this representation should have a lower dimension than the original data. Then, the features extracted from multiple performance indicators are further fused, and the method used is also the sparse autoencoder network, so as to obtain a common feature representation. Next, add manual intervention and add the possibility of manual prediction of possible failure at certain time points on the timeline. Finally, the fused features and manual intervention features are used as inputs to perform supervised learning of the neural network to predict the probability of failure.

[0081] After selecting the dynamic calculation module, the task decomposition diagram of the business is constructed and visualized according to the status of system resources and the initial state of the system. After evaluating and solving the calculation benefits of different solutions, the dynamic calculation solution with the maximum system gain is provided to the user.

[0082] After selecting the parallel acceleration module, the distributed data flow graph is dynamically constructed for intelligent railway inspection applications. Through resource-friendly heterogeneous device collaboration, fine-grained pipeline parallelism, efficient virtual memory and page management, efficient data flow graph sub-graph segmentation, heterogeneous computing resource capacity adaptation sub-graph segmentation and data flow graph distributed parallel execution are achieved.

[0083] Compared with the prior art, the present invention has the following beneficial effects: through multi-modal data fusion, the fault point can be accurately located, and automatic, rapid and intelligent analysis can be achieved.

[0084] The present invention is characterized in that:

[0085] 1. Deep fusion of multi-mode data: Make full use of multi-mode data such as images and sensors to improve the accuracy of fault analysis.

[0086] 2. High degree of automation: Through data collection from onboard equipment and intelligent analysis algorithms, the inspection process can be highly automated, greatly reducing the burden on railway operators.

[0087] 3. Rapid fault analysis: Parallel analysis acceleration is achieved based on data flow graphs to quickly obtain analysis results.

[0088] 4. Deep edge-cloud collaboration: Inspection data is calculated while being transmitted, deeply integrating edge cloud computing resource capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0090] Figure 1 This is a diagram of the intelligent inspection system.

[0091] Figure 2 This is a multimodal data fusion diagram.

[0092] Figure 3 This is a diagram of the task splitting mechanism.

[0093] Figure 4 This is a diagram of the dynamic calculation scheme.

[0094] Figure 5 Figure 2 is a diagram of data flow-based parallel acceleration technology.

[0095] Figure 6 It is the system functional architecture and module diagram. DETAILED DESCRIPTION

[0096] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0097] The adaptive cooperative forwarding method based on moderate service in a wireless network of the present invention comprises the following steps:

[0098] An intelligent railway inspection method based on multi-mode data fusion provided by the present invention comprises the following steps:

[0099] Step 1: Use sparse autoencoder network to extract features of contact network;

[0100] Step 2: Obtain the feature representation of the performance index in a low-dimensional space;

[0101] Step 3: Further feature fusion of features extracted from multiple performance indicators based on manual intervention;

[0102] Step 4: Implicit feature characterization of railway equipment in multimodal data;

[0103] Step 5: Mining the mapping relationship between local features and global features;

[0104] Step 6: Analyze and predict potential failures of the railway system;

[0105] Step 7: Build an edge-cloud fusion architecture consisting of intelligent inspection instruments, inspection edge servers, and inspection analysis platforms;

[0106] Step 8: Improve computing localization through data transmission and computing driven by inspection services, and reduce the delay of inspection service analysis and the load of cloud computing centers;

[0107] Step 9: For intelligent railway inspection applications, dynamically build a distributed data flow graph and add a description of streaming input and output and its storage access characteristics;

[0108] Step 10: Implement distributed parallel execution of data flow graphs through efficient segmentation of data flow graph subgraphs and segmentation of subgraphs adapted to heterogeneous computing resource capabilities.

[0109] Preferably, the step 1 comprises the following steps:

[0110] Step 1.1: Extract the features of the contact network morphology by using a sparse autoencoder network;

[0111] Step 1.2: Extract features of contact network usage time by using sparse autoencoder network;

[0112] Step 1.2: Extract the characteristics of contact network usage by using sparse autoencoder network;

[0113] Preferably, the step 1.1 comprises the following steps:

[0114] Step 1.1.1: Construct the geometric topological structure formed by different types of features;

[0115] Step 1.1.2: Analyze the unique embedded structure of multimodal features;

[0116] Step 1.1.3: Feature selection and representation through multimodal complex heterogeneous feature sparseness;

[0117] Preferably, step 2 includes the following steps (such as Figure 2 shown):

[0118] Step 2.1: Extract features from multiple input features respectively;

[0119] Step 2.2: Perform Fourier encoding on the extracted features;

[0120] Step 2.3: Splice the encoded features into the autoencoder neural network;

[0121] Step 2.4: Get the common features after feature compression;

[0122] Step 2.5: Add human intervention to reduce the error of feature compression process;

[0123] Preferably, step 6 comprises the following steps:

[0124] Step 6.1: Use a sparse autoencoder network to extract features from historical sample data such as contact network morphology, contact network usage time and characteristics;

[0125] Step 6.2: Obtain the characteristic representation of the performance index;

[0126] Step 6.3: Use the sparse autoencoder network to further fuse the features extracted from multiple performance indicators;

[0127] Step 6.4: Get the common feature representation

[0128] Step 6.5: Add human intervention and add the possibility of human-predicted failures at certain time points on the timeline;

[0129] Step 6.6: Take the fusion features and manual intervention features as input, conduct supervised learning with the neural network, analyze the fault location, and predict the fault probability;

[0130] Preferably, step 8 comprises the following steps:

[0131] Step 8.1: Use the triple T =<ID,Op,Cstr> To describe the atomic task, ID represents the atomic task serial number, Op represents the set of operations of the atomic task on the data, and Cstr represents the set of constraints of the atomic task, including the dependent data set and the maximum execution time tolerated.

[0132] Step 8.2: Decompose the tasks based on inspection domain knowledge and expert system;

[0133] Step 8.3: Use quadruple to describe the computation task;

[0134] Step 8.4: Perform calculation planning based on cooperative game;

[0135] Preferably, the step 8.2 includes the following steps (such as Figure 3 as shown):

[0136] Step 8.2.1: Decompose the tasks based on inspection domain knowledge and expert system;

[0137] Step 8.2.2: Construct task decomposition graph DG = (V, E, {or, join});

[0138] Step 8.2.3: The nodes v∈V in the graph represent atomic tasks, and the edges e∈E represent the dependencies between atomic tasks;

[0139] Step 8.2.4: The connectors or and join are used in pairs to represent different ways of decomposing subtasks. You only need to select one of them to execute.

[0140] Preferably, the step 8.3 comprises the following steps:

[0141] Step 8.3.1: Construct a quad<S,{DG},Action,Gain> Describes the computing task, S represents the status of system resources, S 0 Indicates the initial state of the system. {DG} = {(V, E, {split, join})} is the task decomposition diagram of all services booked in the system. At each time t, the node can choose to perform a subtask of a certain service, forming a decision set:

[0142] Action(t) = {A i},A i ∈V

[0143] Among them, A i Represents the set of subtasks executed by point i.

[0144] Step 8.3.2: Construct a decision transfer function The overall benefits of the system are affected by resources and decisions. The calculation of the value reflects the process of cooperative game among computing nodes, that is, the computing overhead caused by the node executing the subtask and the final overall transmission benefit of the system. The computational cost is evaluated based on the computational complexity of the subtask, the amount of data to be processed, and the resource conditions of the node.

[0145] Step 8.3.3: Evaluate the transfer benefit based on the difference in the amount of data after the subtask is executed. Define the decision transfer function Transfer as follows:

[0146]

[0147] Action1 and Action2 represent the corresponding decision sets after i and j exchange strategies respectively.

[0148] Preferably, the step 8.4 includes the following steps (such as Figure 4as shown):

[0149] Step 8.4.1: With the goal of maximizing the overall benefit of the system, a propagation planning problem is constructed with the constraint condition of ensuring that a complete path in each task decomposition graph is covered by the decision set;

[0150] Step 8.4.2: Based on the iterative negotiation mechanism, complete the solution of the transmission planning cooperative game process.

[0151] Preferably, the step 8.4.2 comprises the following steps:

[0152] Step 8.4.2.1: First find a feasible solution Action(t 0 ), and calculate the overall system benefit

[0153] Step 8.4.2.2: Within the range of iterations, select any two nodes to execute the decision transfer function and choose the decision solution that meets the transfer condition (i.e., the transfer benefit is greater than zero);

[0154] Step 8.4.2.3: At the end of the iteration, the optimal decision solution is obtained and the next round of iterative negotiation begins;

[0155] Preferably, step 9 comprises the following steps:

[0156] Step 9.1: Expand the data flow programming model and abstract a set of operations without data dependencies into data flow nodes. The weight of the node is the processing size of the intermediate data per unit time. The nodes are connected by directed edges, and the weight of the edge represents the size of the data migration dependency between tasks.

[0157] Step 9.2: To express various business characteristics, add two types of time attributes t and d to the edge. The former represents the shortest time interval for data arrival, and the latter represents the latest completion time of the corresponding operation of the node;

[0158] Preferably, the step 10 comprises the following steps:

[0159] Step 10.1: Perform sequence analysis and time-space estimation on the execution time of the data flow graph;

[0160] Step 10.2: To optimize the layout of the distributed data flow graph, the dynamic data flow graph is parallelized to heterogeneous computing resources. The computing nodes where the tasks are finally placed are:

[0161]

[0162] Preferably, the step 10.1 includes the following steps (such as Figure 5 shown):

[0163] Step 10.1.1: Sort each task in the data flow graph in descending order of the total amount of dependent data, starting with the first task (denoted as t i ), determine whether the ratio of computing node load exceeds k times;

[0164] Step 10.1.2: If it exceeds, select a computing node with lower load to place the task;

[0165] Step 10.1.3: If there are no dependent tasks, place the current task directly on the computing node with lower load;

[0166] Step 10.1.4: If t i There is a dependent branch, and the computing node locations of other tasks in the dependent branch are known. i Place it on the computing node with the least amount of data migration;

[0167] Step 10.1.5: Let the set of other tasks in the dependent branch be T = {t j}, and t j The placed computing node is denoted as c j , t j For t i The amount of input data generated is d ij , then t i Placed in c j The amount of data migration brought about by the above is:

[0168]

[0169] According to the present invention, an intelligent railway inspection system based on multi-mode data fusion is provided, and the intelligent railway inspection system based on multi-mode data fusion uses the above-mentioned intelligent railway inspection method based on multi-mode data fusion. The system is based on data fusion expression, including a multi-mode data fusion module, an intelligent analysis module, a dynamic transmission module and a parallel acceleration module (such as Figure 6 shown).

[0170] This system includes a graphical user interface for railway operators, through which interactive operations such as multi-mode data fusion, parallel acceleration, and inspection result analysis can be realized. When using this system, first open the graphical user interface, then select the target inspection railway section and inspection time period according to the needs of the railway operator, if intelligent analysis is to be performed, select the intelligent analysis module, if dynamic calculation is to be performed, select the dynamic calculation module, and if parallel acceleration is to be performed, select the parallel acceleration module.

[0171] The multi-mode data fusion module runs on the inspection analysis platform as a background program. For inspection data such as rail deformation, brush anomaly, wheel wear, and foreign body intrusion, two types of interfaces are provided: query and monitoring. The query interface specifies the storage location of the inspection data (such as edge storage media or cloud computing center storage media) and the fault category (contact network, wheel, rail or foreign body). The monitoring interface can specify the monitoring frequency, time interval range, and data sampling point interval. After obtaining the original inspection data through the query and monitoring interfaces, the multi-modal feature extraction function can be selected to characterize the mapping mechanism between the power supply equipment's own features, local features, and global features hidden in the multi-modal data. At the same time, the sparse representation and selection method of multi-modal complex heterogeneous features can be selected to construct the geometric topological structure formed by different types of attributes and reveal the unique embedded structure of multi-modal attributes.

[0172] After selecting the intelligent analysis module, the fusion data expression technology uses a sparse autoencoder network to extract features from historical sample data such as the contact network morphology, contact network usage time and characteristics, and obtains the feature representation of these performance indicators in low-dimensional space. During the monitoring process, each performance indicator is monitored separately, and these indicators are intrinsically related. We use multi-mode data values ​​such as contact network morphology (video, image) and contact network usage time and characteristics (text) to jointly predict the possibility of contact network failure. The first step of the model is to extract features from multiple input features, then encode them separately and splice them into the autoencoder neural network. The common features of the two features after compression can be obtained. A simple test can be used to see whether the shared features are well representative. The minimum mean square error (MSE) method is used to measure the reconstruction criterion. In the actual application process, considering that people can predict many things that computers cannot predict, adding human intervention can achieve better prediction results. Next, the prediction model based on neural network is studied. First, feature extraction is performed on historical sample data such as contact network morphology, contact network usage time and characteristics. The extraction method uses a sparse autoencoder network. The feature representations of these performance indicators are obtained separately, and this representation should have a lower dimension than the original data. Then, the features extracted from multiple performance indicators are further fused, and the method used is also the sparse autoencoder network, so as to obtain a common feature representation. Next, add manual intervention and add the possibility of manual prediction of possible failure at certain time points on the timeline. Finally, the fused features and manual intervention features are used as inputs to perform supervised learning of the neural network to predict the probability of failure.

[0173] After selecting the dynamic calculation module, the task decomposition diagram of the business is constructed and visualized according to the status of system resources and the initial state of the system. After evaluating and solving the calculation benefits of different solutions, the dynamic calculation solution with the maximum system gain is provided to the user.

[0174] After selecting the parallel acceleration module, the distributed data flow graph is dynamically constructed for intelligent railway inspection applications. Through resource-friendly heterogeneous device collaboration, fine-grained pipeline parallelism, efficient virtual memory and page management, efficient data flow graph sub-graph segmentation, heterogeneous computing resource capacity adaptation sub-graph segmentation and data flow graph distributed parallel execution are achieved.

[0175] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which will not affect the essence of the present invention.

Claims

1. An intelligent railway inspection method based on multi-mode data fusion, characterized in that: The following steps are involved: Step 1: Use sparse autoencoder network to extract features of contact network; Step 2: Obtain the feature representation of the performance index in a low-dimensional space; Step 3: Further feature fusion of features extracted from multiple performance indicators based on manual intervention; Step 4: Implicit feature characterization of railway equipment in multimodal data, wherein the multimodal data includes video and image data of contact network morphology and text data of contact network usage time and characteristic text data; Step 5: Mining the mapping relationship between local features and global features; Step 6: Analyze and predict potential failures of the railway system; Step 7: Build an edge-cloud fusion architecture consisting of intelligent inspection instruments, inspection edge servers, and inspection analysis platforms; Step 8: Improve computing localization and reduce inspection service analysis latency and cloud computing center load through edge-cloud fusion architecture. The step 8 comprises the following steps: Step 8.1: Using Triples >To describe the atomic task, ID represents the atomic task serial number, Op represents the set of operations of the atomic task on the data, and Cstr represents the set of constraints of the atomic task, including the dependent data set and the maximum tolerated execution time; Step 8.2: Decompose the tasks based on inspection domain knowledge and expert system; Step 8.3: Use quadruple to describe the computation task; Step 8.4: Perform calculation planning based on cooperative game; The step 8.2 comprises the following steps: Step 8.2.1: Decompose the tasks based on inspection domain knowledge and expert system; Step 8.2.2: Construct the task decomposition graph DG = (V, E, {or, join}); Step 8.2.3: Nodes in the graph Represents an atomic task, edge Represents dependencies between atomic tasks; Step 8.2.4: The connectors or and join are used in pairs to represent different ways of decomposing subtasks. You only need to select one of them to execute. The step 8.3 comprises the following steps: Step 8.3.1: Construct a quad Describes the computing task, S represents the status of system resources, represents the initial state of the system; {DG}={(V,E,{split,join})} is the task decomposition diagram of all services booked in the system; at each time t, the node selects a subtask to execute a certain service, forming a decision set: , ， in, express A collection of subtasks executed at a point; Step 8.3.2: Construct a decision transfer function ; The overall benefits of the system are affected by resources and decisions, The calculation of the value reflects the process of cooperative game among computing nodes, that is, the computing overhead caused by the node executing the subtask and the final overall transmission benefit of the system. , the computational cost is evaluated based on the computational complexity of the subtask, the amount of data to be processed, and the resource conditions of the node; Step 8.3.3: Evaluate the transfer benefit based on the difference in the amount of data after the subtask is executed; define the decision transfer function Transfer as follows: ， and Respectively represent the decision sets corresponding to the exchange strategies of i and j; The step 8.4 comprises the following steps: Step 8.4.1: With the goal of maximizing the overall benefit of the system, a propagation planning problem is constructed with the constraint condition of ensuring that a complete path in each task decomposition graph is covered by the decision set; Step 8.4.2: Based on the iterative negotiation mechanism, complete the solution of the cooperative game process of the transmission planning; The step 8.4.2 comprises the following steps: Step 8.4.2.1: First find a feasible solution for the algorithm , and calculate the overall system benefit ; Step 8.4.2.2: Within the range of iterations, select any two nodes to execute the decision transfer function and choose the decision solution with a transfer benefit greater than zero; Step 8.4.2.3: At the end of the iteration, the optimal decision solution is obtained and the next round of iterative negotiation begins; Step 9: According to the above steps, dynamically construct a distributed data flow graph, and add a description of the streaming input and output and the storage access characteristics of the streaming input and output; Step 10: Implement distributed parallel execution of data flow graphs through efficient segmentation of data flow graph subgraphs and segmentation of subgraphs adapted to heterogeneous computing resource capabilities.

2. The intelligent railway inspection method based on multi-mode data fusion according to claim 1 is characterized in that: The step 1 comprises the following steps: Step 1.1: Extract the features of the contact network morphology by using a sparse autoencoder network; Step 1.2: Extract features of contact network usage time by using sparse autoencoder network; Step 1.2: Extract the characteristics of contact network usage by using sparse autoencoder network; The step 1.1 comprises the following steps: Step 1.1.1: Construct the geometric topological structure formed by different types of features; Step 1.1.2: Analyze the unique embedded structure of multimodal features; Step 1.1.3: Feature selection and representation are achieved through multimodal complex heterogeneous feature sparseness.

3. The intelligent railway inspection method based on multi-mode data fusion according to claim 1 is characterized in that: The step 2 comprises the following steps: Step 2.1: Extract features from multiple input features respectively; Step 2.2: Perform Fourier encoding on the extracted features; Step 2.3: Splice the encoded features into the autoencoder neural network; Step 2.4: Get the common features after feature compression; Step 2.5: Add human intervention to reduce the error of feature compression process.

4. The intelligent railway inspection method based on multi-mode data fusion according to claim 1 is characterized in that: The step 6 comprises the following steps: Step 6.1: Add human intervention and add the possibility of human-predicted failures at certain time points on the timeline; Step 6.2: Take the fusion features and manual intervention features as input, conduct supervised learning of the neural network, analyze the fault location, and predict the fault probability.

5. The intelligent railway inspection method based on multi-mode data fusion according to claim 1 is characterized in that: The step 9 comprises the following steps: Step 9.1: Expand the data flow programming model and abstract a set of operations without data dependencies into data flow nodes. The weight of the node is the processing size of the intermediate data per unit time. The nodes are connected by directed edges, and the weight of the edge represents the size of the data migration dependency between tasks. Step 9.2: To express various business characteristics, two types of time attributes t and d are added to the edge. The former represents the shortest time interval for data arrival, and the latter represents the latest completion time of the corresponding operation of the node.

6. The intelligent railway inspection method based on multi-mode data fusion according to claim 1 is characterized in that: The step 10 comprises the following steps: Step 10.1: Perform sequence analysis and time-space estimation on the execution time of the data flow graph; Step 10.2: To optimize the layout of the distributed data flow graph, the dynamic data flow graph is parallelized to heterogeneous computing resources. The computing nodes where the tasks are finally placed are: ， Among them, c j Indicates the computing node where the final task is placed, c k represents all nodes, t k represents task k, T represents the task set, d ik Indicates t k Relative t i Generated input data.

7. The intelligent railway inspection method based on multi-mode data fusion according to claim 6 is characterized in that: The step 10.1 comprises the following steps: Step 10.1.1: Sort each task in the data flow graph in descending order of the total amount of dependent data, starting with the first task, and record the tasks as , determine whether the ratio of computing node load exceeds k times; Step 10.1.2: If it exceeds, select a computing node with lower load to place the task; Step 10.1.3: If there are no dependent tasks, place the current task directly on the computing node with lower load; Step 10.1.4: If There is a dependent branch, and the computing node locations of other tasks in the dependent branch are known. Place it on the computing node with the least amount of data migration; Step 10.1.5: Let the set of other tasks in the dependent branch be T = {t k }, and t k The placed computing nodes are denoted as , t k for The amount of input data generated is d ik , then t k Placed in The amount of data migration brought about by the above is: 。 8. An intelligent railway inspection system based on multi-mode data fusion, characterized in that: The intelligent railway inspection system based on multi-mode data fusion uses the intelligent railway inspection method based on multi-mode data fusion described in any one of claims 1 to 7.

Images