Track inspection data service system based on Hadoop
By adopting the Hadoop-based rail inspection data service system in the on-orbit inspection data management system, multiple problems in data management and analysis are solved, efficient data storage, classified management, distributed processing and macro analysis are realized, and the system's disaster recovery capabilities and data security are improved.
Patent Information
- Application Number
- CN202210648201.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-09
- Publication Date
- 2025-04-18
- Estimated Expiration
- 2042-06-09
AI Technical Summary
The existing track inspection data management system has problems such as efficient file storage and classification management, chaotic data hierarchical management, lack of scalability of application models, poor compatibility, insufficient data value mining, and weak disaster recovery capabilities.
A rail inspection data service system based on Hadoop is adopted, including a data acquisition module and a data processing module. The data processing module uses the data alignment unit to match the current track detection data and historical data with feature points and local interpolation compression, generates data alignment track detection data, and performs distributed processing and preprocessing through the data processing unit to generate data reports. The system also includes a database support module, file storage unit, analysis module, report generation module, data sharing module and data visualization module.
Through data alignment and distributed processing, the efficient utilization and classification management capabilities of rail inspection data can be improved, unified access and efficient query statistics can be achieved, the macro analysis and visualization capabilities of data can be enhanced, and the system's disaster recovery capabilities and data security can be improved.
Smart Images

Figure CN114936224B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data processing, and in particular to a track inspection data service system based on Hadoop. Background Art
[0002] "Internet +" has become a major strategy for the common development of various industries in my country. High-tech information technologies such as the Internet, big data, and cloud computing are developing rapidly. "Internet +" integrates these information technologies with various fields, making traditional businesses digital and online, and realizing the innovation and transformation of traditional industries.
[0003] In the past, the analysis and processing of track inspection data was basically based on a single machine or C / S architecture, which easily formed "information islands", and it was difficult to systematically classify and save basic data, resulting in incomplete and macroscopic analysis results. With the help of the current "Internet +" new technology, a large amount of track inspection data is collected through the Internet for systematic classification, storage, mining and analysis, breaking the "information island" barrier, forming a unified big data analysis cloud platform, and providing comprehensive and macroscopic data visualization analysis results. This has become the development direction and market hotspot of track inspection data mining and analysis.
[0004] The track inspection data service platform is a data platform that conforms to the current high-level development of social information technology and serves the reform of the track inspection system. Its main purpose is to strengthen track inspection analysis, establish a standardized management system, promote the sharing of track system data and the mutual cooperation between various businesses, provide efficient, accurate and reliable data basis for decision-making, and improve the predictability and pertinence of track maintenance work. There are some problems in the current management of track inspection information:
[0005] (1) It cannot meet the requirements of efficient storage and classification management of various files generated by various track inspection businesses. The compatibility of various types of information is poor, and unified access, efficient and accurate query and statistics cannot be achieved. This is a major drawback of current track inspection information.
[0006] (2) Various types of data are not managed in a hierarchical manner, and their use is also relatively chaotic. There is no unified data storage management center and data application platform;
[0007] (3) The application business and application model do not have good scalability, and the platform business functions cannot be improved in stages;
[0008] (4) Poor compatibility, unable to be well integrated with other information systems, and high system operation and maintenance costs;
[0009] (5) Insufficient mining of data value, unable to deeply explore and discover the patterns, rules and relationships hidden in massive data, unable to extract high-value-density knowledge from low-value-density data, and at a stage of "excessive data but insufficient information";
[0010] (6) There is no good disaster recovery capability, the data center's automatic instantaneous failover capability is weak, and data security cannot be guaranteed. Summary of the invention
[0011] Based on this, an object of the present invention is to provide a track inspection data service system based on Hadoop to at least solve the deficiencies in the above-mentioned technology.
[0012] The present invention proposes a track inspection data service system based on Hadoop, including a data acquisition module and a data processing module:
[0013] The data acquisition module is used to collect current track inspection data of each track inspection device;
[0014] The data processing module includes a data alignment unit and a data processing unit, wherein the data alignment unit is used to perform local interpolation and compression on each current track inspection data using historical track inspection data, so as to achieve feature point matching between the historical track inspection data and each current track inspection data, thereby obtaining data-aligned track inspection data;
[0015] The data processing unit is used to perform data preprocessing on the data-aligned track inspection data to obtain multiple groups of preprocessed track inspection data, and perform distributed processing on the multiple groups of preprocessed track inspection data to determine whether the data-aligned track inspection data meets the preset requirements. If the data-aligned track inspection data meets the preset requirements, a corresponding data report is generated to realize information interaction between the user and each of the track inspection devices.
[0016] Furthermore, the Hadoop-based track inspection data service system also includes a database support module, which is used to generate multiple track inspection ledger databases based on the track inspection data obtained by each track inspection device, and generate multiple permission management databases based on user permissions. The multiple track inspection ledger databases include at least a plane curve database, a vertical curve database, a CP3 database and a sleeper information database, and the multiple permission management databases include at least a specification management database.
[0017] Furthermore, the data processing module also includes a file storage unit, which is used to evenly distribute files smaller than a preset storage size according to the memory space they occupy, so that the merged files meet the preset data block maximum threshold.
[0018] Furthermore, the file storage unit is specifically used for:
[0019] When a file storage request is received, a data queue is initialized, and multiple temporary queues, multiple merge queues, and multiple file information mapping tables are created, wherein the number of the temporary queues is smaller than the number of the merge queues;
[0020] Obtaining a currently uploaded file, and comparing the storage size of the currently uploaded file with the remaining storage space of the temporary queue, if the remaining storage space of the temporary queue is greater than the storage size of the currently uploaded file, storing the currently uploaded file in the temporary queue, and checking whether the currently uploaded file exists in the file information mapping table;
[0021] If the currently uploaded file exists in the file information mapping table, determining whether the space occupied by all files in the temporary queue reaches a first threshold of a preset threshold space of the temporary queue;
[0022] If the space occupied by all files in the temporary queue reaches a first threshold of the preset threshold space of the temporary queue, all files in the temporary queue are merged until the preset threshold space of the temporary queue is reached, so as to obtain a merged queue corresponding to the temporary queue;
[0023] Find the file that occupies the largest storage space in the temporary queue, and determine whether the storage size of the file that occupies the largest storage space is greater than a second threshold of the preset threshold space of the temporary queue;
[0024] If the storage size of the file that occupies the largest storage space is greater than the second threshold of the preset threshold space of the temporary queue, the currently uploaded file will be stored in the merged queue corresponding to the temporary queue, and friendly files will be searched for in other temporary queues. All the friendly files found will be stored in the merged queue corresponding to the temporary queue for file packaging and uploading.
[0025] Furthermore, the file storage unit is also used for:
[0026] If the storage size of the file with the largest storage space is not greater than the second threshold of the preset threshold space of the temporary queue, continue uploading files to the temporary queue until the space occupied by all files in the temporary queue is greater than or equal to the second threshold of the preset threshold space of the temporary queue;
[0027] The currently uploaded file is stored in the merged queue corresponding to the temporary queue, and friendly files are searched in other temporary queues, and all the friendly files found are stored in the merged queue corresponding to the temporary queue for file packaging and uploading.
[0028] Furthermore, the data alignment unit is specifically used for:
[0029] Given two identical feature sequences with equal time intervals in the historical track inspection data and each of the current track inspection data, and calculating the distance between any two points of the two identical feature sequences to obtain a distance matrix of the two identical feature sequences;
[0030] The optimal planning path of the distance matrix of the two identical feature sequences is calculated using a dynamic programming method, and the minimum dynamic time warping distance of the two identical feature sequences is calculated based on the optimal planning path to align the data of the two identical feature sequences.
[0031] Furthermore, the Hadoop-based track inspection data service system also includes an analysis module, which is used to perform data analysis on the data collected by the data collection module, and the data analysis at least includes contour data analysis, corrugation data analysis, plate warpage analysis and weld straightness analysis.
[0032] Furthermore, the Hadoop-based track inspection data service system also includes a report generation module, which is used to convert the data output by each module into a corresponding data report for users to view.
[0033] Furthermore, the Hadoop-based track inspection data service system also includes a data sharing module, which is used to create data sharing rules and use the data sharing rules to provide a unified data distribution export service to each platform.
[0034] Furthermore, it also includes a data visualization module, which includes a first visualization module and a second visualization module. The first visualization module is used to connect the sampling data collected by the data acquisition module at different times to show the degree of change of the measurement signal of the data acquisition module with time; the second visualization module is used to integrate the data signal sampled by the data acquisition module to show the current signal strength of the data acquisition module.
[0035] Compared with the prior art, the beneficial effects of the present invention are as follows: the current track inspection data and the historical track inspection data are aligned with characteristic points as markers through the data alignment unit, thereby eliminating the interference of mileage errors for subsequent data analysis; the data-aligned track inspection data are split, and multiple groups of pre-processed track inspection data are processed in a distributed manner, thereby improving the efficient use and classified management of track inspection data, so that users can achieve unified access, efficient and accurate query and statistics. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1It is a structural block diagram of a track inspection data service system based on Hadoop in an embodiment of the present invention;
[0037] Figure 2 It is a structural block diagram of a data processing module in an embodiment of the present invention;
[0038] Figure 3 This is a model distribution diagram of a track inspection data service system based on Hadoop in an embodiment of the present invention;
[0039] Figure 4 The unified authentication and authority control in the embodiment of the present invention involves various database tables;
[0040] Figure 5 An organizational chart of a railway department in an embodiment of the present invention;
[0041] Figure 6 An example diagram of the star drawing method in an embodiment of the present invention;
[0042] Figure 7 It is an object-oriented class structure inheritance diagram in an embodiment of the present invention;
[0043] Figure 8 This is a schematic diagram of algorithm merging after optimization in an embodiment of the present invention;
[0044] Fig. 9 Schematic diagram of the overall architecture of the track inspection data service system in an embodiment of the present invention;
[0045] Fig.10 This is a technical architecture diagram of the track inspection data service system in an embodiment of the present invention;
[0046] Fig.11 This is a schematic diagram of authority management of a track inspection data service system according to an embodiment of the present invention;
[0047] Fig.12 This is an interface diagram of each service component of Cloudera Manager in an embodiment of the present invention;
[0048] Fig.13 This is a diagram of a user management interface of a track inspection data service system in an embodiment of the present invention;
[0049] Fig.14 This is a diagram of the mechanism management interface of the track inspection data service system in an embodiment of the present invention;
[0050] Fig.15 This is a diagram of a role management interface of a track inspection data service system in an embodiment of the present invention;
[0051] Fig.16 This is a menu management interface diagram of the track inspection data service system in an embodiment of the present invention;
[0052] Fig.17 This is a diagram of the SQL monitoring interface of the track inspection data service system in an embodiment of the present invention;
[0053] Fig.18 This is a diagram of the operation log interface of the track inspection data service system in an embodiment of the present invention;
[0054] Fig.19 This is a diagram of the error log interface of the track inspection data service system in an embodiment of the present invention.
[0055] Description of main component symbols:
[0056]
[0057]
[0058] The following specific implementation manner will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION
[0059] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Several embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.
[0060] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be a central element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a central element at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are for illustrative purposes only.
[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.
[0062] First of all, it should be noted that with the rapid development of railways, the massive data generated during the train maintenance process has not been effectively utilized. These data are of great significance to equipment self-diagnosis, track intelligent analysis, information fusion and data post-processing. The track inspection data service platform conforms to the current development level of information technology, and builds a track inspection data intelligent service platform based on big data analysis, cloud computing and other technologies to achieve efficient collection of various types of information in daily train operation and maintenance operations, interconnection of various types of maintenance equipment, and data sharing of various information platforms.
[0063] The present invention adopts Hadoop as the big data infrastructure to realize the construction of the track inspection data service system. In this application, the track inspection data service system includes the entry of daily maintenance data, the storage of maintenance report files, and the processing of maintenance data. Users can view the processed information on the track inspection data service system through mobile terminals such as PCs and mobile phones, including online viewing of various reports and real-time display of track inspection instrument maintenance work data.
[0064] Hadoop is an open source distributed computing platform on Apache Nutch. It not only provides users with a distributed infrastructure, but also makes the underlying details of the system very transparent. At the same time, since Hadoop is mainly developed in Java, Hadoop also inherits the excellent cross-platform features of Java. The most important thing is that Hadoop can reduce costs.
[0065] See also Figures 1 to 3 , shown is a track inspection data service system based on Hadoop in an embodiment of the present invention, including but not limited to a data acquisition module, a database support module, a data processing module, an analysis module, a report generation module, a data sharing module and a data visualization module. The data acquisition module is used to collect track inspection data of various track inspection devices, and the track inspection devices include but are not limited to track inspection instruments, track measuring instruments, corrugation instruments, structure inspection instruments, contact network monitoring instruments and numerical track gauges. The data acquisition module transfers the track inspection data uploaded by the above-mentioned track inspection devices to corresponding modules for corresponding analysis and processing, and classifies and saves the processed data so that users can view them through the corresponding user interface.
[0066] The Hadoop-based track inspection data service system in the present application also has an early warning function. When each module analyzes and processes its input data and detects that the input data is abnormal, it sends an early warning message in time to remind the operator to use the corresponding track inspection equipment to re-inspect.
[0067] Specifically, the data processing module includes a data alignment unit and a data processing unit, wherein the data alignment unit is used to perform local interpolation and compression on each current track inspection data using historical track inspection data, so as to achieve feature point matching between the historical track inspection data and each current track inspection data, thereby obtaining data-aligned track inspection data;
[0068] It should be noted that this embodiment adopts a dynamic warping algorithm, namely the DTW (Dynamic Time Warping) algorithm, uses the previous measurement results as a template, takes the data measured this time as the data to be aligned, and performs local interpolation and compression on the features of the data measured this time to achieve matching of feature points, thereby achieving high-precision alignment of the two measurement data.
[0069] Furthermore, the data alignment unit is specifically used for:
[0070] Given two identical feature sequences with equal time intervals in the historical track inspection data and each of the current track inspection data, and calculating the distance between any two points of the two identical feature sequences to obtain a distance matrix D of the two identical feature sequences;
[0071] In a specific implementation, two identical feature sequences with equal time intervals are given in the historical track inspection data and each current track inspection data, which are query sequences S={s1, s2, ..., s m}, matching sequence M = {m1, m2, ..., m n}, calculate the distance between any two points of the query sequence and the matching sequence according to the following formula:
[0072] d(i, j) = ||s i -m j || w ;
[0073] Wherein, i=1, 2, ..., m, j=1, 2, ..., n; when w=1, it is Manhattan distance; when w=2, it is Euclidean distance.
[0074] The resulting distance matrix is:
[0075]
[0076] Furthermore, the DP (dynamic programming) method is used to find the optimal planning path of the distance matrix D according to the following formula:
[0077] P best ={p1, p2, ..., p k ,…,p K};
[0078] Where p k Represents the path planning position, that is, pk =(i, j) k Indicates m With m n Aligned, where max(m,n)≤K≤m+n-1.
[0079] In order to ensure that all searched paths are meaningful, any searched path must meet the following constraints:
[0080] (1) Boundary: The starting point and end point of the shortest path are fixed, as shown in the following formula:
[0081]
[0082] (2) Monotonicity: In the same measurement, the feature of the previous moment cannot appear after the feature of the next moment after calculation, that is, when p is given k =(s i , m j ) and p k+1 =(s i′ , m j′ ), i′≤i+1, j′≤j+1.
[0083] p1 and p K The regular path between k , determined by constructing a cost matrix, the matrix element γ(i, j) is defined as:
[0084] γ(i,j)=d(i,j)+min[γ(i-1,j-1), γ(i-1,j), γ(i,j-1)];
[0085] In the formula, i∈{1,2,…,m}, j∈{1,2,…,n}, γ(0,0)=0, γ(i,0)=γ(0,j)=∞.
[0086] The optimal regular path P best The dynamic time warping distance DTW(S, M) should minimize the cumulative distance between S and M. The dynamic time warping distance is calculated as shown in the following formula:
[0087]
[0088] Furthermore, the data processing unit is used to perform data preprocessing on the data-aligned track inspection data to obtain multiple groups of preprocessed track inspection data, and to perform distributed processing on the multiple groups of preprocessed track inspection data to determine whether the data-aligned track inspection data meets the preset requirements; if the data-aligned track inspection data meets the preset requirements, a corresponding data report is generated to enable information interaction between the user and each of the track inspection devices.
[0089] In the specific implementation, the data processing unit splits the above-mentioned data-aligned track inspection data to obtain multiple groups of split track inspection data.<key,value> The track inspection data with the same intermediate result will be divided into the same processing sub-unit for processing, and it will be judged whether the track inspection data with the above data alignment meets the requirements of its corresponding processing sub-unit. If the track inspection data with the data alignment meets the requirements, it means that the track inspection data detected by the track inspection device is normal, that is, the track detected by the track inspection device is in a normal state. After the processing sub-unit processes the data, regardless of whether it meets the preset requirements, a corresponding data report will be generated to realize information interaction between the user and each track inspection device.
[0090] In some optional embodiments, the Hadoop-based track inspection data service system also includes a database support module, which is used to generate multiple track inspection ledger databases based on the track inspection data obtained by each track inspection device, and generate multiple permission management databases based on user permissions, the multiple track inspection ledger databases include at least a plane curve database, a vertical curve database, a CP3 database and a sleeper information database, and the multiple permission management databases include at least a specification management database.
[0091] The plane curve database and the vertical curve database are the relevant elements and attributes of the plane curve and the vertical curve that are statistically obtained based on the track inspection data obtained by each track inspection device. The information is as follows:
[0092] The main attributes involved in the plane curve are: intersection number, latitude (m), longitude (m) (or north coordinate m, east coordinate m, elevation m), turn, radius (m), slow length 1 (m), slow length 2 (m), tangent length (m), curve length (m), straight slow point (Km), slow straight point (Km), straight line length (m), super elevation (mm), track gauge widening (mm), breakpoint mileage (Km), corrected mileage (Km), straight line slope (m), total length along the slope (m), remarks, remarks 2. The plane curve database table structure is shown in Table 1 below.
[0093] Table 1
[0094]
[0095]
[0096] The main attributes of vertical curves are: serial number, mileage of slope change point (km), track surface elevation (m), slope rate (‰), slope length (m), radius (m), tangent length T1 (m), tangent length T2 (m), external arrow distance E (m), and remarks. The vertical curve database table structure is shown in Table 2 below.
[0097] Table 2
[0098]
[0099]
[0100] The element information of plane curves and vertical curves has strong structural characteristics, and the relational database MySQL is used to store and maintain the data in the background.
[0101] In addition to establishing corresponding data tables to store the plane curve and vertical curve information, in order to facilitate the system to index and maintain the information in these two data tables, it is also necessary to establish specific line information (unit department, line name, line type, etc.) to associate with it. There is a one-to-many relationship between the line information table and the plane curve and vertical curve tables. The line information database table structure is shown in Table 3 below.
[0102] Table 3
[0103] Table 3.3 Line information database table structure
[0104]
[0105]
[0106] This table 3 is associated with the system department table to form a many-to-one relationship. After establishing the corresponding data table to store the information, it is necessary to determine the corresponding operation function of the platform to complete the maintenance of the information.
[0107] Furthermore, the CP3 database contains CP3 coordinate information, and the related attributes involved in the CP3 coordinate information include: CP3 control point number, north coordinate (m), east coordinate (m), elevation (m), etc. The table structure of the CP3 coordinate information table database is shown in Table 4.
[0108] Table 4
[0109]
[0110] The main associated ones are the specific route information table and the control point detailed quantity table. Table 4 is associated with the CP3 control point detailed quantity table to form a many-to-one relationship. The specific route information table is consistent with Table 3. The CP3 control point detailed quantity table can obtain relevant attributes: buried point name, starting mileage, ending mileage, length (m), CPⅢ quantity (pairs), etc. The CP3 control point detailed quantity database table structure is shown in Table 5.
[0111] Table 5
[0112]
[0113] The platform also provides maintenance operation functions similar to those of the plane curve and vertical curve service modules, and supports batch import functions if the specified templates are met.
[0114] Furthermore, the relevant attributes in the sleeper information data table include: design mileage, sleeper number, fastener type, fastener parameters (left, right, inside, outside, height adjustment), line characteristics (bridge, culvert, tunnel, etc.), limit, etc.; at the same time, this information is also associated with specific line information.
[0115] The range of fastener types is: WJ-7(A), WJ-7(B), WJ-8(A), WJ-8(B), SFC (in-line), SFC (staggered), 300-1a, 300-1u, etc. The table structure of the sleeper list information database is shown in Table 6.
[0116] Table 6
[0117]
[0118]
[0119] The information in table 6 forms a many-to-one relationship with the specific line information table through the foreign key line_id. The system also provides maintenance operation functions and supports batch import functions when the specified template is met.
[0120] In this embodiment, the authority management database also includes a device authorization code management database, wherein the relevant attributes in the device authorization code management database include: applicant user, authorization time, device model, device name, authorized computer model, authorized computer serial number (or machine code, etc.), authorization code, deregistered user, deregister time, deregister flag, etc. At the same time, this information is also associated with the system department table to form a many-to-one relationship to specify the ownership of the device authorization code, which is convenient for future data query and statistics. The table structure of the device authorization code management database is shown in Table 7.
[0121] Table 7
[0122]
[0123]
[0124] The above data table 7 sets additional information such as the applicant user, application time, deregistered user, deregistration time, etc., and combined with the system log information, it is convenient for the administrator to grasp the status of the device authorization code from application, use to deregistration at any time.
[0125] The standard management database includes a management system (web) and a portal website (web). Multiple systems share a service backend, which uses the Shiro framework for unified login authentication and authority control. Unified authentication and authority control involve database tables such as Figure 4 shown.
[0126] against Figure 4 The description of each table is shown in Table 8:
[0127] Table 8
[0128]
[0129]
[0130] Furthermore, the Hadoop-based track inspection data service system also includes an analysis module, which is used to perform data analysis on the data collected by the data acquisition module, and the data analysis at least includes contour data analysis, corrugation data analysis, plate warpage analysis and weld straightness analysis; the Hadoop-based track inspection data service system also includes a report generation module, which is used to convert the data output by each module into corresponding data reports for users to view.
[0131] It should be noted that during the operation of the system, the operating status of the entire system is monitored in real time, and the errors generated during the operation are saved to the database error log table through the report generation module, which is convenient for administrators to find platform bugs. The structure of the error log data table is shown in Table 9.
[0132] Table 9
[0133]
[0134] It should be understood that when the system is running, there will be system logs to record the operations of all logged-in users of the entire system, so that analysts can timely grasp the system's high-visit pages and operations, the time of occurrence, and some system vulnerabilities, and developers can optimize the system based on the analysis results. The system log data table structure is shown in Table 10.
[0135] Table 10
[0136]
[0137]
[0138] The standard management database also includes organization management, where the organizational structure of the railway department (taking Nanchang Railway Bureau as an example) is as follows: Figure 5 As shown:
[0139] Each railway bureau is subordinate to China Railway Corporation, and each section has corresponding workshops and work area departments. The track inspection data service platform adopts a tree structure for organizational management. The organizational management data table structure is shown in Table 11.
[0140] Table 11
[0141]
[0142] In this embodiment, the Hadoop-based track inspection data service system also includes a data visualization module, which includes a first visualization module and a second visualization module. The first visualization module is used to connect the sampled data collected by the data acquisition module at different times to show the degree of change of the measurement signal of the data acquisition module with time; the second visualization module is used to integrate the data signal sampled by the data acquisition module to show the current signal strength of the data acquisition module.
[0143] It should be noted that the track inspection data service system mainly applies three types of applications: multidimensional data visualization, hierarchical information visualization and time series data visualization.
[0144] 1. Multidimensional data visualization
[0145] Multidimensional data visualization is a way to display high-dimensional data in a two-dimensional plane. Traditional algorithms are complex, time-consuming and have poor compatibility. Multidimensional data visualization can be divided into geometry-based visualization methods and icon-based visualization methods based on data features.
[0146] The geometry-based visualization method can use parallel vertical curves to represent different dimensions, depict the values of multidimensional data on the coordinate axis and connect the coordinate points on the axis to display the multidimensional data in two-dimensional space.
[0147] Icon-based visualization methods mainly use geometric shapes as icons to depict multidimensional data. The characteristic attributes of icons reflect the dimension of information, and the connection between icons and multidimensional data is used to reflect the visualization effect. The representative method of icon-based visualization methods is star drawing, which maps the information dimension from point to line. The length of the line segment represents the size of the value, such as Figure 6 shown.
[0148] 2. Visualization of hierarchical information
[0149] The inheritance of computer files and Java classes is a very typical hierarchical information. Figure 7 This type of hierarchical information has very obvious structural characteristics. The visualization of hierarchical information is mainly achieved through node connections.
[0150] Node connections are mainly used to draw different nodes to represent the data content of information, and the lines between nodes represent the relationship between data. Figure 7 In the figure, each class name represents each node, the blue solid line represents the actual inherited parent class, and the green dotted line represents the parent interface implemented by the class.
[0151] 3. Time Series Data Visualization
[0152] Time series visualization is the visualization of data sampled over time. There are three main ways to display it: line chart, stacked chart, and horizon chart.
[0153] Line graph: By connecting the sampling values at different times, you can intuitively show the degree to which the measured signal changes over time.
[0154] Stacked graph: The signal sampled over time is integrated and the area is calculated. This visualization method can well display the magnitude of the current signal. However, when the signal is negative, the visualization effect will be greatly reduced.
[0155] Horizon chart: The horizon chart is a differential processing of the measured signal. In the horizon chart, the signal numbering rate over time can be clearly observed, and the depth of color represents the effect of the change.
[0156] In other optional embodiments, the data visualization module dynamically customizes and finely controls the menus and function buttons of the system, providing a standard expansion method for the function expansion of the system. The menu data table structure is shown in Table 12.
[0157] Table 12
[0158]
[0159]
[0160] Among them, the menu URL types are: 1. Ordinary pages (such as user management, / sys / user) 2. Nested complete external pages, links starting with http(s) 3. Nested server pages, using the iframe: prefix + target URL (such as SQL monitoring, iframe: / druid / login.html, the iframe: prefix will be replaced with the server address).
[0161] Specifically, in the visual interface of the data visualization module, the system roles support unlimited creation and expansion, and specific system service menus and functions can be dynamically assigned to any role, which determines the specific service and function range of the system. The role data table structure is shown in Table 13.
[0162] Table 13
[0163]
[0164]
[0165] In this system, there should be a many-to-many relationship between roles and departments (institutions). When a department (institution) is assigned a corresponding role, all personnel under the department (institution) will have the platform services and functions owned by the role. It is unnecessary to bind each person to a role one by one. The structure of the table of corresponding relationships between roles and institutions is shown in Table 14.
[0166] Table 14
[0167]
[0168] It should be understood that, similarly, in this system, there should be a many-to-many relationship between roles and menus (function buttons). The same role has the right to use multiple menus (function buttons), and a menu (function button) can be owned by different roles at the same time. The structure of the table of corresponding relationships between roles and menus is shown in Table 15.
[0169] Table 15
[0170]
[0171] In this embodiment, the system encrypts the user's sensitive information, and the system super administrator can disable and enable the user. There should be a many-to-one relationship between users and departments (institutions). A department contains multiple users, and a user can only belong to a department. The structure of the user information data table is shown in Table 16.
[0172] Table 16
[0173]
[0174] In this system, there should also be a many-to-many relationship between users and roles. The same user can have multiple roles, and a role can be owned by different users at the same time. The roles owned by a user are the collection of the roles corresponding to the user and the roles corresponding to the department (institution). The structure of the user-role correspondence table is shown in Table 17.
[0175] Table 17
[0176]
[0177] In this embodiment, after a user successfully logs into the system, the platform will return a string of access codes (the access code is unique and has an expiration time limit. If there are special control requirements, it is possible to consider adding IP or MAC fields for combined access control). The user must carry the unique access code in the subsequent access process of platform services and functions, otherwise the platform will reject the access request. The user token table structure is shown in Table 18:
[0178] Table 18
[0179]
[0180] Furthermore, in order to overcome the problem of wasting space in common storage methods and make better use of space, the track inspection data service system adopts a small file storage method based on space optimization. The data processing unit also includes a file storage unit, which stores files smaller than the preset storage size (5MB) as small files. The implementation process of the algorithm is as follows: Figure 8 As shown. Evenly distribute the small files according to the size of the memory space they occupy, so that the merged file can reach the set data block maximum threshold. Compared with the simple algorithm, the advantage of this algorithm is that it can better utilize the space of each data block and reduce storage overhead;
[0181] Specifically, the file storage unit is specifically used for:
[0182] When a file storage request is received, a data queue is initialized, and multiple temporary queues, multiple merge queues, and multiple file information mapping tables are created, wherein the number of the temporary queues is smaller than the number of the merge queues;
[0183] Obtaining a currently uploaded file, and comparing the storage size of the currently uploaded file with the remaining storage space of the temporary queue, if the remaining storage space of the temporary queue is greater than the storage size of the currently uploaded file, storing the currently uploaded file in the temporary queue, and checking whether the currently uploaded file exists in the file information mapping table;
[0184] If the currently uploaded file exists in the file information mapping table, determining whether the space occupied by all files in the temporary queue reaches a first threshold of a preset threshold space of the temporary queue;
[0185] If the space occupied by all files in the temporary queue reaches a first threshold of the preset threshold space of the temporary queue, all files in the temporary queue are merged until the preset threshold space of the temporary queue is reached, so as to obtain a merged queue corresponding to the temporary queue;
[0186] Find the file that occupies the largest storage space in the temporary queue, and determine whether the storage size of the file that occupies the largest storage space is greater than a second threshold of the preset threshold space of the temporary queue;
[0187] If the storage size of the file with the largest storage space is larger than the second threshold of the preset threshold space of the temporary queue, the currently uploaded file is stored in the merged queue corresponding to the temporary queue, and friendly files are searched in other temporary queues, and all the friendly files found are stored in the merged queue corresponding to the temporary queue for file packaging and uploading;
[0188] If the storage size of the file with the largest storage space is not greater than the second threshold of the preset threshold space of the temporary queue, continue uploading files to the temporary queue until the space occupied by all files in the temporary queue is greater than or equal to the second threshold of the preset threshold space of the temporary queue;
[0189] The currently uploaded file is stored in the merged queue corresponding to the temporary queue, and friendly files are searched in other temporary queues, and all the friendly files found are stored in the merged queue corresponding to the temporary queue for file packaging and uploading.
[0190] In the specific implementation, the specific steps of the algorithm are described as follows:
[0191] (1) After receiving the file storage request, the data queue is initialized, and the configuration information including the merge threshold in the configuration file is read to create a temporary queue, a merge queue and a file information mapping table. The number of temporary queues is less than the number of merge queues.
[0192] (2) Compare the size of the incoming file with the remaining space in the temporary queue. If the remaining space in the temporary queue is larger than the size of the space occupied by the incoming file, store the queue. Otherwise, compare the file size with other temporary queues. If no temporary queue meets the requirements after all temporary queues are compared, create a new temporary queue and store the file in it.
[0193] (3) Check whether there is a currently uploaded file in the file information mapping table. If so, merge the files. If not, record the file information in the file information mapping table.
[0194] (4) Determine whether the space occupied by the files in the temporary queue has reached 90% of the set threshold space. If so, the files in this temporary queue begin to be merged.
[0195] (5) When the temporary queue reaches the threshold, find the file that occupies the largest space. If the size of the file is greater than 1 / 2 of the threshold, go to step 6; otherwise, go to step 7.
[0196] (6) Store the file in the merge queue and search for friendly files in the remaining temporary queues, that is, files with the most suitable space to be filled into the current merge queue, until no friendly files are found. Go to step 8.
[0197] (7) The queue where the file is located does not change. When the total space of the queue is greater than or equal to 1 / 2 of the threshold, return to step 6.
[0198] (8) Package the files in the merge queue and upload them to HDFS.
[0199] Furthermore, in this embodiment, the track inspection data service system also includes a data sharing module, and its data transmission link security measures mainly use technical means such as HTTPS to build an encrypted transmission link, and directly encrypt the data and transmit it in ciphertext to ensure the security of the data transmission process;
[0200] In addition, the goal of data security protection is to ensure that data is accessed and processed within the authorized scope, and to prevent data from being stolen, leaked, or damaged. In addition to network security protection technical measures such as dual-machine hot backup, firewall, intrusion detection, anti-virus, anti-DDoS attack, and vulnerability detection, it also includes:
[0201] (1) Account authority management
[0202] Establish a unified account authority management subsystem to achieve unified management of accounts and access scopes of various business systems, ensure that data is used within the authorized scope, and implement account authority management and approval systems.
[0203] (2) Data desensitization
[0204] From the perspective of protecting the confidentiality of sensitive data, when displaying data, the platform supports blurring of sensitive data according to specific business needs.
[0205] (3) Log management and auditing
[0206] The technical capability requirements for log management and auditing are mainly to record and audit account management operation logs, permission approval logs, data access operation logs, etc., to assist in the implementation of relevant management systems. In terms of technical implementation, a unified log management and auditing subsystem is built according to business needs.
[0207] (4) Real-time monitoring of abnormal operations
[0208] Compared with post-event security technical measures such as log recording and security auditing, real-time monitoring of abnormal behavior is a necessary measure to achieve monitoring, early warning and real-time disposal in the "before" and "during" stages. The abnormal behavior monitoring system should be able to monitor dangerous behaviors such as unauthorized access to data and sensitive operations on data files in real time.
[0209] The data sharing process involves providing data services to third parties, establishing data sharing security-related management system regulations, and providing a unified data distribution export service for the platform to effectively manage data sharing behaviors and prevent data theft, leakage and other security risks. Data distribution export services need to be designed according to data sharing business needs and relevant data standards.
[0210] Regarding the destruction of system data, in principle, physical deletion of data is not performed, provided that there is sufficient storage space. Instead, the data is marked for deletion.
[0211] See also Fig. 9 , shown is the overall architecture of the track inspection data service system in the present invention. Based on the Hadoop distributed processing architecture, the track inspection data service system adopts a hierarchical bottom-up system design, which is divided into five levels: data layer, analysis layer, integration layer, business layer, and visualization layer.
[0212] 1. Data layer
[0213] The data layer is the basic layer of the entire track inspection data service system. It is the source and guarantee of data resources, including structured data and unstructured data.
[0214] Structural data refers to data that can be represented and stored using a relational database, and is presented as two-dimensional data. The structural data involved in the system include the original data of equipment such as track detectors and corrugators, and the positioning data of Beidou satellites. These data are stored in the MySQL relational database.
[0215] Unstructured data refers to data without a regular structure. Various reports, images, and video / audio data are all unstructured data. The unstructured data involved in the system mainly include various reports generated by track inspection instruments, image data collected by patrol inspection instruments, and audio / video data collected by helmet equipment. For the storage of unstructured data, these data are stored in HDFS as files, and the links or paths to the files are stored in the MySQL relational database.
[0216] 2. Analysis layer
[0217] The establishment of the analysis layer is to solve the current problems of large amount of track maintenance data, scattered data, and data that cannot be analyzed and applied. First, in the integration and cloud storage stage, data cleaning work is done on various types of unprocessed raw data. Then, the actual needs of users are analyzed in detail, the data from the data layer is taken out for analysis and packaging, and then transmitted to the data analysis layer through Sqoop, where the data is parsed based on actual needs. At this point, the processed data has a certain readability. The data analysis layer will encapsulate this data into JSON format and pass it to the visualization layer for data display.
[0218] 3. Integration layer
[0219] The role of the integration layer is to establish a unified standard support module, integrate and implement the effective integration of related application components (historical data support, intelligent analysis support, information fusion support, post-processing support, satellite positioning support, etc.). Based on these support components, the corresponding functional modules can be quickly built at the business layer.
[0220] 4. Business layer
[0221] The business layer directly displays the various functions of the track inspection data service system and implements various applications based on the integration layer.
[0222] The business layer strictly implements the classification standards of various applications. The development and management of the overall application are operated under a unified management model. Data between application modules can be shared. The system provides a unified data interface, providing a richer data source for business implementation.
[0223] 5. Visualization layer
[0224] Various data analyses of the track inspection data service system can be displayed through a Web website. Based on the need for data security, staff with different job identities can obtain corresponding resources on the platform according to their own permissions after logging into the system. They can also manage resources, such as importing, sharing, and deleting.
[0225] Furthermore, the track inspection data service system carries the subsystems of various businesses (track inspection instrument analysis, track inspection data management, profile data analysis, corrugation data analysis, three-dimensional constraints, single skid operation analysis, etc.). Considering the unified and distributed deployment characteristics of the architecture of each subsystem, a microservice architecture solution is adopted for the architecture of each subsystem. Fig.10 shown.
[0226] In the track inspection data service system architecture, the Hadoop distributed computing framework is built by the server itself, and on this basis, the distributed scheduling framework and specific microservices (modules that provide business logic services) are independently developed. Among them, the service registration center (Eureka Server), service configuration center (Spring Cloud Config), service monitoring center (SpringAdmin), message bus (Rabbit MQ), service API gateway (Spring Cloud Zuul), single point service login, load balancing (Ribbon, Nginx) and other functions can be reused by various subsystems to form a unified distributed system scheduling architecture, which can unify the management and resource scheduling of various internal microservices.
[0227] When designing each subsystem, there is no need to consider the system structure. Instead, it is only necessary to perform logical division of the business according to its own business characteristics under the distributed system framework, design service modules that meet the needs, and deploy them. The software developed in this mode is more consistent, more scalable, and less expensive, which can greatly shorten the system development time.
[0228] Furthermore, the rail inspection data service system has a lot of internal sensitive data. In view of the security requirements of the current network environment, the rail inspection data service system fully considers the security issues of Web application authentication and authorization. In the process of architecture, the permission management framework has been fully planned. According to the user's identity, department, position and other factors, different system permissions are set to grant different permissions for data addition, deletion, modification and query and command issuance.
[0229] At the same time, sensitive resources within the system are fully protected, and all types of sensitive data will be encrypted and watermarked to prevent data leakage. Fig.11 Shown
[0230] In this application, the Hadoop-based track inspection data service system uses three servers to build a test cluster with three nodes. The configuration of each server is shown in Table 19:
[0231] Table 19
[0232]
[0233] The track inspection data service system uses CDH (Cloudera's Distribution Including ApacheHadoop) for offline deployment. CDH integrates functions such as automated cluster installation, fault monitoring, and email alarms, effectively reducing the installation cost of the cluster, while also reducing the subsequent maintenance cost and improving management efficiency. The specific steps for cluster deployment are as follows:
[0234] (1) Plan each server. The server IP and name are shown in Table 20.
[0235] Table 20
[0236]
[0237] (2) Modify the host name and configure the hosts file of each server. Configure cm-server so that it can log in to other nodes without a password. Secondly, install mariadb on cm-server for later data storage. Then configure the Java environment and finally configure the server requirements of each node.
[0238] (3) Next, install Cloudera Manager. First, download the compression-related software packages, create users and initialize data, then distribute the files modified by cm-server to each node, then create a local source, and finally start the Server and Agent services on cm-server, and start the Agent service on other nodes. After all services are started, you can log in to Cloudera Manager by accessing port 7180 of cm-server.
[0239] Then you can add the corresponding components as needed. In the component service, you can divide the service roles to achieve load balancing. This cluster uses components such as HDFS, Yarn, Zookeeper, Hive, and Flume. The component interface is as follows: Fig.12 shown.
[0240] The track inspection data service system grants login permissions based on the user's department, position and role. The setting of permissions is based on the user's identity and manual verification by the super administrator.
[0241] User authentication mainly involves searching and comparing the information entered by the user requesting login with the information in the user information data table. If the user exists and the current account is available, the user login is considered legitimate. The information recorded in the user information data table is the detailed information of the authorized users who can access the rail inspection data platform, and is the most important table in user management.
[0242] The most important attributes in the user information table are the user name (name), password (password) and salt (salt). The password is encrypted and stored in the user information table with salt to ensure the safety of the user. This setting is to prevent unauthorized or illegal users from logging into the system and causing system damage or data leakage. Only when the user name and password are correct can you enter the track inspection data service platform, otherwise, you will always stay on the login interface. It should be emphasized here that the user password must contain uppercase and lowercase letters, numbers and special symbols to be qualified. After logging in, users can query the corresponding content in the platform according to their permissions.
[0243] Furthermore, according to the demand analysis of the track inspection data service platform in Chapter 3, the platform implements user management, organization management, role management, menu management, SQL monitoring, interface documentation, system log and other functions for track inspection data management.
[0244] In this embodiment, the system has user management functions, organization management functions, role management and menu management. The super administrator can implement the user addition, deletion, modification and query functions in the user management interface. The legal users created by the super administrator can log in to the rail inspection data service platform to query data. Fig.13 shown.
[0245] On the organization management page, the administrator can create various organizations according to the actual organizational structure of the company, and grant different permissions to the staff of different departments of different organizations through the organization, so that the staff can obtain the information within their authority on the rail inspection data service system according to their own authority. Fig.14 shown.
[0246] The roles in the system can be freely expanded without touching the maximum capacity of the database. The specific service menus and functions of the system can be dynamically assigned to any role. The role determines the scope and services of the platform that the user can use. Fig.15 shown.
[0247] You can add extended functions to the platform through menu management. The menu management interface is as follows: Fig.16 As shown in the figure, adding in menu management is a means of extending the system standard. Unifying the standard process can reduce the irregular definitions that appear during system extension and improve the efficiency of subsequent system function development.
[0248] In this embodiment, the system has a SQL monitoring function, which can detect abnormal behavior in the system. For example, when the query speed is very slow, the SQL monitoring report can be analyzed to analyze which service has a problem, whether it is a user query error or the SQL statement is written without considering the indexing problem during development. Fig.17 shown.
[0249] At the same time, this monitoring module not only has the function of SQL monitoring, it also has the functions of monitoring Web applications, URI monitoring, Session monitoring and Spring monitoring, and can perform real-time detection of dangerous behaviors such as unauthorized access, sensitive operations on data files, etc.
[0250] It should be noted that any user who uses any function of the system will leave an operation log, which contains the user's IP, the URL the user accessed, the status code of whether the user's access was successful, and the time the user accessed. With this information, the administrator can effectively understand the situation of each user using the system. The operation log interface is as follows: Fig.18 shown.
[0251] By analyzing the operation logs, we can find out which functions of the system are frequently used and which are rarely used. After the analysis, we can reasonably call the platform resources and give more resources to the frequently used services, thereby improving the efficiency of server use and reasonably allocating server resources.
[0252] At the same time, if the system encounters an unexpected problem, such as data loss, you can restore part of the data through the operation log, improve the disaster recovery of the platform, and reduce losses when accidents occur.
[0253] Whenever an error occurs in the system, the error information will be recorded. The backend will record the error information in the error log database and display it on the front-end page. These error logs help administrators to troubleshoot bugs in the track inspection data service system, effectively improve the efficiency of finding and handling bugs, and promote the long-term and stable operation of the platform. Fig.19 Shown
[0254] In summary, the Hadoop-based track inspection data service system in the above embodiment of the present invention uses a data alignment unit to align the current track inspection data with the historical track inspection data using feature points as markers, thereby eliminating mileage error interference for subsequent data analysis; by splitting the data-aligned track inspection data and performing distributed processing on multiple groups of pre-processed track inspection data, the efficient use and classified management of the track inspection data are improved, so that users can achieve unified access, efficient accurate query and statistics.
[0255] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0256] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
Claims
1. A track inspection data service system based on Hadoop, characterized in that: Including data acquisition module and data processing module: The data acquisition module is used to collect current track inspection data of each track inspection device; The data processing module includes a data alignment unit and a data processing unit. The data alignment unit is used to use the historical track inspection data to perform local interpolation and compression on each current track inspection data, so as to achieve feature point matching between the historical track inspection data and each current track inspection data, thereby obtaining data-aligned track inspection data. The data alignment unit is specifically used to: Given two identical feature sequences with equal time intervals in the historical track inspection data and each of the current track inspection data, and calculating the distance between any two points of the two identical feature sequences to obtain a distance matrix of the two identical feature sequences; Calculating the optimal planning path of the distance matrix of the two identical feature sequences by using a dynamic programming method, and calculating the minimum dynamic time warping distance of the two identical feature sequences according to the optimal planning path, so as to align the data of the two identical feature sequences; The data processing unit is used to perform data preprocessing on the data-aligned track inspection data to obtain multiple groups of preprocessed track inspection data, and perform distributed processing on the multiple groups of preprocessed track inspection data to determine whether the data-aligned track inspection data meets the preset requirements. If the data-aligned track inspection data meets the preset requirements, a corresponding data report is generated to realize information interaction between the user and each of the track inspection devices.
2. The Hadoop-based track inspection data service system according to claim 1, characterized in that: The Hadoop-based track inspection data service system also includes a database support module, which is used to generate multiple track inspection ledger databases based on the track inspection data obtained by each track inspection device, and generate multiple authority management databases based on user authority. The multiple track inspection ledger databases include at least a plane curve database, a vertical curve database, a CP3 database and a sleeper information database, and the multiple authority management databases include at least a specification management database.
3. The track inspection data service system based on Hadoop according to claim 1, characterized in that: The data processing module also includes a file storage unit, which is used to evenly distribute files smaller than a preset storage size according to the memory space they occupy, so that the merged files meet the preset data block maximum threshold.
4. The Hadoop-based track inspection data service system according to claim 3, characterized in that: The file storage unit is specifically used for: When a file storage request is received, a data queue is initialized, and multiple temporary queues, multiple merge queues, and multiple file information mapping tables are created, wherein the number of the temporary queues is smaller than the number of the merge queues; Obtaining a currently uploaded file, and comparing the storage size of the currently uploaded file with the remaining storage space of the temporary queue, if the remaining storage space of the temporary queue is greater than the storage size of the currently uploaded file, storing the currently uploaded file in the temporary queue, and checking whether the currently uploaded file exists in the file information mapping table; If the currently uploaded file exists in the file information mapping table, determining whether the space occupied by all files in the temporary queue reaches a first threshold of a preset threshold space of the temporary queue; If the space occupied by all files in the temporary queue reaches a first threshold of the preset threshold space of the temporary queue, all files in the temporary queue are merged until the preset threshold space of the temporary queue is reached, so as to obtain a merged queue corresponding to the temporary queue; Find the file that occupies the largest storage space in the temporary queue, and determine whether the storage size of the file that occupies the largest storage space is greater than a second threshold of the preset threshold space of the temporary queue; If the storage size of the file that occupies the largest storage space is greater than the second threshold of the preset threshold space of the temporary queue, the currently uploaded file will be stored in the merged queue corresponding to the temporary queue, and friendly files will be searched for in other temporary queues. All the friendly files found will be stored in the merged queue corresponding to the temporary queue for file packaging and uploading.
5. The Hadoop-based track inspection data service system according to claim 4, characterized in that: The file storage unit is also used for: If the storage size of the file with the largest storage space is not greater than the second threshold of the preset threshold space of the temporary queue, continue uploading files to the temporary queue until the space occupied by all files in the temporary queue is greater than or equal to the second threshold of the preset threshold space of the temporary queue; The currently uploaded file is stored in the merged queue corresponding to the temporary queue, and friendly files are searched in other temporary queues, and all the friendly files found are stored in the merged queue corresponding to the temporary queue for file packaging and uploading.
6. The track inspection data service system based on Hadoop according to claim 1, characterized in that: The Hadoop-based track inspection data service system also includes an analysis module, which is used to perform data analysis on the data collected by the data collection module. The data analysis at least includes contour data analysis, corrugation data analysis, plate warpage analysis, and weld straightness analysis.
7. The track inspection data service system based on Hadoop according to claim 1, characterized in that: The Hadoop-based track inspection data service system further includes a report generation module, which is used to convert the data output by each module into a corresponding data report so that the user can view it.
8. The Hadoop-based track inspection data service system according to claim 1, characterized in that: The Hadoop-based track inspection data service system further includes a data sharing module, which is used to create data sharing rules and provide a unified data distribution export service to each platform using the data sharing rules.
9. The track inspection data service system based on Hadoop according to claim 1, characterized in that: It also includes a data visualization module, the data visualization module includes a first visualization module and a second visualization module, the first visualization module is used to connect the sampled data collected by the data acquisition module at different times to show the degree of change of the measurement signal of the data acquisition module with time; The second visualization module is used to perform integration processing on the data signal sampled by the data acquisition module to display the current signal strength of the data acquisition module.
Citation Information
Patent Citations
An enterprise behavior analysis system based on big data
CN109272155A
Subway track dynamic detection data management and analysis system and method
CN112015782A