Intelligent traffic big data lake

Abstract

The invention provides an intelligent traffic big data lake. The intelligent traffic big data lake comprises a center node and at least one edge node, one edge node corresponds to one partition; the edge nodes are connected with the center node, and the edge nodes are used for collecting first original data in the corresponding partitions, performing first preprocessing on the first original data to obtain first credible area data, and performing theme-oriented analysis on the first credible area data to obtain first value area data; the central node is used for collecting, storing and processing second original data; aiming at mass traffic big data with the characteristics of large scale, multiple types, low value density, high speed and the like, the invention provides a region-level logic-unified and physically-dispersed intelligent traffic big data lake, through a central node + edge node architecture, the requirement of strong real-time processing of the traffic big data is met, and the real-time performance of the traffic big data is improved. Through hierarchical storage of the center node and the edge nodes, rapid processing, analysis and mining of mass traffic big data are realized.

Description

Technical Field

[0001] The present invention relates to the field of data processing, in particular to traffic big data, and specifically to a smart traffic big data lake. Background Art

[0002] Traffic big data is a large amount of multi-source, multi-modal, heterogeneous data generated by the "ternary space" of information space, physical world and human society in the process of traffic management, life, construction and development. It contains rich knowledge and value, and has the characteristics of large scale, wide variety, low value density and fast speed.

[0003] Through searching, we found that the current focus of processing traffic big data is mainly on 1) coordinated scheduling and processing of stream data and batch data; 2) data transmission methods; 3) data optimization and storage methods; 4) providing accurate bus schedules that are more suitable for passenger travel arrangements; 5) smart traffic big data analysis systems including interconnected data middle platforms, multiple edge computing subsystems and multiple smart road monitoring subsystems. However, there is a lack of a regional-level logically unified and physically dispersed smart traffic big data lake architecture, which makes it impossible to achieve unified processing of regional-level traffic big data. Summary of the Invention

[0004] The purpose of the present invention is to provide a smart transportation big data lake to solve the problems pointed out in the above background technology.

[0005] The present invention provides a smart transportation big data lake, which includes: a central node and at least one edge node; wherein, one edge node corresponds to a partition; the edge node is connected to the central node, and the edge node is used to collect first original data in the corresponding partition, and to perform first preprocessing on the first original data to obtain first trusted zone data, and to perform subject-oriented analysis on the first trusted zone data to obtain first value zone data, and to send any one or two or more combinations of the first original data, the first trusted zone data and the first value zone data to the central node as needed; the central node is used to collect, store and process second original data in the area where it is located.

[0006] In this invention, a regional-level logically unified and physically decentralized smart transportation big data lake is provided for massive traffic big data with the characteristics of large scale, multiple types, low value density and high speed. Through the central node + edge node architecture, the requirements for strong real-time processing of traffic big data are met. Through the hierarchical storage of central nodes and edge nodes, the rapid processing, analysis and mining of massive traffic big data are realized.

[0007] In one implementation of the present invention, the edge node includes: a first original area, a first trusted area and a first value area; wherein, the first original area is connected to the central node and the first trusted area respectively, the first original area is used to collect and store the first original data, and to send the first original data to the first trusted area, and to send the first original data to the central node on demand; the first trusted area is also connected to the central node and the first value area, the first trusted area is used to receive the first original data, and to perform the first preprocessing on the first original data to obtain and store the first trusted area data, and send the first trusted area data to the first value area, and to send the first trusted area data to the central node on demand; the first value area is also connected to the central node, the first value area is used to receive the first trusted area data, and to perform subject-oriented analysis on the first trusted area data to obtain and store the first value area data, and to send the first value area data to the central node on demand.

[0008] In this implementation, the data on the edge nodes is partitioned and stored for easy management, enabling rapid processing, analysis, and mining of massive amounts of data.

[0009] In one implementation of the present invention, the central node includes: a data storage sub-platform, a data analysis sub-platform, a data capability opening sub-platform, a big data support sub-platform, a unified data governance sub-platform and a unified operation and maintenance management sub-platform; wherein, the data storage sub-platform is connected to the edge node, and the data storage sub-platform is used to collect and store the second original data, and to receive on demand any one or two or more combinations of the first original data, the first trusted zone data and the first value zone data; the unified data governance sub-platform is connected to the data storage sub-platform, and the unified data governance sub-platform is used to extract data from the data storage sub-platform on demand; The data analysis sub-platform is connected to the unified data governance sub-platform, and the data analysis sub-platform is used to provide computing for the unified data governance sub-platform and upper-level traffic applications; the big data support sub-platform is respectively connected to the unified data governance sub-platform, the data analysis sub-platform, the data storage sub-platform and the edge node; the data capability opening sub-platform is respectively connected to the big data support sub-platform, the data analysis sub-platform and the unified data governance sub-platform; the unified operation and maintenance management sub-platform is respectively connected to the data storage sub-platform, the big data support sub-platform, the data analysis sub-platform, the unified data governance sub-platform and the data capability opening sub-platform.

[0010] In this implementation method, through the setting of data storage sub-platform, data analysis sub-platform, data capability opening sub-platform, big data support sub-platform, unified data governance sub-platform and unified operation and maintenance management sub-platform on the central node, unified data governance, unified operation and maintenance management, data capability opening and other capabilities of regional-level traffic big data are realized.

[0011] In one implementation of the present invention, the data storage sub-platform includes: a second original area, a second trusted area and a second value area; wherein the second original area is connected to the second trusted area, the unified data governance sub-platform and the edge node respectively, the second original area is used to collect and store the second original data, and to send the second original data to the second trusted area, and to receive the first original data on demand; the second trusted area is connected to the second value area, the unified data governance sub-platform and the edge node respectively, the second trusted area is used to receive the second original data, and to receive the first trusted area data on demand, and to perform a second pre-processing on the second original data to obtain and store the second trusted area data, and to send the second trusted area data to the second value area; the second value area is also connected to the unified data governance sub-platform and the edge node, the second value area is used to receive the second trusted area data, and to receive the first value area data on demand, and to perform subject-oriented analysis on the second trusted area data to obtain and store the second value area data.

[0012] In this implementation, the data on the data storage sub-platform is stored in a partitioned manner, which facilitates data management and meets the requirements of strong real-time data and rapid processing, analysis and mining.

[0013] In one implementation of the present invention, the calculations provided by the data analysis sub-platform for the unified data governance sub-platform and upper-level traffic applications include at least any one or a combination of two or more of the following: real-time calculations, offline calculations, and AI calculations.

[0014] In one implementation of the present invention, the unified data governance sub-platform implements data governance based on metadata management, data directory management and data security management; the unified data governance sub-platform includes at least any one or two or more combined data governance processes: data exploration, data extraction, data cleaning, data quality verification, data standard detection, data conversion, data label configuration, and data indicator design.

[0015] In one implementation of the present invention, the big data support sub-platform is used to form a general capability pool; on the big data support sub-platform, each traffic application is allocated an independent component instance according to the application, and each traffic application has its own namespace to implement service deployment of each traffic application.

[0016] In one implementation of the present invention, the big data support sub-platform includes at least any one or two or more of the following components: containerization service, cluster container orchestration service, application computing power allocation management, continuous release and continuous integration, distributed application coordination component, distributed full-text retrieval component, distributed object storage component, distributed cache component, distributed message component, distributed database component, and high-precision map engine component.

[0017] In one implementation of the present invention, the unified operation and maintenance management sub-platform is used to implement operation and maintenance monitoring, workflow scheduling and service collaboration for traffic application operations.

[0018] In one implementation of the present invention, the data capability exposure sub-platform is used to provide an open data interface, and the data capability exposure sub-platform includes a production environment, a dedicated zone environment and a sandbox environment; wherein, the production environment is used to provide at least any one or two or more combinations of the following data capability exposure services for each transportation application: resource allocation and management, data development, mining tools, reporting tools, and service packaging tools; the dedicated zone environment is used to provide data capability exposure services for different types of transportation applications and different partners; the sandbox environment is used to provide data capability exposure services for at least the following scenarios: testing of various transportation application scenarios and / or scientific training for competition activities.

[0019] As described above, the smart transportation big data lake described in the present invention has the following beneficial effects:

[0020] (1) Compared with the existing technology, the present invention provides a smart transportation big data lake with a central node + edge node architecture, which can meet the requirements of strong real-time processing of transportation big data, and through hierarchical and partitioned storage, it realizes the rapid processing, analysis and mining of massive data, as well as the effective governance and management of massive data.

[0021] (2) The smart transportation big data lake provided by the present invention realizes the unified data governance, unified operation and maintenance management, and data capability opening of regional transportation big data through hierarchical and partitioned data storage-level governance and hierarchical data capability opening. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 Shown is a structural diagram of the smart transportation big data lake according to an embodiment of the present invention.

[0023] Figure 2 Shown is a schematic diagram of component services of the big data support sub-platform according to an embodiment of the present invention.

[0024] Figure 3Shown is a structural schematic diagram of the second value zone according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The following describes the embodiments of the present invention through specific examples. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments. The details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other unless they conflict.

[0026] It should be noted that the illustrations provided in the following embodiments are merely schematic illustrations of the basic concept of the present invention. The illustrations only show components related to the present invention and are not drawn according to the number, shape, and size of components in actual implementation. In actual implementation, the type, quantity, and proportion of each component may be changed arbitrarily, and the component layout may also be more complex.

[0027] See Figures 1 to 3 The following embodiments of the present invention provide a smart transportation big data lake. Compared with the existing technology, the present invention provides a smart transportation big data lake with a central node + edge node architecture, which can meet the requirements of strong real-time processing of transportation big data, and through hierarchical and partitioned storage, it realizes the rapid processing, analysis and mining of massive data, as well as the effective governance and management of massive data; the smart transportation big data lake provided by the present invention realizes the unified data governance, unified operation and maintenance management, data capability opening and other capabilities of regional-level transportation big data through hierarchical and partitioned data storage-level governance and hierarchical data capability opening.

[0028] A data lake is a centralized repository that allows you to store all your structured and unstructured data at any scale. Specifically, you can store data as is (without structuring it first) and run different types of analytics – from dashboards and visualizations to big data processing, real-time analytics, and machine learning – to guide better decision making.

[0029] The technical solutions in the embodiments of the present invention will be described in detail below with reference to the accompanying drawings in the embodiments of the present invention.

[0030] like Figure 1 As shown, in one embodiment, the present invention provides a regional-level smart transportation big data lake, which includes: a central node 1 and at least one edge node 2 (with Figure 1 The following example includes two edge nodes, namely edge node A and edge node B).

[0031] Specifically, one edge node 2 corresponds to a partition; the edge node 2 is connected to the central node 1, and the edge node 2 is used to collect the first original data (corresponding to traffic big data) in the corresponding partition, and to perform a first preprocessing on the first original data to obtain first trusted zone data, and to perform a subject-oriented analysis on the first trusted zone data to obtain first value zone data, and to send any one or two or more combinations of the first original data, the first trusted zone data and the first value zone data to the central node 1 as needed; the central node 1 is used to collect, store and process the second original data (corresponding to traffic big data) in the area where it is located.

[0032] It should be noted that the above-mentioned "on demand" refers to according to actual needs; specifically, in actual applications, according to the actual needs of the central node 1, the edge node 2 chooses to send any one or two or more combinations of the first original data, the first trusted zone data and the first value zone data to the central node 1; for example, if the central node 1 only needs the first trusted zone data, the edge node 2 will send the first trusted zone data to the central node 1.

[0033] It should be noted that topic-oriented analysis is a data analysis method, the core of which is to organize and analyze data according to specific topics. This method emphasizes the subject matter of the data, that is, the data needs to be classified and organized according to a specific topic or field in order to better understand and analyze the data.

[0034] In one embodiment, the first original data includes at least but is not limited to data generated on the vehicle side and road side of the traffic network, such as vehicle side quasi-static parameters such as vehicle number, vehicle software version, positioning accuracy, time synchronization method, coordinate system type, GNSS speed, position, heading angle, steering wheel angle, vehicle speed, engine torque, fuel consumption and other vehicle operating status information, vehicle number, protocol type, camera number, video quality, URL address and other vehicle video stream information, device category, perception device number, target number, target data packet, longitude, latitude and other road side perception target data, as well as other traffic participant data, traffic event data, traffic flow data, etc.

[0035] In one embodiment, the second original data includes at least but is not limited to vehicle GPS data, payment data, road monitoring data, mobile phone signaling data, social media data, meteorological data, road sensor data, aviation, railway, and water transport data, map and geographic information system (GIS) data, accident report data, and other types of data in urban life that may affect traffic.

[0036] In one embodiment, both the first original data and the second original data are data that have not been cleaned or sorted, and their data formats are relatively original.

[0037] like Figure 1 As shown, in one embodiment, the edge node 2 includes: a first original area 21 , a first trusted area 22 and a first value area 23 .

[0038] Specifically, the first original area 21 is connected to the central node 1 and the first trusted area 22 respectively, and the first original area 21 is used to collect and store the first original data (unprocessed and untreated data), and to send the first original data to the first trusted area 22, and to send the first original data to the central node 1 on demand; the first trusted area 22 is also connected to the central node 1 and the first value area 23, and the first trusted area 22 is used to receive the first original data, and to perform the first preprocessing on the first original data to obtain and store the first trusted area data, and send the first trusted area data to the first value area 23, and to send the first trusted area data to the central node 1 on demand; the first value area 23 is also connected to the central node 1, and the first value area 23 is used to receive the first trusted area data, and to perform subject-oriented analysis on the first trusted area data to obtain and store the first value area data, and to send the first value area data to the central node 1 on demand.

[0039] It should be noted that the data in the first value zone is subject-oriented and is designed to meet specific analysis needs.

[0040] In one embodiment, the first preprocessing includes at least but is not limited to any one or a combination of two or more of the following processing methods: extraction, cleaning, conversion, loading, performing data quality, and verification.

[0041] Specifically, the first trusted zone 22 extracts, cleans, converts, and loads the first original data, performs data quality and verification, and then obtains the first trusted zone data.

[0042] It should be noted that the first trusted area 22 is a data warehouse that processes and processes the first original data stored in the first original area 21. It stores data that has been cleaned and sorted. The data stored in the first trusted area 22 (i.e., the first trusted area data) has a relatively standardized data structure and can support complex analysis operations.

[0043] In one embodiment, the first value area 23 stores the first value area data obtained through subject-oriented analysis. It is a data warehouse specifically used to store data in a certain business field in order to meet certain specific analysis needs, including but not limited to basic information subject categories, transportation management subject categories, comprehensive situation subject categories, etc.

[0044] In one embodiment, the first original area 21 , the first trusted area 22 , and the first value area 23 are consistent with the data storage partition strategy algorithm of the central node 1 .

[0045] In one embodiment, the partitioning strategy algorithms of the first original zone 21 , the first trusted zone 22 , and the first value zone 23 are each set based on the edge node 2 .

[0046] like Figure 1 As shown, in one embodiment, the central node 1 includes: a data storage sub-platform 11, a data analysis sub-platform 12, a data capability exposure sub-platform 13, a big data support sub-platform 14, a unified data governance sub-platform 15 and a unified operation and maintenance management sub-platform 16.

[0047] Specifically, the data storage sub-platform 11 is connected to the edge node 2, and the data storage sub-platform 11 is used to collect and store the second original data, and to receive any one or two or more combinations of the first original data, the first trusted zone data and the first value zone data on demand; the unified data governance sub-platform 15 is connected to the data storage sub-platform 11, and the unified data governance sub-platform 15 is used to extract data from the data storage sub-platform 11 on demand; the data analysis sub-platform 12 is connected to the unified data governance sub-platform 15, and the data analysis sub-platform 12 is used to The platform 15 and upper-level traffic applications provide computing; the big data support sub-platform 14 is respectively connected to the unified data governance sub-platform 15, the data analysis sub-platform 12, the data storage sub-platform 11 and the edge node 2; the data capability opening sub-platform 13 is respectively connected to the big data support sub-platform 14, the data analysis sub-platform 12, and the unified data governance sub-platform 15; the unified operation and maintenance management sub-platform 16 is respectively connected to the data storage sub-platform 11, the big data support sub-platform 14, the data analysis sub-platform 12, the unified data governance sub-platform 15 and the data capability opening sub-platform 13.

[0048] It should be noted that the central node 1, through the setting of the data storage sub-platform 11, the data analysis sub-platform 12, the data capability opening sub-platform 13, the big data support sub-platform 14, the unified data governance sub-platform 15 and the unified operation and maintenance management sub-platform 16, realizes regional-level unified data governance, unified operation and maintenance management, data capability opening, and unified management of big data support platform components.

[0049] like Figure 1 As shown, in one embodiment, the data storage sub-platform 11 includes: a second original area 111 , a second trusted area 112 and a second value area 113 .

[0050] Specifically, the second original area 111 is connected to the second trusted area 112, the unified data governance sub-platform 15 and the edge node 2 (the first original area 21 of the edge node 2) respectively, and the second original area 111 is used to collect and store the second original data, and to send the second original data to the second trusted area 112, and receive the first original data on demand; the second trusted area 112 is connected to the second value area 113, the unified data governance sub-platform 15 and the edge node 2 (the first trusted area 22 of the edge node 2) respectively, and the second trusted area 112 is used to receive The second original data is received, and the first trusted zone data is received on demand, and the second preprocessing is performed on the second original data to obtain and store the second trusted zone data, and the second value zone data is sent to the second value zone 113; the second value zone 113 is also connected to the unified data governance sub-platform 15 and the edge node 2 (the first value zone 23 of the edge node 2), and the second value zone 113 is used to receive the second trusted zone data, and the first value zone data is received on demand, and the subject-oriented analysis is performed on the second trusted zone data to obtain and store the second value zone data.

[0051] It should be noted that the data in the second value zone is subject-oriented and is designed to meet specific analysis needs.

[0052] It should be noted that the data storage sub-platform 11 is partitioned and managed according to the second original area 111, the second trusted area 112, and the second value area 113, which facilitates data management and meets the requirements of strong real-time data and rapid processing, analysis and mining. Specifically, the second original area 111 stores the second original data; the second trusted area 112 stores the second trusted area data; and the second value area 113 stores the second value area data.

[0053] In one embodiment, the second trusted area 112 is a data warehouse obtained by processing and manipulating the second original data stored in the second original area 111. The second trusted area 112 stores cleaned and organized data, which is usually classified and organized according to business themes.

[0054] In one embodiment, the second preprocessing includes at least but is not limited to any one or a combination of two or more of the following processing methods: extraction, cleaning, conversion, loading, performing data quality, and verification.

[0055] Specifically, the data in the second trusted zone 112 has undergone processes such as extraction, cleaning, conversion, and loading, and the data structure is relatively standardized, which can support complex analysis operations.

[0056] Specifically, the second value zone 113 is a data warehouse specifically used to store data in a certain business field, established on the basis of the second trusted zone 112 in order to meet certain specific analysis requirements.

[0057] like Figure 3 As shown, it is an example diagram of the second value area 113, which stores the value data obtained through subject-oriented analysis, including but not limited to basic information subject category, transportation management subject category, comprehensive situation subject category, etc.

[0058] In one embodiment, the calculations provided by the data analysis sub-platform 12 for the unified data governance sub-platform 15 and upper-level traffic applications include at least but are not limited to any one or a combination of two or more of the following: real-time calculations, offline calculations, and AI calculations.

[0059] In one embodiment, the data analysis sub-platform 12 provides three data analysis frameworks, namely real-time, offline, and AI, to build data analysis and computing service capabilities based on different types of data features and customer business needs.

[0060] Specifically, for vehicle-road collaboration and other requirements with large amounts of real-time data, the Flink and Spark Streaming real-time computing frameworks are used to provide high-throughput, low-latency, high-performance processing and analysis of real-time streaming data; for the processing needs of massive static business data accumulated in the system, the Spark and MapReduce offline computing frameworks are used to provide complex analysis and calculations with delayed responses; for the data analysis, mining and modeling needs of front-end intelligent applications, as well as the analysis and processing of video, image and multimodal data, an artificial intelligence modeling development environment is provided through built-in algorithm operators and standardized machine learning and deep learning AI components to support user business innovation.

[0061] In one embodiment, the unified data governance sub-platform 15 implements data governance based on metadata management, data directory management, and data security management.

[0062] Specifically, the unified data governance sub-platform 15 performs modeling and data governance according to the specified data standards and business models, performs digital resource management on all data resources, and manages the entire life cycle of smart transportation big data in accordance with corresponding data specifications.

[0063] It should be noted that the unified data governance sub-platform 15 ensures the accuracy, completeness and reliability of data, improves the availability and credibility of data, and increases the value of data through normalization, standardization and transparency; the unified data governance sub-platform 15 is based on metadata and supports data standard management, data quality management, data asset management, etc., providing better decision support and business support for vehicle-road collaborative services.

[0064] In one embodiment, the unified data governance sub-platform 15 includes at least but is not limited to any one or a combination of two or more of the following data governance processes: data exploration, data extraction, data cleaning, data quality verification, data standard detection, data conversion, data label configuration, and data indicator design.

[0065] like Figure 1 and Figure 2 As shown, in one embodiment, the big data support sub-platform 14 is used to form a general capability pool; on the big data support sub-platform 14, each traffic application is allocated an independent component instance according to the application, and each traffic application has its own namespace to implement the service deployment of each traffic application. At the same time, the general components can serve each traffic application.

[0066] In one embodiment, the traffic application includes at least but is not limited to any one or a combination of two or more of the following: vehicle-road collaborative application, cloud simulation application, smart travel application, and smart logistics application.

[0067] like Figure 2 As shown in the figure, the vehicle-road collaborative application and the cloud simulation application allocate independent component instances according to the application, and implement the service deployment of each application through their respective namespaces.

[0068] It should be noted that the big data support sub-platform 14 serves as the underlying support for the entire smart transportation big data lake, providing high-performance, highly stable, and highly available data warehouse software for the edge node 2 of the data storage sub-platform 11. At the same time, it provides support services for the unified data governance sub-platform 15 and the data analysis sub-platform 12 by providing platform support components.

[0069] In one embodiment, the big data support sub-platform 14 includes at least but is not limited to any one or two or more of the following components: containerization services, cluster container orchestration services, application computing power allocation management, continuous publishing and continuous integration, distributed application coordination components, distributed full-text retrieval components, distributed object storage components, distributed cache components, distributed message components, distributed database components, and high-precision map engine components.

[0070] In one embodiment, the big data support sub-platform 14 provides services to the central node 1 and the edge node 2 on demand at the same time.

[0071] In one embodiment, the unified operation and maintenance management sub-platform 16 is used to implement operation and maintenance monitoring, workflow scheduling, and service collaboration for transportation application operations.

[0072] In one embodiment, the data capability opening sub-platform 13 is used to provide an open data interface, and provide services such as big data support sub-platform capabilities, data query statistics, data analysis and mining, and data storage to the outside world in a unified form and standard. Various applications provide a unified interface calling service to simplify the deployment and upgrade of applications.

[0073] like Figure 1 As shown, in one embodiment, the data capability exposure sub-platform 13 includes a production environment 131 , a dedicated zone environment 132 and a sandbox environment 133 .

[0074] Specifically, the production environment 131 is used to provide at least but not limited to any one or two or more of the following data capability opening services for each transportation application: resource allocation and management, data development, mining tools, reporting tools, and service packaging tools; the dedicated zone environment 132 is used to provide data capability opening services for different types of transportation applications and different partners; the sandbox environment 133 is used to provide data capability opening services for at least but not limited to the following scenarios: testing of various transportation application scenarios and / or scientific training for competition activities.

[0075] It should be noted that the present invention proposes a regional-level smart transportation big data lake that achieves logical unification and physical dispersion for massive data with characteristics such as large scale, multiple types, low value density and fast speed. Through the central node + edge node architecture, it meets the requirements of strong real-time processing of traffic big data; through the partitioned storage of central nodes and edge nodes, it realizes the rapid processing, analysis and mining of massive data; through the unified data governance sub-platform of the central node, it realizes the unified and effective governance and management of massive data.

[0076] The descriptions of the processes or structures corresponding to the above figures have different emphases. For parts that are not described in detail in a certain process or structure, please refer to the relevant descriptions of other processes or structures.

[0077] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the present invention. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical principles disclosed herein are intended to be covered by the claims of the present invention.

Claims

1. A smart transportation big data lake, characterized by: The smart transportation big data lake includes: a central node and at least one edge node; wherein, The edge node corresponds to a partition; the edge node is connected to the central node, and the edge node is used to collect the first original data in the corresponding partition, and to perform a first preprocessing on the first original data to obtain the first trusted zone data, and to perform a subject-oriented analysis on the first trusted zone data to obtain the first value zone data, and to send any one or a combination of two or more of the first original data, the first trusted zone data and the first value zone data to the central node as needed; The central node is used to collect, store and process the second original data in the area where it is located.

2. The smart transportation big data lake according to claim 1 is characterized in that: The edge node includes: a first original area, a first trusted area and a first value area; wherein, The first original area is connected to the central node and the first trusted area respectively, and the first original area is used to collect and store the first original data, and to send the first original data to the first trusted area, and to send the first original data to the central node as needed; The first trusted zone is also connected to the central node and the first value zone, and the first trusted zone is used to receive the first original data, and to perform the first preprocessing on the first original data to obtain and store the first trusted zone data, and send the first trusted zone data to the first value zone, and send the first trusted zone data to the central node as needed; The first value zone is also connected to the central node. The first value zone is used to receive the first trusted zone data and perform subject-oriented analysis on the first trusted zone data to obtain and store the first value zone data and send the first value zone data to the central node as needed.

3. The smart transportation big data lake according to claim 1 is characterized in that: The central node includes: a data storage sub-platform, a data analysis sub-platform, a data capability exposure sub-platform, a big data support sub-platform, a unified data governance sub-platform and a unified operation and maintenance management sub-platform; The data storage sub-platform is connected to the edge node, and the data storage sub-platform is used to collect and store the second original data, and to receive any one or a combination of two or more of the first original data, the first trusted zone data, and the first value zone data as needed; The unified data governance sub-platform is connected to the data storage sub-platform, and the unified data governance sub-platform is used to extract data from the data storage sub-platform on demand; The data analysis sub-platform is connected to the unified data governance sub-platform, and the data analysis sub-platform is used to provide calculation for the unified data governance sub-platform and upper-level traffic applications; The big data support sub-platform is respectively connected to the unified data governance sub-platform, the data analysis sub-platform, the data storage sub-platform and the edge node; The data capability opening sub-platform is respectively connected with the big data support sub-platform, the data analysis sub-platform, and the unified data governance sub-platform; The unified operation and maintenance management sub-platform is respectively connected to the data storage sub-platform, the big data support sub-platform, the data analysis sub-platform, the unified data governance sub-platform and the data capability opening sub-platform.

4. The smart transportation big data lake according to claim 3 is characterized in that: The data storage sub-platform includes: a second original area, a second trusted area and a second value area; wherein, The second original zone is connected to the second trusted zone, the unified data governance sub-platform and the edge node respectively, and the second original zone is used to collect and store the second original data, send the second original data to the second trusted zone, and receive the first original data on demand; The second trusted zone is connected to the second value zone, the unified data governance sub-platform and the edge node respectively, and the second trusted zone is used to receive the second original data, and receive the first trusted zone data as needed, and perform second preprocessing on the second original data to obtain and store the second trusted zone data, and send the second trusted zone data to the second value zone; The second value zone is also connected to the unified data governance sub-platform and the edge node. The second value zone is used to receive the second trusted zone data, and receive the first value zone data on demand, and perform subject-oriented analysis on the second trusted zone data to obtain and store the second value zone data.

5. The smart transportation big data lake according to claim 3 is characterized in that: The calculations provided by the data analysis sub-platform for the unified data governance sub-platform and upper-level traffic applications include at least any one of the following or a combination of two or more: real-time calculations, offline calculations, and AI calculations.

6. The smart transportation big data lake according to claim 3 is characterized in that: The unified data governance sub-platform implements data governance based on metadata management, data catalog management and data security management; The unified data governance sub-platform includes at least one or a combination of two or more of the following data governance processes: data exploration, data extraction, data cleaning, data quality verification, data standard detection, data conversion, data label configuration, and data indicator design.

7. The smart transportation big data lake according to claim 3 is characterized in that: The big data support sub-platform is used to form a general capability pool; on the big data support sub-platform, each traffic application is allocated an independent component instance according to the application, and each traffic application has its own namespace to implement the service deployment of each traffic application.

8. The smart transportation big data lake according to claim 3 or 7, characterized in that: The big data support sub-platform includes at least any one or two or more of the following components: containerization service, cluster container orchestration service, application computing power allocation management, continuous publishing and continuous integration, distributed application coordination component, distributed full-text retrieval component, distributed object storage component, distributed cache component, distributed message component, distributed database component, and high-precision map engine component.

9. The smart transportation big data lake according to claim 3 is characterized in that: The unified operation and maintenance management sub-platform is used to realize operation and maintenance monitoring, workflow scheduling and service coordination of traffic application operations.

10. The smart transportation big data lake according to claim 3, characterized in that: The data capability opening sub-platform is used to provide an open data interface, and the data capability opening sub-platform includes a production environment, a zone environment and a sandbox environment; wherein, The production environment is used to provide at least one or a combination of two or more of the following data capability exposure services for each traffic application: resource allocation and management, data development, mining tools, reporting tools, and service packaging tools; The dedicated zone environment is used to provide data capability exposure services for different types of traffic applications and different partners; The sandbox environment is used to provide data capability exposure services for at least the following scenarios: various traffic application scenario tests and / or scientific training for competition activities.

Images