Urban engineering geology big data platform management method and system

Abstract

A kind of urban engineering geology big data platform management method, including job management department, be configured to the input data packet information of geological database acquisition is managed and classified task;Task script is generated according to task management and the task classification;Get editing command, editing command and task script are sent to geological database;Geological database retrieves original data from geological database according to task script and the input data packet obtained;Geological database is based on watermark mechanism and window mechanism, according to editing command, it is converted to structured output data and spatial output data to structured original data and spatial original data.This application generates task script in job management department, sets up structured database, unstructured database and spatial database in geological database, classifies task script, respectively sent to each database, to improve the efficiency of original data from database retrieval.

Description

Technical Field

[0001] This invention relates to the field of big data cloud platform technology, and in particular to a management method and system for an urban engineering geology big data platform. Background Technology

[0002] Modern large-scale urban construction, especially the design of underground urban spaces, often involves numerous construction stages and units, encompassing surveying, testing, and construction processes. Furthermore, the stress conditions and usage environment of underground structures are more complex, requiring the integrated processing of various information sources and collaborative construction by multiple units. The data formats and forms input into the urban engineering geology big data platform vary significantly between different units and channels. Therefore, it is necessary to transform the raw data obtained from the data source layer into data usable by the application data layer within the database.

[0003] Currently, existing technologies, such as the Chinese patent with announcement number CN116958465A, disclose a spatiotemporal big data cloud service platform based on geological 3D modeling visualization. By classifying, storing, and managing various types of data, it achieves standardized and unified geological data and conveniently presents stratigraphic patterns, enabling surveyors to accurately determine geological features. Classifying and aggregating various types of data allows similar data to be stored in corresponding modules, making storage more standardized, data retrieval faster, and improving server operating efficiency.

[0004] However, the above technical solution still has the following drawbacks: during the process of converting the original data in the data source layer, the efficiency of retrieving the original data is low, resulting in low running efficiency of the conversion process. Summary of the Invention

[0005] The specific solution of this application is as follows: The first aspect of this invention provides a management method for an urban engineering geological big data platform, including a geological data management platform and a geological database.

[0006] The geological data management platform includes a data governance unit, which includes an operation management department. The operation management department is configured to perform task management and task classification on the input data packet information obtained from the geological database.

[0007] Based on the task management and task classification, generate task scripts;

[0008] Obtain the editing command, and send the editing command and task script to the geological database;

[0009] The geological database retrieves structured raw data, unstructured raw data, and spatial raw data from the structured database, unstructured database, and spatial database within the geological database, based on the task script and the acquired input data packet.

[0010] The geological database, based on watermarking and windowing mechanisms, transforms the structured raw data and spatial raw data according to the editing command to obtain structured output data and spatial output data. The structured output data, spatial output data, and unstructured raw data are then merged to obtain the output data.

[0011] More preferably, the task management includes identifying the conversion name in the input data packet information, wherein the conversion name includes test, project type, borehole type, project information, borehole information, project information, and geotechnical naming;

[0012] The task classification includes identifying the classification name, execution strategy, log level, and data type in the input data packet information;

[0013] The classification names include: First ODS Layer and Second ODS Layer;

[0014] The execution strategies include: executing once every five minutes, executing once every hour, and executing once a day;

[0015] The data types include: unstructured data, structured non-spatial data, and structured spatial data;

[0016] The editing commands include: Add Conversion, Execute Once, Start All, and Stop All;

[0017] The log levels include: Detailed, Error, Minimal, and Flowlevel.

[0018] More preferably, based on the task management and task classification, a task script is generated, including:

[0019] Sequentially obtain the transformation name, data type, category name, execution strategy, and log level of each data item in the input data packet information;

[0020] Based on the transformation name, data type, category name, execution strategy, and log level of the data, generate a task script.

[0021] More preferably, when the conversion name is "test", there is no need to obtain the data type and category name, and the execution strategy is to execute once every five minutes; when the conversion name is "project type", "drilling type", "project information" and "geotechnical naming", the execution strategy is to execute once every hour; when the conversion name is "drilling information" and "project information", the execution strategy is to execute once a day.

[0022] When the data type is unstructured data, there is no need to obtain a category name. When the data type is structured non-spatial data, the category name is the first ODS layer. When the data type is structured spatial data, the category name is the second ODS layer.

[0023] Further preferably, the geological database further includes a data type identification unit, which is configured to classify the task script into structured task scripts, unstructured task scripts, and spatial task scripts according to the data type in the task script, send the structured task script to a structured database, send the spatial task script to a spatial database, and send the unstructured task script to an unstructured database.

[0024] More preferably, the geological database retrieves structured raw data, unstructured raw data, and spatial raw data from the structured database, unstructured database, and spatial database within the geological database according to the task script and the acquired input data packet, including:

[0025] The structured database is provided with a first ODS raw data layer, the spatial database is provided with a second ODS raw data layer, and the unstructured database is provided with a file storage server.

[0026] The first ODS raw data layer, the file storage server, and the second ODS raw data layer are all equipped with a transformation name node, an execution strategy node, and a log level node.

[0027] The structured database retrieves the original structured data based on the transformation name, execution strategy, and log level in the structured task script; the spatial database retrieves the original spatial data based on the transformation name, execution strategy, and log level in the spatial task script; and the unstructured database retrieves the original unstructured data based on the transformation name, execution strategy, and log level in the unstructured task script.

[0028] More preferably, the geological database, based on a watermarking mechanism and a windowing mechanism, transforms the structured raw data and spatial raw data according to the editing command to obtain structured output data and spatial output data, including:

[0029] Based on the watermarking mechanism, timestamps are set for the structured raw data and spatial raw data. Based on the window mechanism, the structured raw data and spatial raw data retrieved within a preset time period are collected.

[0030] The structured raw data collected within the preset time period is transformed according to the editing command to obtain structured output data, and the spatial raw data collected within the preset time period is transformed according to the editing command to obtain spatial output data.

[0031] More preferably, the job management department also includes an ETL tool, which divides the input data packet into script name, script path, and type;

[0032] Based on the script name, script path, and type, the original data is retrieved from the geological database, and the original data is transformed to obtain the output data.

[0033] Furthermore, preferably, the data management platform also includes a standardized comparison unit, which will include engineering survey results - project information, including:

[0034] Search and visualize the project by year, project type, number, engineering number, project name, project name, project number, project type, inspection stage, area unit, and project manager.

[0035] A second aspect of the present invention provides an urban engineering geological big data platform management system, the system comprising at least one processor; and a memory storing instructions that, when executed by the at least one processor, implement the steps of the method described in the first aspect.

[0036] Compared to existing technologies, the advantages of this invention lie in its ability to facilitate operators in selecting the raw data to be converted by establishing an operation management department and generating task scripts within it. Furthermore, by setting up structured, unstructured, and spatial databases within the geological database, and classifying the task scripts before sending them to these databases respectively, the invention achieves categorized storage of data of different types and improves the efficiency of retrieving raw data from the databases. Attached Figure Description

[0037] Figure 1 This is a flowchart illustrating the management method of the urban engineering geological big data platform of the present invention;

[0038] Figure 2 This is a schematic diagram of the first data transmission in the urban engineering geology big data platform management method of the present invention;

[0039] Figure 3 This is a schematic diagram of the second data transmission in the urban engineering geology big data platform management method of the present invention;

[0040] Figure 4This is a schematic diagram of the geological data management platform and geological database of the present invention;

[0041] Figure 5 This is a schematic diagram of the mapping relationship between the various levels within the geological database of this invention;

[0042] Figure 6 This is a schematic diagram of the urban engineering geology big data platform management system of the present invention. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] To facilitate understanding of the methods and systems provided in the embodiments of this application, the background of the embodiments of this application will be introduced before introducing the embodiments of this application.

[0045] Modern large-scale urban construction, especially the design of underground urban spaces, often involves numerous construction stages and units, encompassing surveying, testing, and construction processes. Furthermore, the stress conditions and usage environment of underground structures are more complex, requiring the integrated processing of various information sources and collaborative construction by multiple units. The data formats and forms input into the urban engineering geology big data platform vary significantly between different units and channels. Therefore, it is necessary to transform the raw data obtained from the data source layer into data usable by the application data layer within the database.

[0046] Currently, existing technologies, such as the Chinese patent with announcement number CN116958465A, disclose a spatiotemporal big data cloud service platform based on geological 3D modeling visualization. By classifying, storing, and managing various types of data, it achieves standardized and unified geological data and conveniently presents stratigraphic patterns, enabling surveyors to accurately determine geological features. Classifying and aggregating various types of data allows similar data to be stored in corresponding modules, making storage more standardized, data retrieval faster, and improving server operating efficiency.

[0047] However, the above technical solution still has the following drawbacks: during the process of converting the original data in the data source layer, the efficiency of retrieving the original data is low, resulting in low running efficiency of the conversion process.

[0048] like Figure 1 and Figure 2As shown, some embodiments of the present invention relate to a management method for an urban engineering geological big data platform, including a geological data management platform and a geological database. The geological data management platform includes a data governance unit, and the data governance unit includes an operation management department. The operation management department is configured to perform task management and task classification on the input data packet information obtained from the geological database.

[0049] Based on the task management and task classification, generate task scripts;

[0050] Obtain the editing command, and send the editing command and task script to the geological database;

[0051] The geological database retrieves structured raw data, unstructured raw data, and spatial raw data from the structured database, unstructured database, and spatial database within the geological database, based on the task script and the acquired input data packet.

[0052] The geological database, based on watermarking and windowing mechanisms, transforms the structured raw data and spatial raw data according to the editing command to obtain structured output data and spatial output data. The structured output data, spatial output data, and unstructured raw data are then merged to obtain the output data.

[0053] "Urban Engineering Geology Big Data Platform" (data entry, data governance); data source management (data governance data sources, data asset data, metadata data management, operation management, data quality management, data standardization, data release (including services), data supervision), "Beijing Standard Stratigraphic Urban Classification Standard". Service construction and release --> Geological Big Data Service Platform --> Geological Big Data Center Portal Website, 2D Geological Information Service System, 3D Geological Visualization and Data Service System.

[0054] Geological Database Construction: Based on the "Classification and Coding Standards for Geotechnical Engineering Investigation Data", a geological spatiotemporal database will be established to aggregate data from engineering investigation, geological survey, groundwater monitoring, and three-dimensional geological models, so as to achieve unified storage and management of multi-source heterogeneous geological data, including internal and external data, static and dynamic data, historical, current and future data, and spatial and non-spatial data.

[0055] Geological data management platform construction: As the management foundation for geological data, it provides capabilities such as data access, data cleaning, data fusion, data quality, data security, data assets, data services, and data monitoring. Data management personnel manage the raw geological data to form standardized and structured data, and provide unified data service interfaces to external systems, realizing the sharing and utilization of geological exploration data across enterprises, cities, and platforms.

[0056] In these implementations, by setting up a job management department, task management of input data packet information can be performed. This allows for the classification of input data packets according to functional domains, and further classification of the classified input data into tasks. This further categorizes the task-classified input data, facilitating the generation of task scripts. The generated task scripts enable operators to easily select the raw data that needs to be converted.

[0057] Furthermore, the operations management department can also obtain editing commands to manually control the number of conversions, the quantity of conversions, whether to stop the conversion, and manually add conversion tasks for structured raw data and spatial raw data.

[0058] Furthermore, by setting up structured, unstructured, and spatial databases within the geological database, different data types can be classified and stored, and the efficiency of retrieving raw data from the database can be improved.

[0059] Furthermore, by using watermarking and windowing mechanisms, the structured raw data and spatial raw data collected within a preset time period are uniformly converted, thereby improving the resource utilization rate of the geological database.

[0060] like Figure 2 and Figure 3 As shown, in some implementation methods of urban engineering geological big data platform management methods, the task management includes identifying the conversion name in the input data packet information, wherein the conversion name includes test, engineering type, borehole type, engineering information, borehole information, project information and geotechnical naming;

[0061] The task classification includes identifying the classification name, execution strategy, log level, and data type in the input data packet information;

[0062] The classification names include: First ODS Layer and Second ODS Layer;

[0063] The execution strategies include: executing once every five minutes, executing once every hour, and executing once a day;

[0064] The data types include: unstructured data, structured non-spatial data, and structured spatial data;

[0065] The editing commands include: Add Conversion, Execute Once, Start All, and Stop All;

[0066] The log levels include: Detailed, Error, Minimal, and Flowlevel.

[0067] In the above implementation, the input data packet information includes the transformation name, category name, execution strategy, log level, and data type. The operation management department identifies the transformation name through task management, classifying the input data into seven types based on data content and domain: test, engineering type, borehole type, engineering information, borehole information, project information, and geotechnical naming. The operation management department also identifies the category name, execution strategy, log level, and data type of each data type within the transformation name through task classification, facilitating the selection of the original data to be retrieved and transformed from the geological database. Furthermore, task classification can also include transformation description, execution method, acquisition type, transformation path, and synchronization strategy.

[0068] like Figure 2 and Figure 3 As shown, in some implementation methods of urban engineering geological big data platform management, task scripts are generated based on the task management and task classification, including:

[0069] Sequentially obtain the transformation name, data type, category name, execution strategy, and log level of each data item in the input data packet information;

[0070] Based on the transformation name, data type, category name, execution strategy, and log level of the data, generate a task script.

[0071] In these implementations, various task scripts are generated by combining different categories based on transformation name, data type, category name, execution strategy, and log level. By sending the task scripts to the geological database, the geological database can quickly retrieve the original data, thereby improving the efficiency of original data retrieval.

[0072] In some implementation methods of urban engineering geological big data platform management, when the conversion name is "test", it is not necessary to obtain the data type and category name, and the execution strategy is to execute once every five minutes. When the conversion name is "engineering type", "drilling type", "engineering information" and "geotechnical naming", the execution strategy is to execute once every hour. When the conversion name is "drilling information" and "project information", the execution strategy is to execute once a day.

[0073] When the data type is unstructured data, there is no need to obtain a category name. When the data type is structured non-spatial data, the category name is the first ODS layer. When the data type is structured spatial data, the category name is the second ODS layer.

[0074] In these implementations, a first task script is generated when the transformation name is "Project Type," the data type is "Structured Non-Spatial Data," the category name is "First ODS Layer," the execution strategy is "Execute once every hour," and the log level is "error." A second task script is generated when the transformation name is "Project Type," the data type is "Structured Spatial Data," the category name is "Second ODS Layer," the execution strategy is "Execute once every hour," and the log level is "error." This process continues until all task scripts are generated.

[0075] Furthermore, when the name is converted to "test," since test data does not need to distinguish between data types and category names, it is unnecessary to obtain the data types and category names; only the execution strategy needs to be considered, which improves the efficiency of task script generation. Furthermore, since the data converted to "test" is only used for testing by the job management department, the execution strategy for all input data converted to "test" is preset to once every five minutes, generating a third task script.

[0076] Furthermore, due to the differences in content and actual environmental requirements among different types of transformed data, the execution strategies for different types of transformed data differ, while the execution strategies for transformed data of the same type are the same. Therefore, the execution strategy for data can be determined when obtaining the transformation name of the data, eliminating the need to identify the execution strategy, thus improving the task script generation speed.

[0077] Furthermore, since different data types require retrieving raw data from different databases, the classification name of the data can be determined immediately after obtaining its data type, thus improving the speed of task script generation. For example, when processing unstructured data, the operations management department automatically skips the step of obtaining the classification name and directly generates the corresponding unstructured task script based on the data type. Then, it obtains the execution strategy and log level, thereby automating and intelligentizing the data processing workflow and improving the accuracy and efficiency of geological data conversion.

[0078] like Figure 3 As shown, in some implementation methods of urban engineering geological big data platform management methods, the geological database further includes a data type identification unit. The data type identification unit is configured to classify the task script into structured task scripts, unstructured task scripts, and spatial task scripts according to the data type in the task script, send the structured task script to the structured database, send the spatial task script to the spatial database, and send the unstructured task script to the unstructured database.

[0079] In these implementations, since the geological database includes structured databases, unstructured databases, and spatial databases, in the existing scheme, when the geological database receives the task scripts sent by the operation management department, it needs to send all the task scripts to the structured database, unstructured database, and spatial database in sequence. Each database filters all the task scripts, selects the task scripts corresponding to the original data stored in the database for processing, and retrieves the original data corresponding to the task scripts.

[0080] Furthermore, by setting up a data type identification unit, the data types in the task scripts can be identified, classifying task scripts with structured non-spatial data as structured task scripts, task scripts with unstructured data as unstructured task scripts, and task scripts with structured spatial data as spatial task scripts. The data type identification unit then sends structured task scripts to a structured database to retrieve structured raw data, sends unstructured task scripts to an unstructured database to retrieve unstructured raw data, and sends spatial task scripts to a spatial database to retrieve spatial raw data, thereby improving the efficiency of raw data retrieval from the database.

[0081] Furthermore, the task scripts use standard SQL syntax, which facilitates the identification of structured, unstructured, and spatial databases in geological databases.

[0082] like Figure 3 As shown, in some implementation methods of urban engineering geological big data platform management, the geological database retrieves structured raw data, unstructured raw data, and spatial raw data from the structured database, unstructured database, and spatial database within the geological database according to the task script and the acquired input data packet, including:

[0083] The structured database is provided with a first ODS raw data layer, the spatial database is provided with a second ODS raw data layer, and the unstructured database is provided with a file storage server.

[0084] The first ODS raw data layer, the file storage server, and the second ODS raw data layer are all equipped with transformation execution nodes, execution strategy nodes, and log level nodes.

[0085] The structured database retrieves the original structured data based on the transformation name, execution strategy, and log level in the structured task script; the spatial database retrieves the original spatial data based on the transformation name, execution strategy, and log level in the spatial task script; and the unstructured database retrieves the original unstructured data based on the transformation name, execution strategy, and log level in the unstructured task script.

[0086] In these implementations, a first ODS raw data layer is set up in a structured database, a second ODS raw data layer is set up in a spatial database, and a file storage server is set up in an unstructured database. Furthermore, transformation name nodes, execution strategy nodes, and log level nodes are set up in the first ODS raw data layer, the second ODS raw data layer, and the storage server to identify task scripts and retrieve raw data from the database. By setting sub-nodes for test, project type, borehole type, project information, borehole information, project information, and geotechnical naming within the transformation name node; sub-nodes for execution every five minutes, every hour, and daily within the execution strategy node; and sub-nodes for Detailed, Error, Minimal, and Flowlevel within the log level node, the system can quickly retrieve raw data from the database, thereby improving transformation efficiency.

[0087] like Figure 5 As shown, the geological database includes an ODS layer, an ODS node layer, and an ODS child node layer. The ODS layer can contain multiple ODS raw data layers. The ODS node layer contains multiple ODS nodes, and each ODS raw data layer is mapped to one or more ODS nodes. The ODS child node layer contains multiple ODS child nodes, and each ODS node is mapped to one or more ODS child nodes. By establishing a mapping relationship in the database... Figure 5 The multi-level, multi-node mapping relationship shown improves the efficiency of database data retrieval. Specifically, ODS nodes can represent execution strategies, and ODS child nodes can execute strategies every hour.

[0088] Furthermore, the geological database comprises multiple databases, each containing one or more ODS raw data layers. When retrieving raw data, the database retrieves the raw data from the ODS raw data layer within the database based on the classification name specified in the task script.

[0089] like Figure 3 As shown, in some implementation methods of urban engineering geological big data platform management, the geological database, based on watermarking and windowing mechanisms, transforms the structured raw data and spatial raw data according to the editing command to obtain structured output data and spatial output data, including:

[0090] Based on the watermarking mechanism, timestamps are set for the structured raw data and spatial raw data. Based on the window mechanism, the structured raw data and spatial raw data retrieved within a preset time period are collected.

[0091] The structured raw data collected within the preset time period is transformed according to the editing command to obtain structured output data, and the spatial raw data collected within the preset time period is transformed according to the editing command to obtain spatial output data.

[0092] In these implementations, the transformation of structured raw data involves converting the structured raw data retrieved from the data source layer (ODS layer) of the structured database into structured output data that can be stored in the application data layer (ADS layer) of the structured database. Specifically, this requires layered processing, sequentially converting the data in the data source layer (ODS layer) into data in the detailed data layer (DWD layer), then into data in the summary data layer (DWS layer), and finally into data in the application data layer (ADS layer). This process "cleansing, integrating, aggregating, and adapting" the raw data into the form required by the business scenario, making it easy to call upon in the geological data management platform.

[0093] Furthermore, the transformation of the raw spatial data involves converting the raw spatial data retrieved from the data source layer of the spatial database into spatial output data that can be stored in the application data layer of the spatial database.

[0094] Furthermore, since unstructured databases do not have a data source layer, there is no need to transform the unstructured raw data. The structured output data, spatial output data, and unstructured raw data are then merged to obtain output data that can be retrieved and used by the geological data management platform.

[0095] Furthermore, watermarking is an information hiding technology that embeds identifying information (such as copyright information, source identification, and anti-counterfeiting marks) into digital content (such as images, videos, audio, documents, and data) to achieve data copyright protection, source traceability, and anti-counterfeiting verification. Windowing is a core technology in streaming data processing. By dividing an infinite, continuous data stream into finite, manageable windows (such as time windows and counting windows), it enables real-time aggregation, statistics, and analysis of the data. It can process infinite data streams in real time, resolve out-of-order data, and support flexible data analysis, thereby improving processing efficiency. In the process of transforming the structured raw data and spatial raw data, the watermarking mechanism and windowing mechanism complement and coordinate with each other. The watermarking mechanism provides security for the windowing mechanism, while the windowing mechanism provides real-time processing capabilities for the watermarking mechanism.

[0096] Furthermore, based on the editing commands obtained from the geological data management platform, the geological database determines which data in the structured raw data and spatial raw data needs to be transformed and the number of transformations, thereby enabling human control over the transformation of raw data and improving the flexibility of raw data transformation.

[0097] In some implementation methods of urban engineering geological big data platform management, the operation management department also includes an ETL tool, which divides the input data packet into script name, script path and type;

[0098] Based on the script name, script path, and type, the original data is retrieved from the geological database, and the original data is transformed to obtain the output data.

[0099] In these implementations, ETL stands for Extract-Transform-Load. ETL tools encompass data extraction, data transformation, and data loading, supporting scenarios such as enterprise data integration, data warehouse construction, and business intelligence analysis. The data processing flow of ETL tools centers on data extraction, transformation, and loading, converting raw data into a standardized format suitable for analysis and retrieval by the geological data management platform, while ensuring consistency, accuracy, and usability before and after data transformation. By employing ETL tools, the raw data in geological databases can be transformed.

[0100] like Figure 4 As shown, in some implementation methods of urban engineering geological big data platform management, the data management platform further includes a standardized comparison unit, which combines engineering survey results—project information, including:

[0101] Search and visualize the project by year, project type, number, engineering number, project name, project name, project number, project type, inspection stage, area unit, and project manager.

[0102] In these implementations, the data governance unit further includes a data source management department, which manages the data governance data source, data asset data source, and metadata data source.

[0103] Furthermore, the data management platform includes a data resource display unit, which is configured to display data in terms of professional dimensions and data dimensions.

[0104] Professional dimensions are retrieved and displayed from the geological database: Project Number, Exploration Number, Project Type, Project Leader, Project Manager, Completion Date, Completion Department, Construction Unit, Design Unit, Project Location, Exploration Unit, Exploration Stage, Coordinate Location, Plane Coordinate System, Elevation System, Project Description, Site Stratigraphy, Borehole Information, Results Report, Process Documents, Raw Data, Site Stratigraphy, Sequence Number, Stratigraphic Number, Geological Age, Geological Genesis, Soil and Rock Classification, Soil and Rock Name, and Weathering Degree;

[0105] The data dimensions are retrieved and displayed from the geological database: daily data resource catalog, basic information, system tables, enterprise review process tables, engineering information, borehole information, enterprise basic information tables, enterprise related attachment tables, enterprise department information tables, enterprise personnel information tables, role basic information tables, permission basic information tables, personnel certificate information tables, system log information tables, personnel continuing education information tables, labor unit information tables, system version information tables, enterprise basic information, enterprise department information, enterprise personnel information, corresponding databases, files, interfaces, two-dimensional geological maps, and three-dimensional geological models.

[0106] Furthermore, the data management platform includes a standardized comparison unit, which is configured to search and display soil and rock names, queries, addition of standards, batch deletion, stratigraphic sequence, main layer, sub-layer, geological age, geological genesis, soil and rock names, soil and rock classification, color, and stratigraphic description from the geological database.

[0107] Furthermore, the data management platform includes a data linking unit, which is configured to: search and display the total number of monitoring tasks, the number of monitoring jobs, the number of monitoring conversions, the record number, category name, conversion name, conversion description, and number of successes in the monitoring records; and the ID, name, description, and category name in the error records from the geological database.

[0108] Furthermore, the data management platform includes an interface publishing unit, which is configured to retrieve resource name, unique identifier, interface category, source, review status, and creation time information from the geological database based on the interface name, unique identifier, review status, and interface category.

[0109] Furthermore, the data management platform includes a system management unit, which is configured to retrieve and display the following information from the geological database based on the administrative division, business area, and full name of the organization: full name of the organization, abbreviation of the organization, unified social credit code, contact person, and contact number.

[0110] like Figure 6 As shown, the present invention also provides an urban engineering geological big data platform system. The radiation information monitoring system includes at least one processor, a memory, an input device, and a display device. The input device is used to obtain input from the outside. The memory stores instructions. When the instructions are executed by at least one processor, the steps of the method described in the method embodiment are implemented, and the running results are displayed on the display device, thus implementing the steps of the method described in the method embodiment.

[0111] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather as descriptions of features that can embody specific embodiments of a particular invention. Specific features described in this specification within the context of an independent embodiment may also be implemented in combination with a single embodiment. Conversely, various features described within the context of a single embodiment may also be implemented independently in multiple embodiments, or in any suitable sub-combination. Furthermore, while features may be described for combination and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and the claimed combination may be redirected to a sub-combination or a variation thereof.

[0112] Similarly, although operations are described in the accompanying drawings in a specific order, it should not be construed as requiring that such operations be performed in the specific order shown or in sequential order, or that all illustrated operations be performed, in order to achieve the desired result. In certain cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0113] Specific implementations of the subject matter have been described. Other implementations are within the scope of the following claims. For example, the activities described in the claims can be performed in a different order and still achieve the desired result. As an example, the processes described in the drawings do not necessarily require a specific order or sequence to be shown in order to achieve the desired result. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

1. A management method for an urban engineering geology big data platform, comprising a geological data management platform and a geological database, characterized in that, The geological data management platform includes a data governance unit, which includes an operation management department. The operation management department is configured to perform task management and task classification on the input data packet information obtained from the geological database. Based on the task management and task classification, generate task scripts; Obtain the editing command, and send the editing command and task script to the geological database; The geological database retrieves structured raw data, unstructured raw data, and spatial raw data from the structured database, unstructured database, and spatial database within the geological database, based on the task script and the acquired input data packet. The geological database, based on watermarking and windowing mechanisms, transforms the structured raw data and spatial raw data according to the editing command to obtain structured output data and spatial output data. The structured output data, spatial output data, and unstructured raw data are then merged to obtain the output data.

2. The management method for the urban engineering geology big data platform according to claim 1, characterized in that, The task management includes identifying the transformation name in the input data packet information, wherein the transformation name includes test, engineering type, borehole type, engineering information, borehole information, project information, and geotechnical naming; The task classification includes identifying the classification name, execution strategy, log level, and data type in the input data packet information; The classification names include: First ODS Layer and Second ODS Layer; The execution strategies include: executing once every five minutes, executing once every hour, and executing once a day; The data types include: unstructured data, structured non-spatial data, and structured spatial data; The editing commands include: Add Conversion, Execute Once, Start All, and Stop All; The log levels include: Detailed, Error, Minimal, and Flowlevel.

3. The management method for the urban engineering geology big data platform according to claim 1, characterized in that, Based on the task management and task classification, a task script is generated, including: Sequentially obtain the transformation name, data type, category name, execution strategy, and log level of each data item in the input data packet information; Based on the transformation name, data type, category name, execution strategy, and log level of the data, generate a task script.

4. The urban engineering geology big data platform management method according to claim 2, characterized in that, When the conversion name is "Test", there is no need to obtain the data type and category name, and the execution strategy is to execute once every five minutes. When the conversion name is "Project Type", "Drilling Type", "Project Information" and "Geotechnical Name", the execution strategy is to execute once every hour. When the conversion name is "Drilling Information" and "Project Information", the execution strategy is to execute once a day. When the data type is unstructured data, there is no need to obtain a category name. When the data type is structured non-spatial data, the category name is the first ODS layer. When the data type is structured spatial data, the category name is the second ODS layer.

5. The urban engineering geology big data platform management method according to claim 2, characterized in that, The geological database also includes a data type identification unit, which is configured to classify the task script into structured task scripts, unstructured task scripts, and spatial task scripts according to the data type in the task script, send the structured task script to the structured database, send the spatial task script to the spatial database, and send the unstructured task script to the unstructured database.

6. The urban engineering geology big data platform management method according to claim 2, characterized in that, The geological database retrieves structured raw data, unstructured raw data, and spatial raw data from the structured database, unstructured database, and spatial database within the geological database, based on the task script and the acquired input data packet. This includes: The structured database is provided with a first ODS raw data layer, the spatial database is provided with a second ODS raw data layer, and the unstructured database is provided with a file storage server. The first ODS raw data layer, the file storage server, and the second ODS raw data layer are all equipped with a transformation name node, an execution strategy node, and a log level node. The structured database retrieves the original structured data based on the transformation name, execution strategy, and log level in the structured task script; the spatial database retrieves the original spatial data based on the transformation name, execution strategy, and log level in the spatial task script; and the unstructured database retrieves the original unstructured data based on the transformation name, execution strategy, and log level in the unstructured task script.

7. The management method for the urban engineering geology big data platform according to claim 1, characterized in that, The geological database, based on watermarking and windowing mechanisms, transforms the structured raw data and spatial raw data according to the editing commands to obtain structured output data and spatial output data, including: Based on the watermarking mechanism, timestamps are set for the structured raw data and spatial raw data. Based on the window mechanism, the structured raw data and spatial raw data retrieved within a preset time period are collected. The structured raw data collected within the preset time period is transformed according to the editing command to obtain structured output data, and the spatial raw data collected within the preset time period is transformed according to the editing command to obtain spatial output data.

8. The management method for the urban engineering geology big data platform according to claim 1, characterized in that, The job management department also includes an ETL tool, which divides the input data packet into script name, script path and type; Based on the script name, script path, and type, the original data is retrieved from the geological database, and the original data is transformed to obtain the output data.

9. The management method for the urban engineering geology big data platform according to claim 1, characterized in that, The data management platform also includes a standardized comparison unit, which will compare the engineering survey results and project information, including: Search and visualize the project by year, project type, number, engineering number, project name, project name, project number, project type, inspection stage, area unit, and project manager.

10. A big data platform system for urban engineering geology, characterized in that, The system includes at least one processor; and a memory storing instructions that, when executed by the at least one processor, perform the steps of the method according to any one of claims 1-9.

Images