A Method for Managing the Cloud Lake Architecture of the Production Operations of a Fully Automated Container Terminal

Abstract

The present invention provides a method for managing the cloud lake architecture of the production operations of a fully automated container terminal. The cloud lake system is used to manage the massive data information of the container terminal. The cloud lake system has functions of data classification, data storage, data search, and data application. Data classification classifies the collected data information according to the system categories to which the data information belongs. Data storage stores the data information in different databases. When a certain data information needs to be searched, data search provides a retrieval of the database based on the category characteristics of the data information to be searched and feeds the results back to the business personnel. The cloud lake system can ensure the visualization, availability, and operability of the data information. The present invention can achieve effective management of the terminal data.

Description

Technical Field

[0001] The present invention relates to the technical field of terminal data information management, and particularly relates to a management method for the cloud lake architecture of the production operations of an automated container terminal. Background Art

[0002] With the continuous advancement of the strategy of building a transportation power, the port trade volume has been increasing year by year, and the level of terminal automation and intelligence has been continuously improved. As a result, a large amount of data has been generated by various port systems, including but not limited to terminal equipment information, cargo information, office task information, production system information, etc. Data information is a quantitative scale for measuring the port status and is of great significance in port operation scheduling and task arrangement. The existing data information management methods in ports usually develop an information system by themselves or outsource it to a third party to achieve data information storage and retrieval. Although this method can effectively manage data information, it has deficiencies in aspects such as data retrieval speed and data storage capacity. Summary of the Invention

[0003] The purpose of the present invention is to solve the management problem of the massive data information of container terminals, and provide a management method for the cloud lake architecture of the production operations of an automated terminal, which can achieve data without leaving the lake, shorten the data analysis link, directly connect to services such as BI reports and interactive queries, solve the data island problem, and at the same time can also achieve unified data storage, support connection to capabilities such as machine learning and data analysis, support the use of various data processing systems such as time series analysis and spatio-temporal analysis engines, reduce data migration between systems, and achieve data sharing and real-time analysis capabilities.

[0004] The present invention provides a management method for the cloud lake architecture of the production operations of a fully automated container terminal, which uses a cloud lake system to manage the massive data information of the container terminal and realizes functions such as data classification, data storage, data search, and data application; data classification classifies the collected data information according to the system category to which the data information belongs, and data storage stores the data information in different databases; when a certain data information needs to be searched, data search provides a retrieval of the database based on the category characteristics of the data information to be searched and feeds the result back to the business personnel; the cloud lake system can ensure the visualization, availability, and operability of the data information.

[0005] Preferably, the system categories to which the data belongs are mainly divided into: horizontal transportation system, production operation system and production auxiliary system, intelligent building system and intelligent office system, and intelligent security system.

[0006] Furthermore, the data generated by the horizontal transportation system, production operation system and production auxiliary system, intelligent building system and intelligent office system, and intelligent security system can be divided into four categories: horizontal transportation data, production application data, equipment data, and video image data.

[0007] Preferably, there are five categories of databases for storing data information, namely ES database, Oracle database, MySQL database, HBase database and file system; the horizontal transportation data is stored in the ES database; the production application data is stored in the Oracle database and MySQL database; the equipment data is stored in the HBase database, Oracle database or MySQL database; the video image data is stored in the file system.

[0008] Preferably, the data search function is completed based on the components or engines of the Yunhu system; the main components or engines include data warehouse components, interactive SQL engines, real-time OLAP components, distributed in-memory computing engines, distributed graph databases, full-text retrieval components and full-text retrieval components.

[0009] Preferably, the data search mainly includes the following steps:

[0010] S1. Determine the data source of the data information to be searched through the retrieval program;

[0011] S2. Determine the data category to which the data information to be searched belongs based on the data source;

[0012] S3. Determine the database category to which the data information to be searched belongs based on the data category;

[0013] S4. Search for the data information to be searched in the database to which the data information to be searched belongs through the search program.

[0014] Among them, the retrieval program determines the data source of the data information to be searched is to determine which system the data to be searched belongs to. Among them, the data source of the data information to be searched is four types of systems, namely the horizontal transportation system, the production operation system and the production assistance system, the intelligent building system and the intelligent office system, and the video image system.

[0015] Among them, determining the data category to which the data information to be searched belongs based on the data source is achieved in the following way:

[0016] If the data source is the horizontal transportation system, the data category to which the data information to be searched belongs is horizontal transportation data; if the data source is the production operation system and the production assistance system, the data category to which the data information to be searched belongs is production application data; if the data source is the intelligent building system and the intelligent office system, the data category to which the data information to be searched belongs is equipment data; if the data source is the video image system, the data category to which the data information to be searched belongs is video image data.

[0017] Furthermore, determining the database category to which the data information to be searched belongs based on the data category is achieved in the following way:

[0018] If the data category is horizontal transportation data, the database category to which the data information to be searched belongs is the ES database; if the data category is production application data, the database categories to which the data information to be searched belongs are the Oracle database and the MySQL database; if the data category is building data among the equipment - type data, the database categories to which the data information to be searched belongs are the Oracle database or the MySQL database; if the data category is other data of the equipment - type data, the database category to which the data information to be searched belongs is the HBase database; if the database category is video image data, the database category to which the data information to be searched belongs is the file system.

[0019] Further, search for the data information to be searched in the database to which the data information to be searched belongs, which specifically includes the following steps:

[0020] If the data information to be searched belongs to relational databases, namely the Oracle database and the MySQL database, the data warehouse component, the interactive SQL engine, the real - time OLAP component, the distributed in - memory computing engine, and the distributed graph database search for the data information to be searched; if the data information to be searched does not belong to relational databases, the full - text retrieval component searches for the information to be searched.

[0021] Preferably, the data application function should clearly display the data asset details.

[0022] Further, the data asset details include the data asset source, the data asset user information, the data asset usage method and effect; the data asset details visualize the data, facilitating business personnel to understand the full - cycle activity process of the data information.

[0023] Preferably, the data application function should ensure the stability of the data collection and processing process.

[0024] Further, the collected data content should be available and stable, the data content should be unambiguous, meeting the standards and quality requirements required by the business; the data asset should be easy to read, enabling business personnel to understand and easily search.

[0025] Preferably, the data application function should ensure the maintainability of the data asset.

[0026] Further, the data asset maintenance includes weighing the construction cost of the data asset and quickly applying new data dimensions; to ensure data operation, quality quantification management models and value quantification management models can be used to quantify the management of data.

[0027] The present invention realizes the management of container terminal data information by the cloud lake system. The cloud lake system classifies the data information according to the system category to which the data information belongs, and then stores the data information in different databases according to the data information classification. When business personnel need to retrieve a certain data information, the cloud lake system retrieves the database based on the category characteristics of the data information to be searched. At the same time, the cloud lake system has a data application function, and this function has characteristics such as data visualization, data availability, and data operability.

[0028] Compared with the prior art, the present invention can converge and store all structured / semi-structured / unstructured data in the container terminal, realize the data lake architecture, build warehouses in the lake, provide real-time OLAP capabilities, achieve data without leaving the lake, shorten the data analysis link, directly connect to business such as BI reports and interactive queries, and solve the problem of data islands. Equipped with components such as a data warehouse component, an interactive SQL engine, and a real-time OLAP component, it has capabilities such as "T+0" data real-time entry into the lake, batch processing, real-time stream processing, stream-batch integration, real-time retrieval, and interactive query, meeting the full-scenario business requirements of automated container terminals. Brief Description of the Drawings

[0029] Figure 1 It is the overall process of the cloud lake architecture management method for the full-automated container terminal production business of the present invention.

[0030] Figure 2 It is the data platform framework planning of the cloud lake architecture management method for the full-automated container terminal production business of the present invention.

[0031] Figure 3 It is the search process of the data search function of the cloud lake architecture management method for the full-automated container terminal production business of the present invention. Detailed Embodiment

[0032] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. The described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments.

[0033] The cloud lake architecture management method for the full-automated container terminal production business provided by the present invention uses a cloud lake system to manage a large amount of data information of the container terminal, and can realize functions such as data classification, data storage, data search, and data application; the data classification function classifies according to the system category to which the data belongs; the data storage function stores the data in different databases according to the data classification; the data search function retrieves the database based on the category characteristics of the data to be searched; in addition, the cloud lake system can ensure the visualization, availability, and operability of the data information.

[0034] Such as Figure 1As shown, the overall process of the cloud lake architecture management method for the production operations of a fully automated container terminal in the present invention includes the following steps.

[0035] S1. Overall process;

[0036] S11. All the data collected from various systems is uploaded to the raw database, and the raw database classifies the uploaded data through a classification program;

[0037] S12. The classified data is respectively transmitted to different databases. Horizontal transportation data is stored in the ES database, production application data is stored in the Oracle database and the MySQL database, equipment - type data is stored in the HBase database or the Oracle database and the MySQL database, and video image data is stored in the file system;

[0038] S121. Determine whether the data belongs to horizontal transportation data. If so, go to S122; if not, go to S123;

[0039] S122. The data is stored in the ES database, and after the operation is completed, go to S13;

[0040] S123. Determine whether the data belongs to production application data. If so, go to S124; if not, go to S125;

[0041] S124. The data is stored in the Oracle and MySQL databases, and after the operation is completed, go to S13;

[0042] S125. Determine whether the data belongs to equipment - type data. If so, go to S126; if not, go to S127;

[0043] S126. The data is stored in the HBase database or the Oracle and MySQL databases, and after the operation is completed, go to S13;

[0044] S127. The data belongs to video image data and is stored in the file system, and after the operation is completed, go to S13;

[0045] S13. Connect the databases to the components or engines of the data lake system, and retrieve the data to be searched in the databases according to the data category characteristics;

[0046] S14. The cloud lake system follows the criteria of data visualization, data availability, and data operability to maintain the stability of the data collection process and the data processing process.

[0047] The cloud lake system management method of the embodiments of the present invention sequentially realizes the functions of data classification, data storage, data search, and data application. It can not only effectively manage data information, but also improve the data management level in terms of data retrieval speed, data storage capacity, etc., thereby improving the efficiency of operations such as port operation scheduling and task arrangement.

[0048] In the above technical solution, the system categories to which the data belongs are mainly divided into: horizontal transportation system, production operation system and production auxiliary system, intelligent building system and intelligent office system, and intelligent security system.

[0049] In the above technical solution, the data generated by the horizontal transportation system, production operation system and production auxiliary system, intelligent building system and intelligent office system, and intelligent security system can be divided into four categories: horizontal transportation data, production application data, equipment - type data, and video image data.

[0050] In the embodiments of the present invention, through the above - mentioned data classification, the orderliness and integrity of data information can be ensured, which is convenient for subsequent data storage.

[0051] In the above technical solution, the horizontal transportation data is stored in the ES database; the production application data is stored in the Oracle database and the MySQL database; the equipment - type data is stored in the HBase database, the Oracle database, or the MySQL database; the video image data is stored in the file system.

[0052] In the embodiments of the present invention, through the diversification of databases, the orderliness of data storage can be ensured, the data format and data information can be accurately stored, and the data retrieval speed can also be accelerated.

[0053] As Figure 2 shown, the data platform framework planning of the full - automation container terminal production business cloud lake architecture management method of the present invention is used to sort out the corresponding relationship between the types of data generated by six different systems of the automation container terminal and the databases in which the data is stored. The specific content is as follows.

[0054] S2. Data platform framework planning;

[0055] S21. The horizontal transportation system of the automation container terminal generates data, the data type is horizontal transportation data, and the horizontal transportation data is stored in the ES database;

[0056] S22. The production operation system of the automation container terminal generates data, the data type is production data, and the production auxiliary system of the automation container terminal generates data, the data type is application data. These two types of data together form production application data, and the production application data is stored in the Oracle database and the MySQL database;

[0057] S23. The intelligent office system of the automated container terminal generates data, and the data type is equipment data. The intelligent building system of the automated container terminal generates data, and the data type is building data. These two types of data together constitute equipment - type data. The equipment data is stored in the HBase database, and the building data is stored in the Oracle database and the MySQL database;

[0058] S24. The intelligent security system of the automated container terminal generates data, and the data type is video - image data, and stores the video - image data in the file system.

[0059] The Yunhu system data platform framework planning shows the relationship network of systems, data, and databases, which is the premise of data information management in the automated terminal, and also the premise of the orderliness of data storage, the storage data format, the accuracy of data information, and the efficiency of retrieval.

[0060] In the above - mentioned technical solution, the data search function is completed based on the components or engines of the Yunhu system; the components or engines mainly include a data warehouse component, an interactive SQL engine, a real - time OLAP component, a distributed in - memory computing engine, a distributed graph database, and a full - text retrieval component. In the embodiments of the present invention, different components or engines retrieve different databases, improving the data retrieval efficiency and ensuring business accuracy.

[0061] In the above - mentioned technical solution, the data search mainly includes the following steps:

[0062] S3. Data search;

[0063] S31. Determine the data source of the data information to be searched through a retrieval program.

[0064] S32. Determine the data category to which the data information to be searched belongs based on the data source.

[0065] S33. Determine the database category to which the data information to be searched belongs based on the data category.

[0066] S34. Search for the data information to be searched in the database to which the data information to be searched belongs through a search program.

[0067] In the above - mentioned technical solution, the data source of the data information to be searched is four types of systems, namely the horizontal transportation system, the production operation system and the production auxiliary system, the intelligent building system and the intelligent office system, and the video - image system. The retrieval program detects which of the above - mentioned systems the data source of the data information to be searched is.

[0068] In the above - mentioned technical solution, the method of determining the data category to which the data information to be searched belongs based on the data source is as follows:

[0069] If the data source is a horizontal transportation system, the data category to which the data information to be searched belongs is horizontal transportation data. If the data source is a production operation system and a production assistance system, the data category to which the data information to be searched belongs is production application data. If the data source is a smart building system and a smart office system, the data category to which the data information to be searched belongs is equipment data.

[0070] If the data source is a video image system, the data category to which the data information to be searched belongs is video image data.

[0071] In the above technical solution, the retrieval program determines the database category to which the data information to be searched belongs through the data category, and is implemented in the following manner:

[0072] If the data category is horizontal transportation data, the database category to which the data information to be searched belongs is the ES database. If the data category is production application data, the database categories to which the data information to be searched belongs are the Oracle database and the MySQL database. If the data category is building data in the equipment data, the database category to which the data information to be searched belongs is the Oracle database or the MySQL database; if the data category is other data in the equipment data, the database category to which the data information to be searched belongs is the HBase database. If the database category is video image data, the database category to which the data information to be searched belongs is the file system.

[0073] In the above technical solution, the search program searches for the data information to be searched in the database to which the data information to be searched belongs, and specifically adopts the following method:

[0074] If the data information to be searched belongs to a relational database, that is, the Oracle database and the MySQL database, the data warehouse component, the interactive SQL engine, the real-time OLAP component, the distributed memory computing engine, and the distributed graph database search for the data information to be searched. If the data information to be searched does not belong to a relational database, the full-text retrieval component searches for the information to be searched.

[0075] As Figure 3 shown, the fully automated container terminal production business cloud lake architecture management method of the present invention, its data search process mainly includes the following steps.

[0076] S4. Search process;

[0077] S41. Whether the data source is a horizontal transportation system. If so, go to S42; if not, go to S45;

[0078] S42. The data belongs to horizontal transportation data;

[0079] S43. The horizontal transportation data is stored in the ES database;

[0080] In S44, the ES database uses a full-text retrieval component to search the information to be searched, and after the operation is completed, it proceeds to S419;

[0081] In S45, check if the data source is the intelligent building system. If so, proceed to S47; if not, proceed to S46;

[0082] In S46, check if the data source is the intelligent office system. If so, proceed to S47; if not, proceed to S410;

[0083] In S47, the data belongs to device - type data;

[0084] In S48, the device - type data is stored in an Oracle, MySQL database, or HBase database;

[0085] In S49, the HBase database uses a full - text retrieval component to search the information to be searched. The Oracle and MySQL databases use a data warehouse component, an interactive SQL engine, a real - time OLAP component, a distributed in - memory computing engine, and a distributed graph database to search the data information to be searched. After the operation is completed, proceed to S419;

[0086] In S410, check if the data source is the intelligent security system. If so, proceed to S411; if not, proceed to S414;

[0087] In S411, the data belongs to video image data;

[0088] In S412, the video image data is stored in a file system;

[0089] In S413, the file system uses a full - text retrieval component to search the information to be searched;

[0090] In S414, check if the data source is the production operating system. If so, proceed to S416; if not, proceed to 415;

[0091] In S415, the data source is the production auxiliary system;

[0092] In S416, the data belongs to production application data;

[0093] In S417, the production application data is stored in an Oracle or MySQL database;

[0094] In S418, the Oracle and MySQL databases use a data warehouse component, an interactive SQL engine, a real - time OLAP component, a distributed in - memory computing engine, and a distributed graph database to search the data information to be searched. After the operation is completed, proceed to S419;

[0095] In S419, end the data search.

[0096] In the embodiments of the present invention, for the data search, based on the data source, data category, and database category, different components or engines are used to search in different databases, so as to improve the data information retrieval speed and the efficiency of the automated terminal scheduling operation.

[0097] In the above technical solution, the data application function should clearly display the data asset details, ensure the stability of the data collection and processing process, and ensure the maintainability of the data assets.

[0098] Among them, the data asset details include the data asset source, data asset user information, data asset usage methods and effects, reflecting data visualization, which is a window for business personnel to observe the full-cycle activities of data information.

[0099] Among them, the stability of the data collection and processing process includes that the data content is stable and understandable, the data assets are open quickly and conveniently, reflecting data availability, which is a prerequisite for business personnel to quickly search and directly read.

[0100] Among them, the data asset maintenance includes weighing the construction cost of the data assets and quickly applying new data dimensions, reflecting data operability, which is a guarantee for business personnel to maintain the continuous optimization of data asset management.

[0101] In the embodiments of the present invention, data application enables data information to have the characteristics of data visualization, data availability, and data operability:

[0102] Among them, data visualization is specifically manifested in aspects such as metadata visualization management, data asset category visualization management, platform business data source visualization management, data modeling visualization management, and data consumer visualization management.

[0103] Among them, data availability is specifically manifested in aspects such as the data content being unambiguous and meeting the standard and quality requirements, the data tasks having operation and maintenance capabilities, and the assets and asset categories being easy to find and understandable.

[0104] Among them, data operability is manifested in that optimization management and data operation can be realized according to the quality quantification model and value quantification model, and data quality optimization and value mining can be achieved.

[0105] Finally, it should be noted that the above embodiments are only used for exemplifying and explaining the present invention, and are not intended to limit the present invention to the scope of the described embodiments. In addition, those skilled in the art can understand that the present invention is not limited to the above embodiments, and more variations and modifications can be made according to the teachings of the present invention, and these variations and modifications all fall within the scope claimed by the present invention.

Claims

1. A method for managing the cloud lake architecture of the production operations of a fully automated container terminal, characterized in that, The Yunhu system is adopted to manage the massive data information of a container terminal, realizing data classification, data storage, data search, and data application. Data classification classifies the collected data information according to the system categories to which the data information belongs. Data storage stores the data information in different databases. When a certain data information needs to be searched, data search provides a retrieval of the database based on the category characteristics of the data information to be searched and feeds the results back to the business personnel. The Yunhu system can ensure the visualization, availability, and operability of the data information. The system categories to which the data belongs are divided into: horizontal transportation system, production operation system and production auxiliary system, intelligent building system and intelligent office system, and intelligent security system. The data generated by the horizontal transportation system, production operation system and production auxiliary system, intelligent building system and intelligent office system, and intelligent security system can be divided into four categories: horizontal transportation data, production application data, equipment data, and video image data. The horizontal transportation data is stored in the ES database. The production application data is stored in the Oracle database and the MySQL database. The equipment data is stored in the HBase database, the Oracle database, or the MySQL database. The video image data is stored in the file system. For the data search, the data source, data category, and database category are used as the basis, and different components or engines are used to search in different databases. The data search is completed based on the components or engines of the Yunhu system. The components or engines include a data warehouse component, an interactive SQL engine, a real-time OLAP component, a distributed memory computing engine, a distributed graph database, and a full-text retrieval component. The data search includes the following steps: S1. Determine the data source of the data information to be searched through a retrieval program. S2. Determine the data category to which the data information to be searched belongs based on the data source. S3. Determine the database category to which the data information to be searched belongs based on the data category. S4. Search for the data information to be searched in the database to which the data information to be searched belongs through a search program.

2. The method for managing the cloud lake architecture of the production operations of a fully automated container terminal according to claim 1, characterized in that, Determine the data source of the data information to be searched through a retrieval program and determine which system the data information to be searched belongs to. Among them, the data source of the data information to be searched is divided into four systems, namely, the horizontal transportation system, the production operation system and the production auxiliary system, the intelligent building system and the intelligent office system, and the video image system.

3. The method for managing the cloud lake architecture of the production operations of a fully automated container terminal according to claim 2, characterized in that, Determine the data category to which the data information to be searched belongs based on the data source, and implement it in the following way: If the data source is the horizontal transportation system, the data category to which the data information to be searched belongs is horizontal transportation data; if the data source is the production operation system and the production auxiliary system, the data category to which the data information to be searched belongs is production application data; if the data source is the intelligent building system and the intelligent office system, the data category to which the data information to be searched belongs is equipment data; if the data source is the video image system, the data category to which the data information to be searched belongs is video image data.

4. The method for managing the cloud lake architecture of the production operations of a fully automated container terminal according to claim 3, characterized in that, Determine the database category to which the data information to be searched belongs based on the data category, and implement it as follows: If the data category is horizontal transportation data, the database category to which the data information to be searched belongs is the ES database; if the data category is production application data, the database categories to which the data information to be searched belongs are the Oracle database and the MySQL database; if the data category is building data among the equipment - type data, the database categories to which the data information to be searched belongs are the Oracle database or the MySQL database; if the data category is other data of the equipment - type data, the database category to which the data information to be searched belongs is the HBase database; if the database category is video image data, the database category to which the data information to be searched belongs is the file system.

5. The method for managing the cloud lake architecture of the production operations of a fully automated container terminal according to claim 4, characterized in that, Search for the data information to be searched in the database to which the data information to be searched belongs, which specifically includes the following steps: If the data information to be searched belongs to a relational database, that is, the Oracle database and the MySQL database, use the data warehouse component, the interactive SQL engine, the real - time OLAP component, the distributed in - memory computing engine, and the distributed graph database to search for the data information to be searched; if the data information to be searched does not belong to a relational database, use the full - text retrieval component to search for the information to be searched.

6. The method for managing the cloud lake architecture of the production operations of a fully automated container terminal according to claim 5, characterized in that, The data application includes clearly presenting the data asset rules to ensure the stability of the data collection and processing process and the maintainability of the data assets; the data asset rules include the data asset source, the data asset user information, the data asset usage method and effect; the stability of the data collection and processing process includes stable and understandable data content and fast and convenient data asset opening; the data asset maintenance includes weighing the construction cost of the data assets and quickly applying new data dimensions.

Images