Real-time data warehouse method and system for lithium battery manufacturing industry

Abstract

This invention provides a real-time data warehouse method and system for the lithium battery manufacturing industry. The method includes: using Flink streams and message queues to form a data acquisition module, labeling the collected data, distinguishing data from different libraries and tables, encapsulating the data from different libraries and tables into a unified custom data format, and caching it in a unique topic in the message queue; and using Flink SQL, data lake, and data warehouse to form a data writing module. Using Flink SQL, data from different libraries and tables is decoupled based on data labels, and written to different tables in the data lake Hudi. Data updates and deletions are supported, and the system is automatically synchronized with the data warehouse Hive. This invention solves the technical problems of limited data processing scale, poor real-time data processing performance, and excessive system load.

Description

Technical Field

[0001] The present invention relates to the field of lithium battery production data processing, and in particular to a real-time data warehouse method and system for the lithium battery manufacturing industry. Background Art

[0002] With the continued use of traditional energy sources such as oil, coal, and natural gas, it is crucial to better utilize lithium batteries. The lithium battery manufacturing industry generates a large amount of data every day. Using big data technology to create value and serve the lithium battery manufacturing industry can fully realize the value of lithium batteries.

[0003] Traditional lithium battery manufacturing data is typically stored in relational databases, such as MySQL, Oracle, and SQL Server. For example, the existing invention patent application, publication number CN105244559A, titled "A Method for Detecting Comprehensive Temperature Performance of Lithium-ion Batteries," includes establishing a first / second relational database for each ratio range of discharge / charge capacity to initial discharge / charge capacity, and each ratio range of discharge / charge energy to initial discharge / charge energy, and corresponding preset values ​​for discharge / charge performance at multiple high and low temperatures. Another example is the existing invention patent application, publication number CN110716140A, titled "A Method for Detecting Electric Vehicle Lithium Battery Capacity Based on Temperature Measurement." This method considers the impact of temperature and depth of discharge on battery internal resistance and first establishes a relational database for internal resistance R, depth of discharge DOD, and temperature T. The method then measures the operating temperature and depth of discharge of the lithium battery in real time. The two data are then compared in the established database to determine the internal resistance corresponding to the two data. The total battery capacity is then determined based on the determined internal resistance, ultimately yielding the real-time remaining battery capacity. The aforementioned existing technologies cannot accommodate massive amounts of data, nor can they support some high-concurrency operations. Alternatively, they use offline data warehouses to store massive amounts of data. Relying on HDFS and adopting a distributed architecture, although they can store massive amounts of lithium battery data, they cannot update and insert lithium battery usage data in real time, cannot meet real-time requirements, and cannot provide timely feedback on lithium battery usage, which in turn affects the decision-making and processing of lithium batteries by personnel in various departments.

[0004] In recent years, some companies have begun using Flink SQL CDC technology to implement real-time data insertion, update, and deletion. However, as the number of database tables increases, the CPU and I / O pressure on the business database of the data source will be too high. This method is not suitable for the booming lithium battery manufacturing industry.

[0005] In summary, the existing technology has technical problems such as limited data processing scale, poor real-time data processing and excessive system load. Summary of the Invention

[0006] The technical problem to be solved by the present invention is how to solve the technical problems in the prior art of limited data processing scale, poor real-time performance of data processing and excessive system load.

[0007] The present invention solves the above technical problems by adopting the following technical solutions: A real-time data warehouse method for the lithium battery manufacturing industry includes:

[0008] S1, a data acquisition module is constructed using Flink streams and message queues to collect battery manufacturing data;

[0009] S2. Add library and table labels to the battery manufacturing data to distinguish the library and table to which the battery manufacturing data belongs;

[0010] S3. Encapsulate battery manufacturing data from different libraries and tables into unified custom format data based on library and table tags;

[0011] S4. Cache the unified custom format data to the corresponding topic of the message queue;

[0012] S5. Decouple the content of the corresponding topic in the message queue using Flink SQL. Perform SQL filtering based on the library table tag of each battery manufacturing data to separate the data in a unified custom format as the decoupled battery manufacturing data.

[0013] S6. Use the data writing module to write the decoupled battery manufacturing data into the difference table of the data lake Hudi, and synchronize the decoupled battery data to the pre-installed data warehouse Hive.

[0014] This paper improves upon existing data storage technology in the lithium battery manufacturing industry by collecting data from multiple databases and tables using Flink streaming, reducing CPU and I / O pressure on the data source database. Data tagging and unified data management avoid the problem of excessive message queue topics. By using the emerging data lake technology Hudi as an intermediary between message queues and the Hive data warehouse, this solves the problem of Hive's lack of support for real-time updates and deletions.

[0015] In a more specific technical solution, step S1 includes:

[0016] S11. Open the archive log of the preset business database to record the operation changes of the lithium battery data;

[0017] S12. Use Flink Streaming to collect data from the data source business database of no less than two libraries and no less than two tables as battery manufacturing data.

[0018] This invention uses Flink streams and message queues to form a data collection module, reducing the load on the data source business database caused by multi-database and multi-table data collection through Flink SQL. The collected data is labeled to distinguish data from different libraries and tables. The data from different libraries and tables is then encapsulated into a unified custom data format and cached in a unique topic in the message queue. This reduces the load on the server caused by an excessive number of topics and avoids the problem of excessive server memory and disk usage caused by too many topics. This invention collects data from multiple libraries and tables through Flink streams, reducing the CPU and I / O pressure on the data source database.

[0019] In a more specific technical solution, in step S2, the label includes but is not limited to: the subordinate database name and subordinate table name of each battery manufacturing data.

[0020] In a more specific technical solution, step S3 includes: unifying the format of the custom format data including: lithium battery database name, lithium battery data table name, field name and data operation type.

[0021] In a more specific technical solution, in step S5, Flink SQL, data lake Hudi, and data warehouse are integrated to build a data writing module.

[0022] In a more specific technical solution, in step S5, Flink SQL is used to decouple and process unified custom format data from different libraries and tables according to library and table tags to obtain decoupled battery manufacturing data.

[0023] In a more specific technical solution, the following logic is used to decouple and process unified custom format data in different libraries and tables:

[0024] .

[0025] In a more specific technical solution, step S6 includes:

[0026] S61. Modify the Hive version in the source code of Data Lake Hudi to adjust the Hive version to the cluster version, and compile Data Lake Hudi based on it to obtain the integrated package of Data Lake Hudi and Data Warehouse Hive.

[0027] S62. Put the integration package into the preset directory of Hive in the data warehouse;

[0028] S63. Restart the data storage service of the data warehouse Hive.

[0029] The present invention adopts Flink SQL + data lake + data warehouse to form a data writing module. Through Flink SQL, data in different libraries and tables are decoupled according to data labels, and then written to different tables in the data lake Hudi. Data updates and deletions are supported and automatically synchronized to the data warehouse Hive, solving the problem that Hive data does not support real-time updates and deletions. It combines the high performance and management capabilities of the data warehouse with the flexibility of the data lake, enabling mutual sharing of data and solving the problem that the data warehouse Hive does not support real-time updates and deletions.

[0030] In a more specific technical solution, in step S61, the data lake Hudi is compiled using the following logic to obtain an integrated package;

[0031] .

[0032] In a more specific technical solution, the real-time data warehouse system for the lithium battery manufacturing industry includes:

[0033] Flink stream data collection module, which is used to form a data collection module using Flink streams and message queues to collect battery manufacturing data;

[0034] The tagging module is used to tag the battery manufacturing data with library and table labels to distinguish the library and table to which the battery manufacturing data belongs. The tagging module is connected to the Flink stream data acquisition module.

[0035] The format unification module is used to encapsulate the battery manufacturing data of different libraries and tables into a unified custom format data according to the library table labels. The format unification module is connected to the labeling module;

[0036] A cache module is used to cache the unified custom format data to the corresponding topic of the message queue. The cache module is connected to the format unification module;

[0037] The data decoupling module is used to decouple the content of the corresponding topics in the message queue through Flink SQL. SQL filtering is performed based on the library table tag of each battery manufacturing data to separate the data in a unified custom format as the decoupled battery manufacturing data. The data decoupling module is connected to the cache module.

[0038] The write synchronization operation module is used to use the data write module to write the decoupled battery manufacturing data into the difference table of the data lake Hudi, and synchronize the decoupled battery data to the preset data warehouse Hive. The write synchronization operation module is connected to the data decoupling module.

[0039] This invention offers the following advantages over existing technologies: It improves upon existing data storage technology in the lithium battery manufacturing industry by collecting data from multiple databases and tables through Flink streaming, reducing CPU and I / O pressure on the data source database. Data tagging and unified data management avoid the problem of excessive message queue topics. By using the emerging data lake technology Hudi as a "medium" between message queues and the Hive data warehouse, this solves the problem of Hive's lack of support for real-time updates and deletions.

[0040] This invention uses Flink streams and message queues to form a data collection module, reducing the load on the data source business database caused by multi-database and multi-table data collection through Flink SQL. The collected data is labeled to distinguish data from different libraries and tables. The data from different libraries and tables is then encapsulated into a unified custom data format and cached in a unique topic in the message queue. This reduces the load on the server caused by an excessive number of topics and avoids the problem of excessive server memory and disk usage caused by too many topics. This invention collects data from multiple libraries and tables through Flink streams, reducing the CPU and I / O pressure on the data source database.

[0041] This paper adopts Flink SQL + data lake + data warehouse to form a data writing module. Through Flink SQL, data in different libraries and tables are decoupled according to data labels, and then written to different tables in the data lake Hudi. It supports data update and deletion and automatically synchronizes it to the data warehouse Hive, solving the problem that Hive data does not support real-time update and deletion. It combines the high performance and management capabilities of the data warehouse with the flexibility of the data lake, enabling mutual sharing of data and solving the problem that the data warehouse Hive does not support real-time update and deletion.

[0042] The present invention solves the technical problems existing in the prior art of limited data processing scale, poor real-time performance of data processing and excessive system load. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a schematic diagram of the basic steps of the real-time data warehouse method for the lithium battery manufacturing industry according to Example 1 of the present invention;

[0044] Figure 2 This is a schematic diagram of data stream processing in a real-time data warehouse system for the lithium battery manufacturing industry according to Example 1 of the present invention;

[0045] Figure 3 This is a schematic diagram of the specific steps of integrating Hudi with Hive in Example 1 of the present invention. DETAILED DESCRIPTION

[0046] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0047] Example 1

[0048] like Figure 1 and Figure 2 As shown, the real-time data warehouse method for the lithium battery manufacturing industry provided by the present invention includes the following basic steps:

[0049] Step S1: Use Flink stream and message queue to form a data acquisition module;

[0050] In this embodiment, the business database archive log is enabled to record the operational changes of lithium battery data, and Flink Streaming is used to collect data from the business database of multiple databases and tables.

[0051] Step S2: Label the collected data and distinguish the data in different libraries and tables;

[0052] In this embodiment, lithium battery data in different databases and tables are marked with different tags. In this embodiment, the tags include but are not limited to: the database name and table name to which each piece of data belongs, so as to facilitate subsequent data decoupling.

[0053] Step S3: Encapsulate the data in different libraries and tables into a unified custom data format;

[0054] In this embodiment, since the data in multiple libraries and tables are messy and not convenient for unified data processing, each piece of data is converted into a unified data format for management. In this embodiment, the unified data format includes but is not limited to: the library name, table name and field name to which each piece of lithium battery data belongs, and the operation type of the data.

[0055] Step S4: The unified data is then stored in a unique topic in the message queue. In this embodiment, the number of topics is set according to the actual manufacturing scenario.

[0056] Step S5: Use Flink SQL, data lake, and data warehouse to form a data writing module. Use Flink SQL to decouple data from different libraries and tables based on data labels.

[0057] In this embodiment, Flink SQL is used to decouple the topic content in the message queue, and SQL filtering is performed based on the different library and table tags on each data piece, thereby separating the lithium battery data from different libraries and tables:

[0058]

[0059] Step S6: Write data to different tables in the data lake Hudi to update and delete data, and automatically synchronize it to the data warehouse Hive;

[0060] In this embodiment, the separated lithium battery data is written into different Hudi tables. As an emerging technology, Hudi is difficult to get started with. Its biggest advantage is that it supports real-time update and deletion of data. However, it cannot perform efficient hierarchical management of data like the data warehouse Hive. Hive itself does not support update and deletion of data. By integrating Hudi with Hive, the changed data in Hudi will be synchronized to Hive, thereby indirectly realizing real-time addition, update and deletion of Hive data.

[0061] like Figure 3 As shown, in this embodiment, in step S6, the specific steps of Hudi integration Hive operation include:

[0062] Step S61: Modify the Hive version in the Hudi source code to the Hive version used in the cluster, and compile Hudi. In this embodiment, the compilation command will be different for different component versions.

[0063]

[0064] In this embodiment, the meanings of the parameters are as follows:

[0065] -Dmaven.test.skip=true means skipping the running of unit tests and the compilation of test code;

[0066] -Dcheckstyle.skip=true means skipping the checkstyle check;

[0067] -Drat.skip=true skips license check;

[0068] -Dspark2.4 indicates that when compiling Hudi this time, the Spark version specified is 2.4.X;

[0069] -Dflink1.13 indicates that when compiling Hudi this time, the Flink version specified is 1.13.X;

[0070] -Dscala-2.11 indicates that when compiling Hudi this time, the corresponding Scala version is specified as 2.11.X;

[0071] -Dhadoop.version=3.0.0 means that when compiling Hudi this time, the Hadoop version specified is 3.0.0;

[0072] -Pflink-bundle-shade-hive2 indicates that when compiling Hudi this time, the Hive version specified is 2.X.

[0073] Step S62: Put the obtained Hudi and Hive integration package into the auxlib directory at the same level as Hive's lib directory;

[0074] Step S63: Restart the Hive Metastore and Hiveserver2 services.

[0075] In summary, this invention improves existing data storage technology in the lithium battery manufacturing industry. By using Flink streaming to collect data from multiple databases and tables, it reduces CPU and I / O pressure on the data source database. By tagging and centralizing data management, it avoids the problem of excessive message queue topics. By using the emerging data lake technology Hudi as an intermediary between message queues and the Hive data warehouse, it addresses the issue of Hive's lack of support for real-time updates and deletions.

[0076] This invention uses Flink streams and message queues to form a data collection module, reducing the load on the data source business database caused by multi-database and multi-table data collection through Flink SQL. The collected data is labeled to distinguish data from different libraries and tables. The data from different libraries and tables is then encapsulated into a unified custom data format and cached in a unique topic in the message queue. This reduces the load on the server caused by an excessive number of topics and avoids the problem of excessive server memory and disk usage caused by too many topics. This invention collects data from multiple libraries and tables through Flink streams, reducing the CPU and I / O pressure on the data source database.

[0077] This paper adopts Flink SQL + data lake + data warehouse to form a data writing module. Through Flink SQL, data in different libraries and tables are decoupled according to data labels, and then written to different tables in the data lake Hudi. It supports data update and deletion and automatically synchronizes it to the data warehouse Hive, solving the problem that Hive data does not support real-time update and deletion. It combines the high performance and management capabilities of the data warehouse with the flexibility of the data lake, enabling mutual sharing of data and solving the problem that the data warehouse Hive does not support real-time update and deletion.

[0078] The present invention solves the technical problems existing in the prior art of limited data processing scale, poor real-time performance of data processing and excessive system load.

[0079] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A real-time data warehouse method for the lithium battery manufacturing industry, characterized by: The method comprises: S1, a data acquisition module is constructed using Flink streams and message queues to collect battery manufacturing data; S2. Add library and table labels to the battery manufacturing data to distinguish the library and table to which the battery manufacturing data belongs; S3. Encapsulate battery manufacturing data from different libraries and tables into unified custom format data based on library and table tags; S4. Cache unified custom format data to a unique topic in the message queue; S5. Decouple the content of the corresponding topic in the message queue using Flink SQL. Perform SQL filtering based on the library table tag of each battery manufacturing data to separate the data in a unified custom format as the decoupled battery manufacturing data. S6. Use the data writing module to write the decoupled battery manufacturing data into the difference table of the data lake Hudi and synchronize the decoupled battery data to the pre-installed data warehouse Hive. S6 includes: S61, modifying the Hive version in the source code of the data lake Hudi to adjust the Hive version to the cluster version, and compiling the data lake Hudi based on it to obtain an integrated package of the data lake Hudi and the data warehouse Hive; S62. Put the integration package into the preset directory of Hive in the data warehouse; S63. Restart the data storage service of the data warehouse Hive.

2. The real-time data warehouse method for the lithium battery manufacturing industry according to claim 1 is characterized in that: The step S1 comprises: S11. Open the archive log of the preset business database to record the operation changes of the lithium battery data; S12. Collect data from no less than two libraries and no less than two tables of the data source business library through Flink Streaming to serve as the battery manufacturing data.

3. The real-time data warehouse method for lithium battery manufacturing industry according to claim 1 is characterized in that: In step S2, the label includes: the subordinate database name and subordinate table name of each piece of battery manufacturing data.

4. The real-time data warehouse method for the lithium battery manufacturing industry according to claim 1 is characterized in that: The step S3 includes: the format of the unified custom format data includes: lithium battery database name, lithium battery data table name, field name and data operation type.

5. The real-time data warehouse method for lithium battery manufacturing industry according to claim 1 is characterized in that: In step S5, Flink SQL, the data lake Hudi, and the data warehouse are integrated to construct a data writing module.

6. The real-time data warehouse method for lithium battery manufacturing industry according to claim 1 is characterized in that: In step S5, the unified custom format data of different libraries and different tables are decoupled and processed according to the library and table tags through Flink SQL to obtain decoupled battery manufacturing data.

7. The real-time data warehouse method for lithium battery manufacturing industry according to claim 6 is characterized in that: The following logic is used to decouple the processing of the unified custom format data in different libraries and tables: 。 8. The real-time data warehouse method for lithium battery manufacturing industry according to claim 1 is characterized in that: In step S61, the data lake Hudi is compiled using the following logic to obtain the integration package; 。 9. A real-time data warehouse system for the lithium battery manufacturing industry, configured to execute the real-time data warehouse method for the lithium battery manufacturing industry as claimed in any one of claims 1 to 8, characterized in that: The system comprises: Flink stream data collection module, which is used to form a data collection module using Flink streams and message queues to collect battery manufacturing data; a labeling module, configured to label the battery manufacturing data with a library table label to distinguish the library table to which the battery manufacturing data belongs, and connected to the Flink stream data acquisition module; A format unification module, configured to encapsulate the battery manufacturing data of different libraries and different tables into unified custom format data according to the library table labels, wherein the format unification module is connected to the marking module; A cache module, used for caching the unified custom format data to a corresponding topic of a message queue, the cache module being connected to the format unification module; A data decoupling module is used to decouple the content of the corresponding topic in the message queue through Flink SQL, and perform SQL filtering based on the library table tag of each battery manufacturing data to separate the unified custom format data as decoupled battery manufacturing data. The data decoupling module is connected to the cache module; A write synchronization operation module is used to use the data write module to write the decoupled battery manufacturing data into the difference table of the data lake Hudi, and synchronize the decoupled battery manufacturing data to the preset data warehouse Hive. The write synchronization operation module is connected to the data decoupling module.

Images