Method for constructing data assets for finished lithium battery testing

Abstract

The present invention discloses a method for constructing lithium battery finished product inspection data assets, comprising the following steps: collecting lithium battery finished product test data; using a flink tool to write the test data into a Hudi data lake to generate a Hudi data warehouse, and synchronizing the data to a Hive database to generate a Hive table, wherein the Hive table is synchronized with the table and field attributes of the Hudi data warehouse; adding fields and annotation information to the Hive table to generate a result table synchronized from the Hudi data warehouse to the Hive database; synchronizing the table data generated by processing the test data in the Hudi data warehouse to the result table to generate the lineage relationship of the test data; and using an Atlas tool to record the lineage relationship of the test data. The present invention is suitable for real-time analysis and interactive query, can achieve low query latency, and meet the strict requirements of lithium battery finished product inspection data processing time.

Description

Technical Field

[0001] The present invention relates to the field of data management technology, and in particular to a method for constructing lithium battery finished product inspection data assets. Background Art

[0002] With the rapid development of emerging markets such as electric vehicles, lithium batteries have become a vital component of the energy sector. To ensure the quality and safety of lithium batteries during production, finished product testing is essential. The comprehensive testing of finished lithium batteries generates a vast amount of test data, which is considered a critical business asset and can bring value to the company. In industrial production, analysts own and control data resources through procurement, production, testing, and sales. They can process, analyze, and provide data to optimize industrial production yields. Data resources are resources that can generate expected economic benefits or service potential for industrial production. Faced with the data from different test steps and various disorganized indicators and parameters of finished lithium battery products, it is necessary to transform them into valuable assets. By analyzing and mining this massive amount of data, problems in lithium batteries can be quickly and accurately detected, improving production efficiency and product quality.

[0003] Data asset management is an important indicator for enhancing the practical value of data analysis. When building data assets, we need to consider the scenarios of finished product testing and the goal of reducing failure rates, provide the ability to find data more quickly and maintain asset information more conveniently, and provide guidance for data management in different business domains of lithium batteries. The application scenarios for building data assets can be summarized into two categories:

[0004] (1) Finding data: This is mainly done by data analysts, who analyze and apply data processed in the data warehouse. Faced with disorganized data, data technicians need to first divide the data into different databases and tables and store them in MySQL. A lot of time is spent on sorting out the fields. Business analysts need to consult with technicians or review the provided database documents. Sometimes, although the business logic is correct, an abnormal data may lead to deviation in the results. Business personnel spend a lot of energy to solve the technical problems. In addition, due to the lack of a data asset visualization tool, new business analysts have to repeat these steps.

[0005] To address the issue of not knowing where the required data is, the metadata management tool Atlas can be used to collect information about different libraries, tables, and fields, and use data lineage relationships to discover the logic between tables. For example, the data column-level lineage processing method based on the Hive data warehouse proposed in patent application publication number CN111813796A. However, the Atlas framework currently only collects data from Hive databases. Hive is typically used for batch queries and large-scale data conversion. It converts queries into MapReduce or Tez jobs and processes data in batches. This results in relatively slow query performance, making it unsuitable for real-time analysis and interactive queries. However, finished product inspection data processing places strict requirements on data processing time.

[0006] (2) Data management: This is mainly done by data developers, with the goal of making data more secure and reusable by more people. However, due to the large variety of lithium battery finished product testing instruments and low data reuse, when a large number of assets are added, it is difficult to quickly search for the desired results. Summary of the Invention

[0007] The technical problem to be solved by the present invention is how to provide an asset construction method suitable for finished lithium battery product testing data.

[0008] The present invention solves the above technical problems through the following technical means:

[0009] A method for constructing lithium battery finished product inspection data assets is proposed, the method comprising:

[0010] Collect test data of finished lithium batteries;

[0011] Use the Flink tool to write the test data into the Hudi data lake to generate a Hudi data warehouse and synchronize it to the Hive database to generate a Hive table. The Hive table is synchronized with the table and field attributes of the Hudi data warehouse.

[0012] Add fields and annotation information to the Hive table to generate a result table of synchronizing the Hudi data warehouse to the Hive database;

[0013] Synchronize the table data generated by processing the test data in the Hudi data warehouse to the result table to generate the lineage relationship of the test data;

[0014] The Atlas tool is used to record the lineage relationship of the test data.

[0015] Furthermore, adding fields and annotation information to the Hive table to generate a result table of synchronizing the Hudi data warehouse to the Hive database includes:

[0016] Add table comments, field comments, fields, and HDFS addresses to the Hive table to generate a result table of synchronizing the Hudi data warehouse to the Hive database;

[0017] The table annotation is the annotation of the data table in the Hudi data warehouse, and the field annotation is the field annotation of the data table in the Hudi data warehouse;

[0018] The fields include Hudi commit time, Hudi sequence, Hudi primary key, Hudi partition, and Hudi file name.

[0019] Furthermore, synchronizing the table data generated by processing the test data in the Hudi data warehouse to the result table to generate the lineage relationship of the test data includes:

[0020] Convert the Hudi data warehouse layered processing SQL into Hive-SQL;

[0021] The test data is processed in the Hudi data warehouse to generate table data, and the table data is synchronized to the result table of the Hive database according to the Hive-sql specified table data to generate the lineage relationship of the test data.

[0022] Furthermore, the result table is a wide table of detail layers.

[0023] Furthermore, after recording the blood relationship of the test data using the Atlas tool, the method further includes:

[0024] Delete the data in the result table.

[0025] Furthermore, after synchronizing the table data generated by processing the test data in the Hudi data warehouse to the result table and generating the lineage relationship of the test data, the method further includes:

[0026] For related test data, establish a data lineage relationship diagram;

[0027] The Atlas tool is used to record the data lineage relationship diagram for tracking the test data of each link in the lithium battery production process.

[0028] Furthermore, the method further comprises:

[0029] Data asset types corresponding to the lithium battery finished product testing process are added to the Atlas tool, and the data asset types include raw material testing data assets, cell assembly testing data assets, finished product testing data assets, and battery assembly testing data assets.

[0030] Furthermore, after recording the blood relationship of the test data using the Atlas tool, the method further includes:

[0031] The Atlas tool starts the consumer to consume messages, determines the data type of the parameter index based on the lineage relationship of the test data, and stores the parameter index;

[0032] Aggregate the test data and aggregate it into the ES search engine;

[0033] The test data of each link in the lithium battery production process is queried using ES-based data retrieval technology according to the parameter index.

[0034] Furthermore, after collecting the test data of the finished lithium battery, the method further includes:

[0035] Preprocessing the test data to obtain preprocessed data;

[0036] Parameters related to battery performance are extracted from the pre-processed data as feature data, and assets are constructed based on the feature data.

[0037] Furthermore, the method further comprises:

[0038] Configure workspaces for users with different roles based on personal workstations;

[0039] Assign corresponding permissions to different workspaces to display corresponding data asset views based on user roles and permissions.

[0040] The advantages of the present invention are:

[0041] (1) Due to the poor real-time performance of Hive client processing tables, long-term running tasks are not recommended, and are not suitable for processing lithium battery finished product test data. In addition, the table processed in non-Hive data warehouses cannot see the lineage dependency between data in Atlas. In order to solve the problem of generating data lineage display during the external Hudi data warehouse processing process, the present invention establishes a result table synchronized from Hudi to Hive. After the Hive data warehouse generates lineage through one-time processing, the task is normally run in the Hudi data warehouse and output to the Hive result table. In this way, the table data generated by the Hudi data warehouse processing can be normally synchronized to the Hive table and the data lineage dependency can also be displayed in Atlas. It is suitable for real-time analysis and interactive query, can achieve low query latency, and meet the strict requirements of lithium battery finished product test data processing time.

[0042] (2) By establishing a data lineage relationship diagram of related data, it is possible to track the data lineage of each link in the lithium battery production process, so as to better understand the source, changes and impact of the data.

[0043] (3) When querying data, based on the characteristics of lithium battery finished product test data, optimize data indexing and querying, select appropriate query methods to improve query efficiency, and use data aggregation functions to summarize and analyze data in advance. Querying and analyzing dimensions above minutes can save a lot of time.

[0044] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a flow chart of a method for constructing lithium battery finished product inspection data assets proposed in an embodiment of the present invention. DETAILED DESCRIPTION

[0046] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0047] like Figure 1 As shown, an embodiment of the present invention provides a method for constructing lithium battery finished product inspection data assets, the method comprising the following steps:

[0048] S10, collecting test data of finished lithium batteries;

[0049] It should be noted that the test data of lithium battery finished products includes but is not limited to sensor data of the production line, environmental data such as temperature and humidity in each link, and production process data of lithium batteries, etc., which can be collected through Internet of Things technology, sensors, instruments and other means, and stored in the cloud platform or local database.

[0050] S20. Use the flink tool to write the test data into the Hudi data lake to generate a Hudi data warehouse and synchronize it to the Hive database to generate a Hive table. The Hive table is synchronized with the table attributes and field attributes of the Hudi data warehouse.

[0051] It should be noted that the data in the Hudi data lake and the Hive database are both based on the HDFS distributed file system. The Flink distributed file system writes data to the Hudi data lake and synchronizes metadata to the Hive database at the same time.

[0052] S30: Add fields and annotation information to the Hive table to generate a result table of synchronizing the Hudi data warehouse to the Hive database;

[0053] This embodiment can synchronize metadata to the Hive database while writing data to the Hudi data lake, generating an identical data warehouse in Hive. The table properties, field properties, table comments, and field comments of the Hudi tables in the Hudi data warehouse and the Hive tables in the Hive data warehouse are synchronized, allowing the metadata description of the Hudi data warehouse to be viewed in Atlas.

[0054] S40: Synchronize the table data generated by processing the test data in the Hudi data warehouse to the result table to generate the blood relationship of the test data;

[0055] S50: Use Atlas tool to record the blood relationship of the test data.

[0056] Since the processing of lithium battery finished product inspection data has strict time requirements, and the Hive database processes data in batches, the query performance is relatively slow and is not suitable for the processing of lithium battery finished product inspection data; the Hudi data lake adopts a data layout based on column storage and uses index and metadata management strategies to provide fast incremental query performance, which is more suitable for real-time analysis and interactive queries, and can achieve lower query latency, which can meet the strict requirements of lithium battery finished product inspection data processing on data processing time; but the Atlas framework can currently only collect Hive databases, so this embodiment establishes a result table synchronized from Hudi to Hive, and generates lineage through one-time processing in the Hive data warehouse, and then runs tasks in the Hudi data warehouse normally to output to the Hive result table, so that the table data generated by the Hudi data warehouse can be synchronized to the Hive table normally and the data lineage dependency can also be displayed in Atlas, so that Atlas can manage the Hudi data warehouse data and meet the strict requirements of lithium battery finished product inspection data processing time.

[0057] In one embodiment, step S20: using the flink tool to write the test data into the Hudi data lake to generate a Hudi data warehouse and synchronize it to the Hive database to generate a Hive table. The Hive table is synchronized with the table attributes and field attributes of the Hudi data warehouse. The specific implementation process is as follows:

[0058] When writing the test data of finished lithium batteries into the Hudi data warehouse through the Flink tool, the synchronization script settings are:

[0059] 'hive_sync.enable' = 'true',

[0060] 'hive_sync.db' = 'database name',

[0061] 'hive_sync.table' = 'table name',

[0062] 'hive_sync.mode' = 'hms',

[0063] 'hive_sync.metastore.uris'='thrift: / / XXX.XXX.XX.XX:9083',

[0064] This synchronizes the table attributes and field attributes of the Hudi table in the Hudi data warehouse to the Hive table in the Hive database.

[0065] In one embodiment, step S30: adding fields and annotation information to the Hive table and generating a result table of synchronizing the Hudi data warehouse to the Hive database specifically includes the following steps:

[0066] Add table comments, field comments, fields, and HDFS addresses to the Hive table to generate a result table of synchronizing the Hudi data warehouse to the Hive database;

[0067] The table annotation is the annotation of the data table in the Hudi data warehouse, and the field annotation is the field annotation of the data table in the Hudi data warehouse;

[0068] The fields include Hudi commit time, Hudi sequence, Hudi primary key, Hudi partition, and Hudi file name.

[0069] Specifically, this embodiment adds table comments and field comments to the synchronized Hive table. The implementation process is as follows:

[0070] alter table `table name` set tblproperties('comment' = 'table comment'); alter table `table name` change column `field name` `field name` field type comment 'field comment';

[0071] At the same time, to ensure that data can be written normally during subsequent Hudi data warehouse processing, the following fields are added to the Hive table:

[0072] `_hoodie_commit_time` string COMMENT'hudi commit time',

[0073] `_hoodie_commit_seqno`string COMMENT'hudi sequence',

[0074] `_hoodie_record_key` string COMMENT 'hudi primary key',

[0075] `_hoodie_partition_path` string COMMENT 'hudi partition',

[0076] `_hoodie_file_name` string COMMENT'hudi file name',

[0077] Since the data in the Hudi data lake and Hive database are both based on the HDFS distributed file system, the new address configuration in the Hive table is described as follows:

[0078] COMMENT 'TABLE DESCRIPTION' ROW FORMAT SERDE

[0079] 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'WITHSERDEPROPERTIES('hoodie.query.as.ro.table'='false','path'='hdfs address')STOREDAS INPUTFORMAT'org.apache.hudi.hadoop.HoodieParquetInputFormat'OUTPUTFORMAT'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'

[0080] LOCATION 'hdfs address'.

[0081] In one embodiment, step S40 of adding fields and annotation information to the Hive table and generating a result table of synchronizing the Hudi data warehouse to the Hive database includes the following steps:

[0082] S41. Convert the layered processing SQL of the Hudi data warehouse into Hive-SQL;

[0083] S42: Process the test data in the Hudi data warehouse to generate table data, and synchronize the table data to the result table of the Hive database according to Hive-sql to generate the lineage relationship of the test data.

[0084] It should be noted that in this embodiment, based on the newly created result table, the Hive client specifies the corresponding Hudi table according to the converted hive-sql and synchronizes it to the Hive table to run the results at one time, and runs the generated lineage. In this way, the table data generated by the Hudi data warehouse processing can be synchronized to the Hive table normally, and the data lineage can also be displayed in Atlas.

[0085] In one embodiment, the result table is a wide table with detailed layers.

[0086] It should be noted that this embodiment uses a wide table as the result table to sink the core logic, converge the caliber, achieve a unified indicator caliber, and improve development efficiency.

[0087] In one embodiment, after the step S50 of recording the blood relationship of the test data using the Atlas tool, the method further includes the following steps:

[0088] Delete the data in the result table.

[0089] It should be noted that since the values of the configuration columns in the result table created manually in Hive are not automatically generated according to the Hudi program, Hive data cannot be mapped to the Hudi table. Therefore, the data in the result table is deleted, and Hive data is generated in the subsequent Hudi task.

[0090] Furthermore, the specific implementation process of step S50: using the Atlas tool to record the blood relationship of the test data is as follows:

[0091] The metadata of lithium battery finished product test data stored in MySQL is loaded, encapsulated into the data model defined by Atlas, and stored in Hbase. After that, an index is built using Solr. Hive metadata information can then be found on the page through Atlas. The metadata incremental data synchronization function monitors Hive data changes through the Hive-hook script.

[0092] In one embodiment, after the step S40 of synchronizing the table data generated by processing the test data in the Hudi data warehouse to the result table and generating the lineage relationship of the test data, the method further comprises the following steps:

[0093] For related test data, establish a data lineage relationship diagram;

[0094] The Atlas tool is used to record the data lineage relationship diagram for tracking the test data of each link in the lithium battery production process.

[0095] Specifically, in the lithium battery industry, the relationship between data assets is very complex. In order to better track the flow and changes of data, this embodiment implements customized data lineage tracking based on the association between test data. For example, a data lineage relationship diagram is established for the data collected from the lithium battery electrical performance test and charge and discharge test. Because there is a sequence relationship between electrical performance testing and charge and discharge testing, parameters such as battery capacity and battery voltage are related to each other. Atlas generates the relationship lineage between fields based on SQL. By querying parameters such as battery voltage based on lineage, it can record the source and destination of the data throughout the test process, as well as the transformation and processing of the data during the test process, and realize data lineage tracking of each link in the lithium battery production process, so as to better understand the source, changes and impact of the data.

[0096] In one embodiment, the method further comprises:

[0097] Data asset types corresponding to the lithium battery finished product testing process are added to the Atlas tool, and the data asset types include raw material testing data assets, cell assembly testing data assets, finished product testing data assets, and battery assembly testing data assets.

[0098] It should be noted that this embodiment can add new data asset types by extending the model to adapt to the special needs of lithium battery testing. In the data asset viewing table, you can view the English and Chinese names, responsible persons, business descriptions, fields in Chinese and English, partition fields, field processing logic, business domains, topics, and tag levels.

[0099] In one embodiment, after the step S50 of recording the blood relationship of the test data using the Atlas tool, the method further includes the following steps:

[0100] The Atlas tool starts the consumer to consume messages, determines the data type of the parameter index based on the lineage relationship of the test data, and stores the parameter index;

[0101] Aggregate the test data and aggregate it into the ES search engine;

[0102] The test data of each link in the lithium battery production process is queried using ES-based data retrieval technology according to the parameter index.

[0103] In practical applications, search functionality is a top priority, at least to allow analysts to directly find the right historical test data. This is because business domain division and topic tag maintenance are difficult to achieve without ambiguity, allowing analysts to successfully filter out target data. This embodiment utilizes table metadata to provide users seeking data with more comprehensive and accurate information, including business metadata and technical metadata.

[0104] Data retrieval methods include filtering based on business domains, topics, tags, and other hierarchical levels, as well as searching for English and field information within tables. Technically, Elasticsearch was used instead of MySQL as the search tool. ES uses inverted indexing to optimize search speed. Although this consumes more space, the improvement in search performance is significant.

[0105] This embodiment optimizes data indexing and querying based on the characteristics of lithium battery parameter data. First, the data type that needs parameter indexing is determined, and the data is stored in the specified format. The query performance is optimized by adjusting the number of shards and replicas. Specifically:

[0106] In the lithium battery testing business, if you need to query a specific parameter value in the test results, you can use term query to accurately match the parameter value and quickly locate the test result.

[0107] If you need to query the occurrence of a specific keyword in the test results, you can use the match query to perform a full-text search to quickly find relevant test results.

[0108] If you need to query test results for a parameter within a certain range, you can use a range query to quickly filter out test results that meet the conditions. Choosing the appropriate query method can improve query efficiency.

[0109] Furthermore, this embodiment uses data aggregation to summarize and analyze data in advance. For example, if data is aggregated into ES at the second level, parameters can be aggregated according to the minute level (such as median or average). Querying and analyzing dimensions above the minute level can save a lot of time.

[0110] In one embodiment, after the step S10 of collecting test data of finished lithium batteries, the method further includes the following steps:

[0111] Preprocessing the test data to obtain preprocessed data;

[0112] Parameters related to battery performance are extracted from the pre-processed data as feature data, and assets are constructed based on the feature data.

[0113] It should be noted that before processing the finished lithium battery test data, this embodiment requires preprocessing the collected data to ensure data quality and accuracy. Preprocessing methods include, but are not limited to, data cleaning, data denoising, and data normalization. Data can also be subjected to operations such as dimensionality reduction and feature extraction to reduce computational complexity.

[0114] Specifically:

[0115] (1) Data cleaning: In the data preprocessing of the lithium battery industry, due to the lack of unified equipment protocols and unstable IoT signals in the market, the collected equipment data has a high ratio of dirty data. Therefore, invalid data such as outliers, duplicate values, and error values are removed from the data based on data quality judgment to improve data quality.

[0116] (2) Data denoising: In the data preprocessing of the lithium battery industry, many technical parameters need to be denoised (battery capacity: in actual use, battery capacity is affected by many factors, such as charge and discharge rate, temperature, etc., which generates a lot of noise. Cycle life: cycle life is affected by many factors such as battery usage, charge and discharge environment, charge and discharge rate, etc., which generates a lot of noise. Internal resistance: internal resistance is affected by many factors such as battery charge and discharge rate, temperature, electrochemical reaction, etc., which generates a lot of noise. Charging efficiency: charging efficiency is one of the important indicators to measure battery performance, but charging efficiency is affected by many factors such as battery charging method, charging environment, charging rate, etc., which generates a lot of noise.) Noise can be removed through smoothing algorithms and other technologies to reduce data fluctuations and errors, improve data readability and the accuracy of analysis results.

[0117] (3) Data normalization: In data preprocessing for the lithium battery industry, different battery models have different capacity units, such as mAh and Ah. Different battery models have different cycle life units, such as times and hours. Different battery models have different internal resistance units, such as mΩ and Ω. Different battery models have different charging efficiency units, such as %, ppm, etc. Data can be normalized through standardization and normalization techniques to compare and analyze data of different scales.

[0118] After data preprocessing, feature engineering is required to extract useful features from the raw data. Feature engineering is a crucial step in model training and directly impacts model performance. A variety of methods, including but not limited to principal component analysis (PCA), can be used to reduce data redundancy and noise.

[0119] Specifically, parameters related to battery performance, such as battery capacity and internal resistance, are extracted from the preprocessed data. PCA analysis is performed using the PCA class in sklearn, and the desired number of principal components is selected. The degree to which each principal component explains the data variance is analyzed to determine how many principal components should be retained to preserve sufficient data information. The most relevant features, such as battery capacity and internal resistance, are selected from the principal components for lithium battery performance prediction.

[0120] Strong interpretability: PCA can extract the most relevant features, which can be interpreted as contributions to lithium battery performance, making it easier to analyze and optimize lithium battery performance.

[0121] Visual analysis: PCA can reduce the dimensionality of data to two or three dimensions, which allows the performance characteristics of lithium batteries to be visualized and analyzed on a two-dimensional or three-dimensional plane.

[0122] Strong noise reduction capability: PCA has excellent noise reduction capability, which can filter out noise and irrelevant features in the data and improve the accuracy of lithium battery performance prediction.

[0123] Furthermore, data collected online is organized into a data warehouse and modeled according to the ODS, DWD, DWS, and ADS layers (including data from electrical tests, mechanical tests, and environmental simulations). The Dolphinscheduler framework is used to perform quality checks on the incoming data. Data quality tasks operate by scheduling custom SQL statements using Spark programs to generate quality results.

[0124] Specifically, data quality assessments include: Uniqueness Verification: This primarily checks whether the collected device field results are unique and duplicate. Integrity Verification: This determines whether the result field is null, empty, or a garbled value. Accuracy Verification: This determines whether the result value of a field is within a set range. Timeliness Verification: This determines whether the collected and written data falls within the set timeframe, which is divided into collection timeliness and storage timeliness. Data quality tasks also include single-table checks, single-table custom SQL checks, multi-table accuracy, and comparison of values between two tables.

[0125] This embodiment performs data quality detection to check whether the data meets the symbol requirements and ensure the accuracy of the displayed data. The detection can be performed at each layer of the data warehouse, usually at the ODS layer.

[0126] It should be understood that data quality detection can be performed before or after the generation of data lineage dependency relationships. This embodiment chooses to perform data quality detection before the generation of data lineage dependency relationships.

[0127] In one embodiment, the method further comprises the following steps:

[0128] Configure workspaces for users with different roles based on personal workstations;

[0129] Assign corresponding permissions to different workspaces to display corresponding data asset views based on user roles and permissions.

[0130] This embodiment uses the asset dashboard to provide asset views for users of different roles, from data management to analysts and individuals. Its main function is to objectively describe the current status of assets and provide guidance for asset construction optimization based on the asset indicator evaluation system. The specific implementation steps are:

[0131] (1) Define roles and permissions

[0132] First, you need to define the roles and corresponding permissions in the system, such as administrator, ordinary user, data analyst, etc. Each role should have different permissions and can access different data assets.

[0133] (2) Create a data asset view

[0134] Create corresponding data asset views based on different role permissions. Determine the source of data assets. The source of data assets for the lithium battery business includes multiple different test processes such as static testing, charge and discharge testing, and electrical performance testing. Determining the source of data assets is the first step in creating a data asset view. It requires a comprehensive investigation and collection of lithium battery business data assets. After completing the collection and cleaning of data assets, you can start creating data asset views. Data asset views can be presented in a variety of ways, such as tables, charts, maps, etc. Different views can be customized according to user needs so that users can better understand and utilize data assets.

[0135] (3) Classify and label data assets

[0136] All data assets are categorized and labeled to facilitate displaying the corresponding data asset views based on permissions. For example, data assets can be categorized into manual testing, fully automated equipment testing, semi-automated equipment testing, etc., and each data asset can be labeled accordingly.

[0137] (4) Display the corresponding data asset view

[0138] When a user logs in to the system, the corresponding data asset view is displayed based on the user's role and permissions. For example, administrators can see all data asset information, ordinary users can only see their own data asset information, and data analysts can only see data asset information related to analysis.

[0139] (5) Viewing, downloading, sharing, and other operations of data assets

[0140] Provide corresponding data asset operation functions based on user permissions. For example, administrators can view, download, and share data assets, while ordinary users can only view and download their own data assets. Data analysts can view and download data assets related to analysis.

[0141] (6) Maintenance and Update

[0142] As the system is used and the business evolves, roles and permissions may change and require timely maintenance and updates. For example, when a new role is added or the permissions of a role are adjusted, the role and permission settings in the system need to be updated accordingly, and the corresponding data asset views need to be redefined.

[0143] Furthermore, the asset indicator evaluation system specifically includes:

[0144] 1) Asset quantity indicators: total number of assets (information on connected data sources, number of database tables, information on different data warehouse tiers, etc.), number of new assets (the amount of new data is counted by hour or day).

[0145] 2) Resource consumption indicators: operating resources (using scripts to monitor server CPU / memory / hard disk space and other information), production consumption computing resources (based on the resource information on the YARN scheduling container, statistics on currently used and idle CPU and memory information).

[0146] 3) Quality indicators:

[0147] Accuracy: Calculated by dividing the number of correct data points by the total number of data points. For example, if there are 100 data points for lithium battery voltage, and 90 of them are correct, then the accuracy is 90%.

[0148] Completeness: Calculated by dividing the number of complete data points by the total number of data points. For example, if a lithium battery needs to contain 100 attribute values, but only 80 attributes provide complete data, then the completeness is 80%.

[0149] Consistency: This is calculated by dividing the number of matching data points between different data sources by the total number of data points. For example, if two different manufacturers' equipment test a lithium battery inspection step and the results have a total of 100 data points, and 80 of them match, then the consistency is 80%.

[0150] Reliability: Calculated by dividing the number of data points by the total number of data points. For example, if a lithium battery is tested five times using known test conditions and trigger results, and four of them are correct, the reliability is 80%.

[0151] Metadata coverage ratio: Based on the existing Chinese comments, field type, default value, field length and other metadata information / total data volume of the database table.

[0152] Time-consuming tasks and lists: tasks that exceed the specified time and tasks with abnormal CPU / memory usage.

[0153] Number of no accesses in the past 90 days: Tables or fields in the data warehouse that have not been queried within 90 days.

[0154] In this embodiment, the workspaces of different analysts can be configured according to their personal workstations, and the administrator can assign the business domains, data hierarchies, themes, and required libraries, tables, and fields that the corresponding analysts need to use. Hierarchical information and field information can be queried in the asset dashboard, and the relationship between different tables can be obtained using data lineage. When checking the asset indicators of the assets you manage, the current system resource usage can be used to better schedule data analysis tasks, and the search function can be used to quickly obtain the information you want. This method is applied to the inspection of finished lithium batteries. By testing and analyzing battery capacity, internal resistance, voltage, and other aspects, it improves product quality and production efficiency, ensuring that analysts can find the right data and find it quickly when faced with disorganized test data.

[0155] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0156] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0157] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A method for constructing lithium battery finished product inspection data assets, characterized in that: The method comprises: Collect test data of finished lithium batteries; Use the Flink tool to write the test data into the Hudi data lake to generate a Hudi data warehouse and synchronize it to the Hive database to generate a Hive table. The Hive table is synchronized with the table and field attributes of the Hudi data warehouse. Add fields and annotation information to the Hive table to generate a result table of synchronizing the Hudi data warehouse to the Hive database; Synchronize the table data generated by processing the test data in the Hudi data warehouse to the result table to generate the lineage relationship of the test data; Use Atlas tool to record the blood relationship of the test data; After recording the blood relationship of the test data using the Atlas tool, the method further includes: The Atlas tool starts the consumer to consume messages, determines the data type of the parameter index based on the lineage relationship of the test data, and stores the parameter index; Aggregate the test data and aggregate it into the ES search engine; The ES-based data retrieval technology is used to query the test data of each link in the lithium battery production process according to the parameter index.

2. The method for constructing lithium battery finished product inspection data assets according to claim 1, wherein: Adding new fields and annotation information to the Hive table and generating a result table of synchronizing the Hudi data warehouse to the Hive database includes: Add table comments, field comments, fields, and HDFS addresses to the Hive table to generate a result table of synchronizing the Hudi data warehouse to the Hive database; The table annotation is the annotation of the data table in the Hudi data warehouse, and the field annotation is the field annotation of the data table in the Hudi data warehouse; The fields include Hudi commit time, Hudi sequence, Hudi primary key, Hudi partition, and Hudi file name.

3. The method for constructing lithium battery finished product inspection data assets according to claim 1, wherein: The process of synchronizing the table data generated by processing the test data in the Hudi data warehouse to the result table to generate the lineage relationship of the test data includes: Convert the Hudi data warehouse layered processing SQL into Hive-SQL; The test data is processed in the Hudi data warehouse to generate table data, and the table data is synchronized to the result table of the Hive database according to the Hive-sql specified table data to generate the lineage relationship of the test data.

4. The method for constructing lithium battery finished product inspection data assets according to claim 1, wherein: The result table is a wide table of detail layers.

5. The method for constructing lithium battery finished product inspection data assets according to claim 1, wherein: After recording the blood relationship of the test data using the Atlas tool, the method further includes: Delete the data in the result table.

6. The method for constructing lithium battery finished product inspection data assets according to claim 1, wherein: After synchronizing the table data generated by processing the test data in the Hudi data warehouse to the result table and generating the lineage relationship of the test data, the method further includes: For related test data, establish a data lineage relationship diagram; The Atlas tool is used to record the data lineage relationship diagram for tracking the test data of each link in the lithium battery production process.

7. The method for constructing lithium battery finished product inspection data assets according to claim 1, wherein: The method further comprises: Data asset types corresponding to the lithium battery finished product testing process are added to the Atlas tool, and the data asset types include raw material testing data assets, cell assembly testing data assets, finished product testing data assets, and battery assembly testing data assets.

8. The method for constructing lithium battery finished product inspection data assets according to claim 1, wherein: After collecting the test data of the finished lithium battery, the method further includes: Preprocessing the test data to obtain preprocessed data; Parameters related to battery performance are extracted from the pre-processed data as feature data, and assets are constructed based on the feature data.

9. The method for constructing lithium battery finished product inspection data assets according to any one of claims 1 to 8, wherein: The method further comprises: Configure workspaces for users with different roles based on personal workstations; Assign corresponding permissions to different workspaces to display corresponding data asset views based on user roles and permissions.

Images