Lithium battery production manufacturing industry data quality detection method and system

Abstract

The application discloses a kind of lithium battery production manufacturing industry data quality detection method and system, method includes based on data source information, library table information and rule field pull at least one target platform storage lithium battery related data, the library table information includes the database instance corresponding under the data source information and the data table of lithium battery related data required to be verified under database embodiment, the rule field is the operation column required to be verified in the data table;Based on the rule field of pre-configured data quality detection rule and single platform, the quality detection of relevant data in the data table of this single platform is carried out, and the same platform data quality detection result is obtained;Based on the rule field corresponding to at least two platforms of pre-configured data quality detection rule, cross-platform data association and merging check are carried out, and cross-platform data quality detection result is obtained;The flexibility of data quality detection of the application can meet the complex and diversified data quality detection demand.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data detection, and particularly relates to a data quality detection method and system for lithium battery production and manufacturing industry. BACKGROUND

[0002] The data warehouse in the lithium battery production and manufacturing business field carries various source data output from the production line. Among them, there are production result data such as work order, process, procedure, work step, production time, equipment, yield, defective product quantity, material, process parameter, etc.; there are actual measurement data generated in the production process, i.e. production process data such as speed, current, permanent magnet strength, pressure, concentration, dust removal wind speed, welding amplitude, welding pressure, resistance, etc.; and there are environmental, power trend data such as temperature and humidity, air cleanliness, lighting, etc. in the production process. Different types of data require different quality characteristics, and the focus of data quality detection is different. For example: the data used by the result data requires high quality, complete field content and accurate data, and the production process data requires high data frequency, large data volume and complete records.

[0003] Related data quality detection schemes mostly use fixed standards to detect the data quality of a single system, and the detection scheme has the principle of universality for various industries. For example, the patent application document with publication number CN112597142A proposes a data quality detection method, which can determine the detection items to be executed on the target database table based on data detection configuration parameters, and realizes a data quality detection scheme that adapts to the system business scene demand; but this scheme is configured for the detection items of multiple systems of the same target job, and it can configure corresponding detection items for different single systems, but does not support cross-platform data quality detection. The patent application document with publication number CN111897806A proposes a big data offline data quality checking method, which checks the data quality based on the checking rule configuration checking statement in the data quality checking system, and the checking rule includes one or more of data timeliness checking rule, field non-empty checking rule, field value domain checking rule, field uniqueness checking rule, field data length checking rule, field specific value proportion checking rule, data volume same period checking rule, data volume ring period checking rule, record number consistency checking rule, data subset checking rule; but the detection rule used is not comprehensive, and cannot meet the diversified detection needs of the lithium battery production and manufacturing business field, such as detecting the correspondence between the input and output of each procedure in the battery production; whether the procedure in the process control plan corresponds to the actual production procedure, etc. The patent application document with publication number CN113220726A proposes a data quality detection method, which uses specific business data quality rules and detection methods designed for business databases to realize source system data quality detection; but this scheme is also only suitable for data quality detection of a single system, and does not support cross-platform data quality detection.

[0004] Therefore, the existing data quality detection scheme does not consider the complex and diversified data quality detection requirements in the lithium battery production and manufacturing business field, and the detection flexibility is insufficient, the detection rules are not comprehensive, and the diversified detection requirements in the lithium battery production and manufacturing business field cannot be met. SUMMARY

[0005] The technical problem to be solved by the present application is how to improve the flexibility of data quality detection and meet the complex and diversified data quality detection requirements.

[0006] The present application solves the above technical problems by the following technical means:

[0007] A data quality detection method for lithium battery production and manufacturing industry is proposed, which comprises:

[0008] Based on the data source information, the library table information and the rule field, at least one target platform stored lithium battery related data is pulled, the library table information includes the corresponding database instance under the data source information and the data table where the lithium battery related data required to be verified under the database embodiment, and the rule field is the operation column required to be verified in the data table;

[0009] Based on the pre-configured data quality detection rule and the rule field of a single platform, the related data in the data table of the single platform is detected for quality, and the same platform data quality detection result is obtained;

[0010] Based on the pre-configured data quality detection rule and the rule field corresponding to at least two platforms, cross-platform data association and merging check are performed, and cross-platform data quality detection result is obtained.

[0011] Further, the pulling of at least one target platform stored lithium battery related data based on the data source information, the library table information and the rule field comprises:

[0012] A detection task model is constructed according to the data detection configuration parameter field, the detection task model includes detection item id and data source information, library table information and rule field of at least one target platform;

[0013] The detection task model is parsed into a json structure, and the detection item id and the data source information, the library table information and the rule field of at least one target platform are obtained, wherein the detection item id is used to obtain the detection code class name to poll the data quality detection rule according to the detection code class name;

[0014] Based on the data source information, the library table information and the rule field, at least one target platform stored data is pulled into the memory for data quality detection.

[0015] Further, the detection task model further comprises partition information, the partition information being used to define a data boundary, and the method further comprises:

[0016] Pulling data stored in at least one target platform based on the data source information, the library table information, the partition information and the rule field.

[0017] Further, the data quality detection rule comprises a numerical reasonableness check item, a field legality check item, a data correlation check item, a data type mapping check item, a code value mapping check item and a general check item.

[0018] The general check item comprises a data timeliness check item, a field non-empty check item, a uniqueness check item and a data consistency check item.

[0019] Further, based on the pre-configured data quality detection rule and the rule field of a single platform, the related data in a data table of the single platform is subjected to quality detection to obtain a same-platform data quality detection result, comprising:

[0020] Analyzing the rule field of the single platform to obtain an operation column in the data table corresponding to the platform that needs to be verified;

[0021] At least one of the numerical reasonableness check item, the field legality check item, the data correlation check item and the general check item is used to process the operation column, and the related data in the data table is subjected to data quality checking.

[0022] Further, based on the pre-configured data quality detection rule and the rule field corresponding to at least two platforms, cross-platform data correlation and merging checking is performed to obtain a cross-platform data quality detection result, comprising:

[0023] Correlating the rule fields corresponding to different target platforms to perform cross-platform data correlation checking and screen out data and the number of data that cannot be correlated;

[0024] Correlating the rule fields corresponding to different target platforms to perform code value mapping checking and screen out data whose code and code value cannot be mapped;

[0025] Performing data type mapping checking on the data tables corresponding to different target platforms to determine whether the cross-platform data field types are one-to-one mapped.

[0026] Further, the data type mapping checking on the data tables corresponding to different target platforms to determine whether the cross-platform data field types are one-to-one mapped, comprising:

[0027] Based on the data source information of different target platforms, database instances and data tables, the data structure of the data table corresponding to different target platforms is obtained and loaded into the memory of the computing engine.

[0028] According to the type of the data source corresponding to the different target platforms, the corresponding mapping method is traversed.

[0029] The data fields of the data table corresponding to different target platforms are converted to lowercase, and the same field list is obtained by matching according to the field name.

[0030] The same field list is traversed, the field type mapping is matched according to the corresponding mapping method, and the data that does not conform to the mapping rule is filtered out.

[0031] Further, the method further comprises:

[0032] Based on the business characteristics of the lithium battery related data to be verified, the required configuration check item is determined, wherein the lithium battery related data includes production process data, production result data and environmental situation data.

[0033] Further, the method further comprises:

[0034] When the target job detection is unqualified based on the same platform data quality detection result and / or cross-platform data quality detection result, an alarm prompt is generated;

[0035] According to the pre-constructed data quality generation reason analysis library, the data quality problem existing in the library table corresponding to the target job is determined.

[0036] In addition, the present application also proposes a lithium battery production and manufacturing industry data quality detection system, the detection system comprises:

[0037] The data pulling module is used for pulling the lithium battery related data stored in at least one target platform based on the data source information, the library table information and the rule field, the library table information comprises the corresponding database instance under the data source information and the data table where the lithium battery related data required to be verified under the database embodiment, and the rule field is the operation column required to be verified in the data table;

[0038] The same platform data detection module is used for performing quality detection on the related data in the data table of the single platform based on the pre-configured data quality detection rule and the rule field of the single platform, and obtaining the same platform data quality detection result;

[0039] The cross-platform data detection module is used for performing cross-platform data association and merging checking based on the pre-configured data quality detection rule and the rule field corresponding to at least two platforms, and obtaining the cross-platform data quality detection result.

[0040] The present application has the advantages of:

[0041] (1) The present application can flexibly cope with data quality detection in different scenarios by pulling data in different target platform partitions into the memory of the computing engine and performing single-platform data quality detection and cross-platform data quality detection in the memory, and can meet complex and diversified data quality detection requirements. For cross-platform data quality detection, there is no need to store the data of two platforms centrally, reducing the workload of data extraction and not occupying extra disk storage space.

[0042] (2) The present application can implement single-platform numerical reasonableness checking, field legality checking, and general-purpose checking, and can also support cross-platform multi-table data correlation checking, data type mapping checking, and code value mapping checking, meeting the diversified detection requirements in the production and manufacturing business field.

[0043] (3) The present application solves the need for cross-platform linkage of data type mapping checking items, and avoids the impact of changes in the data structure of the source platform on downstream platforms.

[0044] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 is a flowchart of a lithium battery production and manufacturing industry data quality detection method according to an embodiment of the present application;

[0046] Figure 2 is a block diagram of the overall flow of a lithium battery production and manufacturing industry data quality detection method according to an embodiment of the present application;

[0047] Figure 3 is a structural diagram of a lithium battery production and manufacturing industry data quality detection system according to an embodiment of the present application. DETAILED DESCRIPTION

[0048] To make the purposes, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0049] As shown in Figures 1-2 , a lithium battery production and manufacturing industry data quality detection method is provided in the first embodiment of the present application. The method comprises the following steps:

[0050] S10, pull lithium battery related data stored in at least one target platform based on data source information, library table information and rule field, the library table information includes the corresponding database instance under the data source information and the data table where the required verified lithium battery related data under the database embodiment, and the rule field is the operation column required to be verified in the data table;

[0051] S20, based on the pre-configured data quality detection rule and the rule field of a single platform, the quality of the related data in the data table of the single platform is detected to obtain the same platform data quality detection result;

[0052] S30, based on the pre-configured data quality detection rule and the rule field corresponding to at least two platforms, the data correlation and merging check across platforms is carried out to obtain the cross-platform data quality detection result.

[0053] The embodiment pulls the data in the data partition of different target platforms to the memory of the computing engine, and carries out single-platform data quality detection and cross-platform data quality detection in the memory. The data quality detection based on the memory can flexibly cope with different scenarios of data quality detection, and can meet the complex and diversified data quality detection requirements. For cross-platform data quality detection, the data of two platforms does not need to be stored centrally, the workload of data extraction is reduced, and no extra disk storage space is occupied.

[0054] In an embodiment, the step S10 of pulling lithium battery related data stored in at least one target platform based on data source information, library table information and rule field specifically includes the following steps:

[0055] S11, constructing a detection task model according to the data detection configuration parameter field, the detection task model including detection item id and data source information, library table information and rule field of at least one target platform;

[0056] Specifically, the representation of the detection task model is shown in Table 1:

[0057] Table 1

[0058] Field Name Field Type Field Description Is Primary Key model_id int(32) Application id Primary Key check_item_id int(32) Detection item id Foreign Key check_source varchar(256) First target platform data source check_db varchar(256) First target platform database instance check_table varchar(256) First target platform data table check_field varchar(256) First target platform rule field check_partition varchar(256) First target platform partition expression target_source varchar(256) Second target platform data source target_db varchar(256) Second target platform database instance target_table varchar(256) Second target platform data table target_field varchar(256) Second target platform rule field target_partition varchar(256) Second target platform partition expression create_date datetime Creation time update_time datetime Update time

[0059] It should be understood that the detection task model in the embodiment is for illustration, and the number of target platforms contained in Table 1 can be set according to actual needs.

[0060] It should be noted that the data source of each target platform can adopt OLAP, OLTP, Hive, TiDB data source, etc.; the database instance can adopt the actual database instance name ， For example ：product_db, etc.; the data table can be a production fact table produce, a process code table process, etc.; the rule field is process code process_code, etc.

[0061] Among them, under the same target platform, it is necessary to ensure that the selected database instance has the data table to be verified; the data table has the rule field to be verified.

[0062] It should be understood that the database instance is a user-defined name of the database, and the data table and the rule field can be filled by the user in combination with the business requirements, and the embodiment is not limited specifically.

[0063] S12, the detection task model is parsed into a json structure, and the detection item id and the data source information, the library table information and the rule field of at least one target platform are obtained, wherein the detection item id is used to obtain the detection code class name to poll the data quality detection rule according to the detection code class name;

[0064] Specifically, the detection item id is obtained by parsing the detection task model, the detection code class name is obtained by querying the detection rule template table according to the detection item id, the data quality detection rule library is polled, and the detection code class is matched, wherein the detection rule template table is shown in Table 2:

[0065] Table 2

[0066] Field Name Field Type Field Description Is Primary Key check_item_id int(32) Detection item id Primary Key check_item_name varchar(100) Detection item name check_item varchar(100) Detection code class Unique Index check_category varchar(100) Detection item category check_class varchar(100) Detection item classification check_rule text Detection item rule create_date datetime Creation time update_time datetime Modification time

[0067] According to the matched detection code class, it is checked whether the configuration parameter is missing, if the parameter is complete, the parameter is brought in, and the data quality detection is performed.

[0068] S13, the data stored in at least one target platform is pulled into the memory for data quality detection.

[0069] In an embodiment, the detection task model further includes partition information, the partition information is used to limit the data boundary, and the method further includes the following:

[0070] Based on the data source information, the library table information, the partition information and the rule field, the data stored in at least one target platform is pulled.

[0071] Specifically, the partition information is specifically a partition expression, and the detection task model constructed in the embodiment includes the partition expression corresponding to the target platform. The partition expression information can be obtained by parsing the detection task model into a json structure; and the data boundary is limited by using the partition expression rule. As shown in Table 1, the format of the partition expression is: partition field = 'partition value', and the operator can be: =, >, >=, <, <=; for example: dt_p = '2022-12-31' or create_time >= '2023-01-01', the interval is limited or connected by and between multiple levels of partitions, and the SQL syntax rule is followed; if the table is a non-partition table or does not need to fill in the partition, it can be left blank and not filled in.

[0072] In an embodiment, the data quality detection rule includes a numerical reasonableness check item, a field legality check item, a data correlation check item, a data type mapping check item, a code value mapping check item, and a general check item.

[0073] Specifically, the general check item includes but is not limited to a data and timeliness check item, a field non-empty check item, a uniqueness check item, a data consistency check item, etc.

[0074] It should be noted that the data quality detection rule library is constructed to evaluate the severity and influence range of the current data quality problem.

[0075] In an embodiment, the step S20 includes the following steps:

[0076] S21, parsing the rule field of the single platform to obtain the operation column in the data table corresponding to the platform that needs to be verified;

[0077] S22, using at least one of the numerical reasonableness check item, the field legality check item, the data correlation check item, and the general check item to process the operation column, and performing data quality checking on the related data in the data table.

[0078] Specifically, the process of performing numerical reasonableness checking on a single data table of a single platform includes:

[0079] (1) According to the first rule field content of the detection task model parameter, the control limit table is queried to obtain the control upper limit control_upper_limit and the control lower limit control_lower_limit content of the field;

[0080] (2) According to the first target platform data source, database instance, data table, and partition expression of the detection task model parameters, data in the partition is obtained and loaded into the memory of the computing engine;

[0081] (3) According to the control upper and lower limits of the detection field, it is judged whether the obtained data exceeds the control limit;

[0082] (4) If there is no data exceeding the control limit, the data has no rationality problem, and the check_flag flag is 0, otherwise, the data has a rationality problem, and the check_flag flag is 1;

[0083] (5) The detection result is recorded in check_result, and the parameter information is written into the detection result table.

[0084] The field legality check of a single platform single data table is a legality check of a single table field. The legal field detection rule is set to judge whether the field has garbled code, or whether it meets the coding rule, or whether it meets the time format, etc. The detection process includes:

[0085] (1) According to the first target platform data source, database instance, data table, and partition expression of the detection task model parameters, data in the partition is obtained and loaded into the memory of the computing engine;

[0086] (2) The first rule field content of the model parameters is parsed, separated by the “|” vertical line, and the first field content after separation is polled to execute the detection method of the content; for example, the rule detection field is set to: material code | material_code. The model detection rule field content is parsed, separated by the “|” vertical line, and the first field content “material code” after separation is polled to find the material code field legality detection method. It is analyzed whether the detection field loaded into the memory meets the coding rule of 10-bit incoming code + 4-bit material version number + 6-bit supplier code + 6-bit date + 8-bit serial number;

[0087] (3) If there is no illegal data, the detection has no problem, and the check_flag flag is 0, otherwise, the detection has a problem, and the check_flag flag is 1;

[0088] (4) The detection result is recorded in check_result, and the parameter information is written into the detection result table.

[0089] In an embodiment, the step S30 of performing cross-platform data association and merging check based on the pre-configured data quality detection rule and the rule field corresponding to at least two platforms to obtain a cross-platform data quality detection result specifically includes the following steps:

[0090] S31, associate the rule fields corresponding to different target platforms, perform cross-platform data association checking, and filter out data and quantities that cannot be associated;

[0091] Specifically, the data association checking refers to checking the data association between two tables on the same platform or across platforms, to determine whether there is data and quantity that cannot be associated between the two tables. In specific examples, the data association checking can be used to determine the corresponding relationship between the product code in the production fact table produce and the output product code in the production input-output table produce_in, to ensure data integrity and ensure the reliability of production traceability; or to determine the corresponding relationship between the work order number produce_order_code in the work order table order and the work order number produce_order_code in the production fact table produce, to detect how many data in the production fact table have not been associated with the correct work order number. The detection process includes:

[0092] (1) According to the detection task model parameters first and second target platform data sources, database instances, data tables, and partition table expressions, obtain data within the partitions of the two platforms and load them into the memory of the computing engine;

[0093] It should be noted that when performing data association checking on related library table data of a single target platform, the association checking is performed on different (library) tables within the single target platform. The specific association process is similar to that of cross-platform data association checking, which will not be described here.

[0094] (2) According to the detection task model parameters first and second rule fields, perform association and filter out data and quantities that cannot be associated;

[0095] (3) If the quantity of data that cannot be associated is 0, then there is no problem with this type of detection, and the check_flag flag is 0. Otherwise, if the quantity of data that cannot be associated is not 0, then there is a problem with this type of detection, and the check_flag flag is 1;

[0096] (4) Record the detection results in the check result, and write the parameter information and the detection results into the detection result table together. The detection result table is shown in Table 3:

[0097] Table 3

[0098]

[0099] S32, associate the rule fields corresponding to different target platforms, perform code value mapping checking, and filter out data whose code and code value cannot be mapped;

[0100] Specifically, the code value mapping check refers to a code and code value mapping check between two tables of the same platform or across platforms, to determine whether the code field and the corresponding value in the code table have a corresponding relationship. In a specific example, the code value mapping check can be used to determine whether the process code process_code, process name process_name in the production fact table produce, and the process code and process name in the process table process are in a one-to-one correspondence. The detection process includes:

[0101] (1) According to the detection task model parameters first and second target platform data sources, database instances, data tables, and partition expressions, obtain data within the partitions of the two platforms (here, the second target platform partition expression can be empty), and load them into the memory of the computing engine;

[0102] (2) According to the detection task model parameters first and second rule fields, perform association and filter out data that does not map code to code value; the rule field refers to the field that needs to be detected, and multiple fields are separated by “|”, for example: process_code | process_name, corresponding checks are performed according to the fields,

[0103] (3) If the unmapped data is 0, there is no problem with this type of detection, and the check_flag flag is 0, otherwise, there is a problem with this type of detection, and the check_flag flag is 1;

[0104] (4) Record the detection results in check_result, and write them into the detection result table together with the parameter information;

[0105] (5) Perform data type mapping check on different target platform corresponding data tables.

[0106] In an embodiment, the step S33: performing data type mapping check on different target platform corresponding data tables, to determine whether the cross-platform data field types are one-to-one mapped.

[0107] Specifically, the data type mapping check is a data type mapping check for isomorphic data tables across platforms, to determine whether the cross-platform data field types are one-to-one mapped. In a specific example, the data type mapping check has two functions: one is to check whether the source platform field has been newly added, modified, or deleted, and whether the downstream platform has been synchronized; the other is to check whether the data field type has been changed, and whether the downstream platform has been synchronized. For example, the target platform process route table technics_line is synchronized from the source platform process route table technics_line, and the data type mapping check can be used to check whether the source platform process route table data field has been changed or the data type has been changed, and whether the downstream platform has been synchronized, if not, a data quality alarm is issued. The steps include:

[0108] According to the data structure of two data tables obtained from the data source, the database instance and the data table under the first target platform and the second target platform in the detection task model, the data structures are loaded into the memory of the computing engine;

[0109] According to the type of the data source corresponding to the different target platforms, the corresponding mapping method is traversed;

[0110] The data fields of the data tables corresponding to the different target platforms are converted into lowercase, and the isomorphic field list is obtained by matching according to the field name;

[0111] The isomorphic field list is traversed, the field type mapping is matched according to the corresponding mapping method, and the data that does not conform to the mapping rule is filtered out.

[0112] It should be noted that if the unmapped data is 0, there is no problem with the detection, and the check_flag flag is 0, otherwise, there is a problem with the detection, and the check_flag flag is 1; the detection result is recorded in the check_result, and the parameter information is written into the detection result table together, as shown in Table 4:

[0113] Table 4

[0114]

[0115] In an embodiment, the method further comprises the following steps:

[0116] Based on the business characteristics of the lithium battery related data to be verified, the required configuration check items are determined, wherein the lithium battery related data includes production process data, production result data and environmental situation data.

[0117] For example, the material data, product code data and work order data in the lithium battery production result data are suitable for field legality check to detect whether the data conforms to the coding rules; the corresponding relationship between the product code in the production fact table produce and the output product code in the production input-output table produce_in is suitable for data correlation check to ensure data integrity and ensure the reliability of production traceability; whether the process code, process name in the production fact table produce and the process code, process name in the process table process are a one-to-one correspondence is suitable for code value mapping check; the data table synchronized from the source platform to the target platform can be suitable for data type mapping check.

[0118] For the data with faster frequency in the process data, such as the pressure test data, the pressure value is measured every second in the pressure test process, and the value rationality check needs to be performed to determine whether the pressure value is out of limit; meanwhile, the equipment information and parameter information of the process data also have certain coding rules, and the code value mapping check can also be performed.

[0119] Similarly, for the environmental and power state data such as temperature and humidity, air cleanliness, and lighting, the value rationality check can also be applied.

[0120] In this embodiment, the descriptive information of the existing data quality requirements of the business department is collected in advance, the necessity of the requirement information is judged by the digital department, the mapping relationship between the data quality requirement and the business data is well established, and the necessary data quality requirement is saved to the database.

[0121] Specifically, the business department requires that the data quality of the material type should be one-to-one corresponding to the actual situation, that is, material type 1 corresponds to raw materials, 2 corresponds to semi-finished products, and 3 corresponds to finished products. The material data of material type 1 can be one-to-one corresponding in the raw material table materiel, the material data of material type 2 can be one-to-one corresponding in the production fact table produce, and the material data of material type 3 should be the data of the battery cell.

[0122] In an embodiment, the method further comprises the following steps:

[0123] When the target job detection is determined to be unqualified based on the same-platform data quality detection result and / or the cross-platform data quality detection result, an alarm prompt is generated;

[0124] According to the pre-constructed data quality generation cause analysis library, the data quality problem existing in the library table corresponding to the target job is determined.

[0125] It should be noted that in this embodiment, when the data quality detection result determines that the target job detection is unqualified, an alarm prompt can be optionally sent, and the data quality problem existing in the library table corresponding to the target job is indicated according to the data quality generation cause analysis library.

[0126] Optionally, the alarm is performed through the construction of an OA process, supplemented by one or more of email notification, short message notification, and enterprise WeChat notification, which acts on the production line staff to timely process data collection problems, and the OA process is closed after repair, forming a data quality improvement closed loop.

[0127] Further, in this embodiment, the root cause analysis of the data quality problem is performed in advance, a data quality generation cause analysis library is constructed, and when the target job is determined to be unqualified, the data quality problem existing in the corresponding library table is given, providing data support for subsequent improvement.

[0128] The embodiment constructs a lithium battery production and manufacturing industry data quality detection rule library, performs all-around data quality inspection on production and manufacturing business field data, improves data accuracy, and avoids the influence of dirty data from expanding.

[0129] In addition, as Figure 3 shown, the second embodiment of the present application proposes a lithium battery production and manufacturing industry data quality detection system, which comprises:

[0130] A data pulling module 10 is configured to pull lithium battery related data stored in at least one target platform based on data source information, library table information and rule fields, wherein the library table information comprises a database instance corresponding to the data source information and a data table in which lithium battery related data required to be verified under a database embodiment, and the rule field is an operation column required to be verified in the data table;

[0131] A same-platform data detection module 20 is configured to perform quality detection on related data in a data table of a single platform based on pre-configured data quality detection rules and rule fields of the single platform, to obtain a same-platform data quality detection result;

[0132] A cross-platform data detection module 30 is configured to perform cross-platform data association and merging inspection based on pre-configured data quality detection rules and rule fields corresponding to at least two platforms, to obtain a cross-platform data quality detection result.

[0133] The embodiment pulls data in different target platform partitions to the memory of the computing engine, and performs single-platform data quality detection and cross-platform data quality detection in the memory. The data quality detection based on the memory can flexibly cope with different scenarios of data quality detection, and can meet complex and diversified data quality detection requirements. For cross-platform data quality detection, there is no need to centrally store data of two platforms, thereby reducing the workload of data extraction and not occupying extra disk storage space.

[0134] In an embodiment, the data pulling module 10 is specifically configured to perform the following steps:

[0135] A detection task model is constructed according to a data detection configuration parameter field, and the detection task model comprises a detection item id and data source information, library table information and rule fields of at least one target platform;

[0136] The detection task model is parsed into a json structure, and a detection item id and data source information, library table information and rule fields of at least one target platform are obtained, wherein the detection item id is used to obtain a detection code class name to poll data quality detection rules according to the detection code class name;

[0137] Pull at least one target platform memory stored data into memory for data quality detection.

[0138] In an embodiment, the pulling module 10 is also configured to obtain a partition expression of the target platform, and the partition expression is used to define a data boundary.

[0139] In an embodiment, the check items configured in the pre-configured data quality detection rule library include a numerical reasonableness check item, a field legality check item, a data correlation check item, a data type mapping check item, a code value mapping check item, and a general check item.

[0140] In an embodiment, the same-platform data detection module 20 is specifically configured to:

[0141] Parse the rule field of a single platform to obtain the operation column in the data table corresponding to the platform that needs to be verified;

[0142] At least one of the numerical reasonableness check item, the field legality check item, the data correlation check item, and the general check item is used to process the operation column, and data quality check is performed on the related data in the data table.

[0143] In an embodiment, the cross-platform data detection module 30 specifically includes:

[0144] A correlation check unit is configured to associate the rule fields corresponding to different target platforms, perform cross-platform data correlation check, and filter out data and quantities that cannot be associated;

[0145] A code value mapping check unit is configured to associate the rule fields corresponding to different target platforms, perform code value mapping check, and filter out data whose code and code value cannot be mapped;

[0146] A data type mapping check unit is configured to perform data type mapping check on the data tables corresponding to different target platforms, and judge whether the cross-platform data field types are one-to-one mapped.

[0147] In an embodiment, the data type mapping check unit is specifically configured to:

[0148] Based on the data sources, database instances, and data tables of different target platforms, the data structure of the data table corresponding to each target platform is obtained and loaded into the memory of the computing engine;

[0149] According to the type of the data source corresponding to different target platforms, the corresponding mapping method is traversed;

[0150] The data fields of the data table corresponding to different target platforms are converted to lowercase, and the same-structure field list is obtained by matching according to the field name.

[0151] Traverse the list of the same field, match the field type mapping according to the corresponding mapping method, and screen out data that does not meet the mapping rule.

[0152] In an embodiment, the system further comprises:

[0153] The inspection item determination unit is configured to determine the inspection items required to be configured in the data quality detection rule library based on the service characteristics of the lithium battery related data to be verified, wherein the lithium battery related data includes process data, production result data and environmental situation data.

[0154] In an embodiment, the system further comprises:

[0155] The alarm module is configured to generate an alarm prompt when the target job detection is unqualified based on the same-platform data quality detection result and / or the cross-platform data quality detection result.

[0156] The analysis module is configured to determine the data quality problem existing in the library table corresponding to the target job according to the pre-constructed data quality cause analysis library.

[0157] It should be noted that other embodiments or implementation methods of the lithium battery production and manufacturing industry data quality detection system described in the present application can refer to the above method embodiments, which will not be repeated here.

[0158] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0159] In addition, the terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features limited by "first", "second" can explicitly or implicitly include at least one feature. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise specifically limited.

[0160] Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and cannot be understood as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A data quality inspection method for lithium battery manufacturing, characterized in that, The method includes: Based on data source information, database table information, and rule fields, at least one lithium battery-related data stored within a target platform is retrieved. The database table information includes the database instance corresponding to the data source information and the data table containing the lithium battery-related data to be verified under the database embodiment. The rule fields are the operation columns to be verified within the data table. Based on pre-configured data quality inspection rules and rule fields of a single platform, the relevant data in the data table of the single platform is inspected for quality, and the data quality inspection results of the same platform are obtained. Based on pre-configured data quality detection rules and rule fields corresponding to at least two platforms, cross-platform data association and merging checks are performed to obtain cross-platform data quality detection results, including: associating rule fields corresponding to different target platforms, performing cross-platform data association checks, and filtering out data and quantities that cannot be associated; associating rule fields corresponding to different target platforms, performing code-value mapping checks, and filtering out data where codes and code values ​​cannot be mapped; and performing data type mapping checks on data tables corresponding to different target platforms to determine whether cross-platform data field types are mapped one-to-one.

2. The data quality inspection method for lithium battery manufacturing as described in claim 1, characterized in that, The process of retrieving at least one lithium battery-related data stored within the target platform based on data source information, database table information, and rule fields includes: A detection task model is constructed based on the data detection configuration parameter fields. The detection task model includes the detection item ID and at least one data source information, database table information and rule fields of the target platform. The detection task model is parsed into a JSON structure to obtain the detection item ID and at least one target platform's data source information, database table information, and rule fields. The detection item ID is used to obtain the detection code class name to poll the data quality detection rules based on the detection code class name. Based on data source information, database table information, and rule fields, at least one piece of data stored within the target platform is pulled into memory for data quality testing.

3. The data quality inspection method for lithium battery manufacturing as described in claim 2, characterized in that, The detection task model further includes partitioning information, which is used to define data boundaries. The method further includes: Retrieve at least one piece of data stored within the target platform based on data source information, database table information, partition information, and rule fields.

4. The data quality inspection method for lithium battery manufacturing as described in claim 1, characterized in that, The pre-configured data quality detection rules include numerical reasonableness checks, field legality checks, data correlation checks, data type mapping checks, code value mapping checks, and general checks.

5. The data quality inspection method for lithium battery manufacturing as described in claim 4, characterized in that, The method, based on pre-configured data quality inspection rules and rule fields of a single platform, performs quality inspection on relevant data in the data table of that single platform to obtain data quality inspection results for the same platform, including: Parse the rule fields of a single platform to obtain the operation columns that need to be verified in the data table corresponding to that platform; The operation column is processed using at least one of the numerical reasonableness check item, the field legality check item, the data correlation check item, and the general check item to perform data quality checks on the relevant data in the data table.

6. The data quality inspection method for lithium battery manufacturing as described in claim 1, characterized in that, The data type mapping check for the data tables corresponding to different target platforms includes: Based on the data sources, database instances, and data tables of different target platforms, the data structure of the corresponding data tables for different target platforms is obtained and loaded into the memory of the computing engine. Iterate through the corresponding mapping methods based on the data source type for different target platforms; Convert the data fields of the data tables corresponding to different target platforms to lowercase, and match them according to the field names to obtain a list of isomorphic fields; Iterate through the list of isomorphic fields, match the field type mappings according to the corresponding mapping methods, and filter out data that does not conform to the mapping rules.

7. The data quality inspection method for lithium battery manufacturing as described in claim 1, characterized in that, The method further includes: Based on the business characteristics of the lithium battery-related data to be verified, the required inspection items are determined, wherein the lithium battery-related data includes production process data, production result data, and environmental situation data.

8. The data quality inspection method for lithium battery manufacturing as described in claim 1, characterized in that, The method further includes: When the target operation is determined to be unqualified based on the same-platform data quality inspection results and / or cross-platform data quality inspection results, an alarm prompt is generated. Based on a pre-built database of data quality causes, identify the data quality issues present in the database tables corresponding to the target job.

9. A data quality inspection system for lithium battery manufacturing, characterized in that, The detection system includes: The data retrieval module is used to retrieve at least one lithium battery-related data stored within a target platform based on data source information, database table information, and rule fields. The database table information includes the database instance corresponding to the data source information and the data table containing the lithium battery-related data to be verified under the database embodiment. The rule fields are the operation columns to be verified within the data table. The same-platform data detection module is used to perform quality detection on relevant data in the data table of a single platform based on pre-configured data quality detection rules and rule fields of a single platform, and obtain the same-platform data quality detection results. The cross-platform data inspection module is used to perform cross-platform data association and merging checks based on pre-configured data quality inspection rules and rule fields corresponding to at least two platforms, and obtain cross-platform data quality inspection results. The cross-platform data detection module includes: The correlation check unit is used to correlate rule fields corresponding to different target platforms, perform cross-platform data correlation checks, and filter out data and quantities that cannot be correlated. The code-value mapping check unit is used to associate rule fields corresponding to different target platforms, perform code-value mapping checks, and filter out data that do not match the code and code value. The data type mapping check unit is used to check the data type mapping of data tables corresponding to different target platforms to determine whether the data field types are mapped one-to-one across platforms.

Images