Data verification method and device, equipment and medium
By automating data checksum adjustment of data items in large and complex systems, the system instability caused by data item inaccuracy is solved, the accuracy of data items and system stability are improved, and labor costs and maintenance costs are reduced.
Patent Information
- Application Number
- CN202411289108.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-14
- Publication Date
- 2024-12-13
AI Technical Summary
In large and complex systems, the system cannot run stably due to inaccuracy of data items, especially when data types do not match, it may cause functional failure, abnormal errors and even system crashes. The existing technology lacks effective data item verification tools, resulting in Human resources are invested heavily and take a long time, and the results are difficult to ensure accuracy.
By obtaining data items with the same field name in each data table, determining standard data items based on the number of data items, adjusting data items of the adjustment type, using preset equipment to confirm standard data items, and automating data checksum adjustments , form standard data tables and exception data tables to improve the accuracy of data items.
It improves the accuracy of data items, reduces labor costs, improves efficiency and accuracy, ensures the stable operation and maintenance of the system, simplifies data item management, and reduces maintenance costs.
Smart Images
Figure CN119127860A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a data verification method, device, equipment and medium. Background Art
[0002] In large and complex systems, built on a microservices architecture, the business logic is intricate, often involving tens of thousands of interfaces and interactions across multiple databases, processing hundreds of thousands or even millions of data. A data item has multiple types, where the type may be an English name, a JAVA type, the database type to which the field belongs, etc. However, given the large size of the development team and the diverse skill levels, it is inevitable that data items of the same type will be inaccurate or inconsistent, such as the English name of the field, the field description, and other data items.
[0003] The inaccuracy of such data items poses a potential threat to the stable operation of the system, especially when electronic devices or software systems need to perform specific logical operations based on a preset data item. Incorrect data items may directly cause the execution path of the application logic to deviate from expectations, leading to functional failures, abnormal errors, and even system crashes. For example, if a data item of a certain type in a field is mistakenly recorded as a numeric type, but is actually stored as a text type, then when attempting to perform numerical calculations, the system will throw an exception due to data type mismatch, seriously affecting user experience and system stability. Summary of the invention
[0004] The embodiments of the present application provide a data verification method, device, equipment and medium for determining the problem in the prior art that the system cannot operate stably due to inaccurate data items.
[0005] The present invention provides a data verification method, which includes:
[0006] Acquire each data with the same field name in each data table; and acquire the data item of the type to be adjusted of each data in each data table;
[0007] For each acquired data item, determining the number of data items of the type to be adjusted in each data item that is the data item;
[0008] Determine the standard data item of the type to be adjusted of the field name according to the number of each data item of the type to be adjusted in each data;
[0009] Based on the standard data item, the data item of the to-be-adjusted type of each data of the field name in each data table is adjusted.
[0010] Further, the determining the standard data items of the type to be adjusted of the field name according to the number of the data items of the type to be adjusted in each data respectively includes:
[0011] Determine the maximum number according to the number of each data item of the type to be adjusted in each data;
[0012] According to the data items of the maximum number of corresponding data, the standard data items of the type to be adjusted of the field name are determined.
[0013] Further, the determining, according to the data item of the data corresponding to the maximum number, the standard data item of the type to be adjusted of the field name includes:
[0014] If there are at least two maximum numbers, then adding data items of data corresponding to the at least two maximum numbers into the instruction and sending it to the preset device;
[0015] A response instruction returned by the preset device is received, and the selected data item carried in the response instruction is determined as a standard data item of the type to be adjusted of the field name.
[0016] Further, adjusting the data item of the type to be adjusted for each data of the field name in each data table includes:
[0017] Obtain each target data table containing the field name; for each data of the field name, obtain the target data item of the data in each target data table, if any target data item obtained is not the standard data item, adjust the target data item corresponding to the data of the field name in each target data table to the standard data item.
[0018] Furthermore, the method further comprises:
[0019] Acquire each abnormal data in which the data item in each data is not the standard data item;
[0020] Add each abnormal data and each data item of the abnormal data to the abnormal data table;
[0021] The abnormal data table is sent.
[0022] Furthermore, the method further comprises:
[0023] Acquire each standard data in which a data item in each data is the standard data item;
[0024] Adding each standard data and the standard data item to the standard data table;
[0025] Send the standard data sheet.
[0026] The present application also provides a data verification device, the device comprising:
[0027] An acquisition module is used to acquire each data with the same field name in each data table; and acquire a data item of a type to be adjusted for each data in each data table;
[0028] A determination module is used to determine, for each acquired data item, the number of data items of the type to be adjusted in each data item; and determine the standard data item of the type to be adjusted for the field name according to the number of data items of the type to be adjusted in each data item.
[0029] A processing module is used to adjust the data item of the to-be-adjusted type for each data of the field name in each data table based on the standard data item.
[0030] Furthermore, the determination module is specifically used to determine the maximum number according to the number of data items of the type to be adjusted in each data; and determine the standard data items of the type to be adjusted of the field name according to the data items of the data corresponding to the maximum number.
[0031] Furthermore, the determination module is specifically used to add data items corresponding to the at least two maximum quantities in an instruction and send it to a preset device if there are at least two maximum quantities; receive a response instruction returned by the preset device, and determine the selected data item carried in the response instruction as a standard data item of the type to be adjusted for the field name.
[0032] Furthermore, the processing module is specifically used to obtain each target data table containing the field name; for each data of the field name, obtain the target data item of the data in each target data table; if any target data item obtained is not the standard data item, then adjust the target data item corresponding to the data of the field name in each target data table to the standard data item.
[0033] Furthermore, the processing module is also used to obtain each abnormal data whose data item in each data is not the standard data item; add each abnormal data and the data item of each abnormal data to an abnormal data table; and send the abnormal data table.
[0034] Furthermore, the processing module is also used to obtain each standard data whose data item in each data is the standard data item; add each standard data and the standard data item to a standard data table; and send the standard data table.
[0035] An embodiment of the present application further provides an electronic device, which includes at least a processor and a memory, and the processor is used to implement the steps of any of the data verification methods described above when executing a computer program stored in the memory.
[0036] An embodiment of the present application further provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the steps of the data verification method as described in any one of the above items are implemented.
[0037] The embodiment of the present application further provides a computer program product, which includes: a computer program code, when the computer program code is executed on a computer, the computer executes the steps of any of the above-mentioned data verification methods.
[0038] Because in the embodiment of the present application, the electronic device obtains each data with the same field name in each data table; and obtains the data item of the type to be adjusted for each data in each data table; for each data item obtained, determines the number of data items of the type to be adjusted in each data for the data item; determines the standard data item of the field name according to the number of data items of the type to be adjusted in each data respectively; based on the standard data item, adjusts the data item of the type to be adjusted for each data in each data table, and adjusts the data item of each data of the field name based on the standard data item, thereby improving the accuracy of the data item, and then facilitating the subsequent execution of other tasks and improving the stable operation of the system. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0040] Figure 1 A schematic diagram of a data verification process provided in an embodiment of the present application;
[0041] Figure 2 A detailed schematic diagram of a data verification process provided in an embodiment of the present application;
[0042] Figure 3 A schematic diagram of the structure of a data verification device provided in an embodiment of the present application;
[0043] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0044] In order to make the purpose, technical solutions and advantages of this application clearer, this application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the embodiments described in this application are only part of the embodiments of this application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.
[0045] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.
[0046] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.
[0047] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.
[0048] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.
[0049] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.
[0050] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description. It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.
[0051] In the technical solution of this application, the acquisition, transmission, storage, and use of data are in compliance with the requirements of relevant national laws and regulations.
[0052] Before introducing the data verification method provided in the embodiment of the present application, in order to facilitate understanding, the technical background of the embodiment of the present application is first introduced.
[0053] In large and complex systems, built on a microservices architecture, the business logic is intricate, often involving tens of thousands of interfaces and interactions across multiple databases, processing hundreds of thousands or even millions of data. A data item has multiple types, where the type may be an English name, a JAVA type, the database type to which the field belongs, etc. However, given the large size of the development team and the diverse skill levels, it is inevitable that data items of the same type will be inaccurate or inconsistent, such as the English name of the field, the field description, and other data items.
[0054] This inaccuracy in data items poses a potential threat to the stable operation of the system, especially when electronic devices or software systems need to perform specific logical operations based on a preset data item. Incorrect data items may directly cause the execution path of the application logic to deviate from expectations, leading to functional failures, abnormal errors, and even system crashes. For example, if a data item of a certain type in a field is mistakenly recorded as a numeric type, but is actually stored as a text type, then when attempting to perform numerical calculations, the system will throw an exception due to data type mismatch, seriously affecting user experience and system stability.
[0055] Currently, there is no open source data item extraction tool, so data item extraction and non-standard verification must be performed manually, which requires a large investment in human resources, is time-consuming, and the accuracy of the results is difficult to guarantee.
[0056] In view of this, an embodiment of the present application proposes a data verification method, apparatus, device and medium, the data verification method comprising: an electronic device obtains each data with the same field name in each data table; and obtains a data item of the type to be adjusted for each data in each data table; for each data item obtained, determines the number of data items of the type to be adjusted in each data item; determines the standard data items of the field name according to the number of data items of the type to be adjusted in each data; based on the standard data items, adjusts the data items of the type to be adjusted for each data of the field name in each data table.
[0057] Embodiment 1:
[0058] Figure 1 A schematic diagram of a data verification process provided in an embodiment of the present application, the process includes the following steps:
[0059] S101: Acquire each data with the same field name in each data table; and acquire a data item of a type to be adjusted for each data in each data table.
[0060] The data verification method provided in the embodiment of the present application is applied to an electronic device, which may be a smart device such as a PC or a server.
[0061] In an embodiment of the present application, when a manager needs to verify a certain type of data item, he or she can click on a preset option, such as the "Verify" option, on the first preset page through the device he or she uses or the preset device, and the electronic device will receive the verification instruction, wherein the electronic device can be a device used by the manager or a device that has a connection relationship with the device used by the manager.
[0062] At this point, the electronic device can verify the data items, and the electronic device can obtain each data table in each database. It should be noted that in actual scenarios, the field names of the data are accurate.
[0063] The field name is used to uniquely identify the data in each column in the database table, the type is the type of other information other than the field name of the data, for example, it can be an English name, database type, etc., and the data item is the specific information corresponding to the type.
[0064] Table 1 is a schematic diagram of a data table provided in an embodiment of the present application:
[0065] age English name 17 Age 16 Age 17 age
[0066] Table 1
[0067] The first line of Table 1 is the field name and type, and the second to fourth lines of Table 1 are the data corresponding to the field name and the data items corresponding to the type. Specifically, "17", "16" and "17" are all data, "age" is the field name, "English name" is the type, and "Age", "Age" and "age" are data items.
[0068] In order to verify the data items, the electronic device may obtain each data with the same field name in each data table, and after obtaining each data, the electronic device may obtain the data item of the type to be adjusted of each data in each data table.
[0069] After obtaining the field names of the data and the data items of the type to be adjusted, the obtained field names and data items of each data can be added to the full data item information table. The field names of the data and the data items of the type to be adjusted can be referred to as full data item information.
[0070] Specifically, the electronic device can traverse each data table in the database in turn for each database; obtain the field name of the data in each data table, and each data item of the type to be adjusted, wherein the type to be adjusted includes the database name, data table name, database type, field description, etc., and the data item of the type to be adjusted can be recorded as dataItem; determine dataItem as the full data item information (all_dataitem_info); and batch insert allDataItem_info into the full data item information table (allDataItemList).
[0071] S102: for each acquired data item, determine the number of data items of the type to be adjusted in each data item.
[0072] After acquiring each data item, the electronic device may determine, for each acquired data item, the number of data items of the to-be-adjusted type in each data of the field name that are data of the data item.
[0073] In actual application scenarios, electronic devices can use a key-value storage structure to track the number of data items in each data item. Specifically, the electronic device can obtain the data item of the type to be adjusted in each data item in turn, check whether the data item already exists as a key in the record, and if it does, directly find the Value corresponding to the key, that is, the number of occurrences of the data item, and add 1 to the Value; if it does not exist, use the data item as a new key and initialize its Value to 1 to record the first occurrence of the data item.
[0074] Specifically, a mapping named dataItemCountMap is defined, which is used to store the relationship between the field name and the number of corresponding data items, specifically: <field name, <number of data items>>. The key of this mapping is the field name, which can be called dataItemKey here, and the value is another mapping, namely dbTypeCountMap, which records each data item and its number in detail, specifically: <number of data items>. <number of data items> is a mapping with data item as the primary key, which is called dataItemDBType here. The value of <number of data items> records the number of each data item, which is called dataItemCount here.
[0075] S103: Determine the standard data items of the type to be adjusted of the field name according to the number of the data items of the type to be adjusted in each data.
[0076] After obtaining the number of data items of the type to be adjusted in each data, in a possible implementation, the electronic device can determine, for each data item, the number of data items of the type to be adjusted for each data, and determine the ratio of the number to the total number of data. After determining the ratio corresponding to each data item, it can be determined whether there is a data item with a ratio greater than a preset ratio. If there is no data item with a ratio greater than the preset ratio, the field name, each data item and the ratio corresponding to each data item are sent to the preset device. The administrator selects the standard data item corresponding to the field name on the preset page of the preset device and clicks a preset button, such as an "OK" button. The electronic device can receive a confirmation instruction returned by the preset device and can determine the data item carried in the confirmation instruction as the standard data item of the type to be adjusted for the field name.
[0077] After determining the standard data item of the type to be adjusted, the electronic device may add the field name and the standard data item corresponding to the type to be adjusted to the standard data item information table.
[0078] S104: Based on the standard data item, adjust the data item of the to-be-adjusted type for each data of the field name in each data table.
[0079] After determining the standard data item corresponding to the field name, the electronic device may adjust the data item of each data of the type to be adjusted corresponding to the field name in each data table. In a possible implementation, the electronic device may, for each data in each data table, adjust the data item of the type to be adjusted to the standard data item if the field name of the data is the field name and the data item of the type to be adjusted of the data is not the standard data item.
[0080] The embodiment of the present application can also be referred to as a data item extraction and irregularity verification solution.
[0081] It should be noted that in the related art, data items are manually analyzed and selected from logs, interface documents or databases, and irregular use of data items is manually verified. The embodiment of the present application realizes automatic extraction of data items from a database, and intelligent verification and output of irregular use of data items.
[0082] The embodiments of the present application can automatically extract information from data and perform irregular verification, reduce labor costs, improve efficiency and accuracy, and automatically verify irregular data items. The data items in the entire system are standardized and consistent, which improves the maintainability and readability of the system and is beneficial to the management of the system. It is beneficial to the management of all aspects of the system throughout its life cycle, improves the maintainability and readability of the system, reduces maintenance costs, helps developers understand the system more quickly, improves development efficiency, and quickly and accurately locates and solves problems. New employees can also become familiar with and master system design more quickly.
[0083] Because in the embodiment of the present application, the electronic device obtains each data with the same field name in each data table; and obtains the data item of the type to be adjusted for each data in each data table; for each data item obtained, determines the number of data items of the type to be adjusted in each data item; determines the standard data item of the field name according to the number of data items of the type to be adjusted in each data item; based on the standard data item, adjusts the data item of the type to be adjusted for each data in each data table, thereby improving the accuracy of the data item, and then facilitating the subsequent execution of other tasks and improving the stable operation of the system.
[0084] Embodiment 2:
[0085] In order to accurately determine the standard data items, based on the above embodiment, in the embodiment of the present application, the data items of the type to be adjusted in each data are respectively the number of each data item, and the standard data items of the type to be adjusted of the field name are determined, including:
[0086] Determine the maximum number according to the number of each data item of the type to be adjusted in each data;
[0087] According to the data items of the maximum number of corresponding data, the standard data items of the type to be adjusted of the field name are determined.
[0088] In actual scenarios, the field names of the data are accurate, and electronic devices usually regard data items with higher frequencies as standard data items of the field names.
[0089] Specifically, the electronic device can determine the maximum number of each data item based on the number of data items of the type to be adjusted in each data. Since the data items of most data in each data are data items corresponding to the maximum number, the data items corresponding to the maximum number can be determined as standard data items of the type to be adjusted for the field name.
[0090] In order to accurately determine the standard data items, based on the above embodiments, in the embodiment of the present application, the data items corresponding to the maximum number of data items determine the standard data items of the type to be adjusted of the field name, including:
[0091] If there are at least two maximum numbers, then adding data items of data corresponding to the at least two maximum numbers into the instruction and sending it to the preset device;
[0092] A response instruction returned by the preset device is received, and the selected data item carried in the response instruction is determined as a standard data item of the type to be adjusted of the field name.
[0093] In actual scenarios, there may be multiple data items whose corresponding quantities are all maximum quantities. In this case, in order to determine the standard data item, the management personnel may determine the standard data item from the data items corresponding to the multiple maximum quantities.
[0094] Specifically, if there are at least two maximum numbers, the electronic device can add the data items corresponding to the at least two maximum numbers in the instruction and send it to the preset device. In order to facilitate the administrator to determine the standard data items, the instruction can also carry the field name. The administrator can select the data item from the at least two data items displayed on the preset page of the preset device. After the administrator completes the selection and clicks the preset button, such as the "OK" button, the preset device can return a response instruction to the electronic device, and the response instruction carries the selected data item. After receiving the response instruction returned by the preset device, the electronic device can obtain the data item carried in the response instruction and determine the data item as the standard data item of the type to be adjusted for the field name.
[0095] Embodiment 3:
[0096] In order to accurately perform data verification, based on the above embodiments, in the embodiment of the present application, the data item of the type to be adjusted for each data of the field name in each data table is adjusted, including:
[0097] Obtain each target data table containing the field name; for each data of the field name, obtain the target data item of the data in each target data table, if any target data item obtained is not the standard data item, adjust the target data item corresponding to the data of the field name in each target data table to the standard data item.
[0098] The electronic device can obtain each target data table containing the field name in each data table. For each data of the field name, the target data item of the type to be adjusted of the data in each target data table is obtained. It should be noted that multiple target data tables may contain the data. If any of the obtained target data items is not a standard data item, it can be determined that the target data item is inaccurate, and the target data item corresponding to the data in each target data table can be adjusted to a standard data item.
[0099] Figure 2 A detailed schematic diagram of a data verification process provided in an embodiment of the present application, the process includes the following steps:
[0100] S201: Obtain each data with the same field name in each data table.
[0101] S202: Obtain a data item of each data in each data table.
[0102] S203: Determine the maximum number according to the number of data items of each data.
[0103] S204: Determine the data item corresponding to the maximum number of data as the standard data item of the type to be adjusted of the field name.
[0104] S205: Based on the standard data items, adjust the data items of the to-be-adjusted type for each data in the field name of each data table.
[0105] In order to facilitate data tracing, based on the above embodiments, in the embodiment of the present application, the method further includes:
[0106] Acquire each abnormal data in which the data item in each data is not the standard data item;
[0107] Add each abnormal data and each data item of the abnormal data to the abnormal data table;
[0108] The abnormal data table is sent.
[0109] To facilitate data tracing, the electronic device can determine, for each data with the same field name, whether the data item of the type to be adjusted is a standard data item, and if it is not a standard data item, the data can be determined to be abnormal data. In this way, the electronic device can determine each abnormal data in each data.
[0110] After acquiring each abnormal data, the electronic device may add each abnormal data and the data item of the to-be-adjusted type of each abnormal data to the abnormal data table.
[0111] The electronic device can send the abnormal data table. Specifically, the abnormal data table can be sent to a preset device. When the staff has information to trace the detailed information of the data, they can view the abnormal data table.
[0112] In order to facilitate data tracing, based on the above embodiments, in the embodiment of the present application, the method further includes:
[0113] Acquire each standard data in which a data item in each data is the standard data item;
[0114] Adding each standard data and the standard data item to the standard data table;
[0115] Send the standard data sheet.
[0116] In order to facilitate data tracing, the electronic device can determine whether the data item of the to-be-adjusted type of the data is a standard data item for each data with the same field name, and if it is a standard data item, the data can be determined to be standard data. In this way, the electronic device can determine each standard data in each data.
[0117] After acquiring each standard data, the electronic device may add each standard data and the data item of the to-be-adjusted type of each standard data to the standard data table.
[0118] The electronic device can send the standard data sheet. Specifically, the standard data sheet can be sent to a preset device. When the staff has information to trace the detailed information of the data, they can view the standard data sheet.
[0119] In an embodiment of the present application, the electronic device can also construct a verification data item information table, which includes a field name, a standard data item of the type to be adjusted corresponding to the field name, and a data item of the type to be adjusted for each abnormal data.
[0120] The embodiment of the present application is divided into three parts: initial extraction of full data items, selection of standard data items, and output of non-standard data items. The specific scheme is: first, connect to the database, obtain the field names of all data in all data tables, and different types of data items to initialize the full data item information; second, count the field names and data items of a certain type, and select the data items with the most counts as standard data items for the data items with the same field names; finally, output the inaccurate data corresponding to the data item as non-standard data.
[0121] Embodiment 4:
[0122] Specifically, the process of determining standard data items is as follows:
[0123] 1) Connect to the database, obtain each full data item in all_dataitem_info, recorded as allDataItemList; traverse the full data item data allDataItemList.
[0124] 2.1) For each dateItem, the field name is dataItemKey, the data item is dataItemDBType, and the number of occurrences is dataItemCount;
[0125] 2.2) If dataItemCountMap gets the key value based on dataItemKey and it is empty, <dataItemKey,<dataItemDBType,1> >put into dataItemCountMap;
[0126] 2.3) If the key value obtained by dataItemCountMap according to dataItemKey is not empty, the key value obtained is dbTypeCountMap;
[0127] 2.3.1) If the dbTypeCountMap key value obtained according to dataItemDBType is empty, <dataItemKey,<dataItemDBType,1> >put into dataItemCountMap;
[0128] 2.3.2) If dbTypeCountMap gets the key value according to dataItemDBType and it is empty, and the key value count is dataItemCount, then <dataItemKey,<dataItemDBType,dataItemCount+1> >put into dataItemCountMap;
[0129] Traverse the key value dataItemKey of dataItemCountMap.
[0130] 3.1) Get the corresponding dbTypeCountMap from dataItemCountMap according to dataItemKey, and select the dataItemDBType with the largest dataItemCount in dbTypeCountMap as the standard data item corresponding to the database type;
[0131] 3.2) According to dataItemKey and dataItemDBType, obtain the standard data item dataitem_desc corresponding to this type of description information from the all_dataitem_info table;
[0132] 3.3) dataItemKey, dataItemDBType, and dataitem_desc are inserted into the dataitem_info table.
[0133] Automatically check irregular data items, specifically:
[0134] 1) Connect to the database and obtain all the data items in the all_dataitem_info table, recorded as allDataItemList;
[0135] 2) Connect to the database and obtain all the standard data items in the dataitem_info table, recorded as dataItemList;
[0136] Traverse the dataItem elements in allDataItemList one by one;
[0137] 3.1) If the database type of dataItem is consistent with the corresponding dataItemList, it is ignored;
[0138] 3.2) If the database type of dataItem is consistent with that in the corresponding dataItemList, encapsulate the inconsistent information and add it to the diffDataItemList list;
[0139] 4) Insert the diffDataItemList list information into the check_dataitem_info table in batches.
[0140] Embodiment 5:
[0141] Based on the same inventive concept, the embodiment of the present application provides a data verification device. Figure 3 For a schematic diagram of a data verification device structure provided in an embodiment of the present application, please refer to Figure 3 , the device comprises:
[0142] The acquisition module 301 is used to acquire each data with the same field name in each data table; and acquire the data item of the type to be adjusted of each data in each data table;
[0143] A determination module 302 is used to determine, for each acquired data item, the number of data items of the type to be adjusted in each data item; and determine the standard data items of the type to be adjusted of the field name according to the number of data items of the type to be adjusted in each data item.
[0144] The processing module 303 is used to adjust the data item of the to-be-adjusted type for each data of the field name in each data table based on the standard data item.
[0145] In a possible implementation, the determination module 302 is specifically used to determine the maximum number according to the number of data items of the type to be adjusted in each data; and determine the standard data items of the type to be adjusted for the field name according to the data items of the data corresponding to the maximum number.
[0146] In one possible implementation, the determination module 302 is specifically used to add data items corresponding to the at least two maximum quantities in an instruction and send it to a preset device if there are at least two maximum quantities; receive a response instruction returned by the preset device, and determine the selected data item carried in the response instruction as a standard data item of the type to be adjusted for the field name.
[0147] In a possible implementation, the processing module 303 is specifically used to obtain each target data table containing the field name; for each data of the field name, obtain the target data item of the data in each target data table, and if any target data item obtained is not the standard data item, then adjust the target data item corresponding to the data of the field name in each target data table to the standard data item.
[0148] In a possible implementation, the processing module 303 is further used to obtain each abnormal data whose data item in each data is not the standard data item; add each abnormal data and the data item of each abnormal data to an abnormal data table; and send the abnormal data table.
[0149] In a possible implementation, the processing module 303 is further used to obtain each standard data in which the data item in each data is the standard data item; add each standard data and the standard data item to a standard data table; and send the standard data table.
[0150] Embodiment 6:
[0151] Based on the same inventive concept, an embodiment of the present application provides an electronic device, which can implement the steps of the data verification method discussed above. Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, it includes: a processor 401, a communication interface 402, a memory 403 and a communication bus 404, wherein the processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404;
[0152] The memory 403 stores a computer program. When the program is executed by the processor 401, the processor 401 performs the following steps:
[0153] Acquire each data with the same field name in each data table; and acquire the data item of the type to be adjusted of each data in each data table;
[0154] For each acquired data item, determining the number of data items of the type to be adjusted in each data item that is the data item;
[0155] Determine the standard data item of the type to be adjusted of the field name according to the number of each data item of the type to be adjusted in each data;
[0156] Based on the standard data item, the data item of the to-be-adjusted type of each data of the field name in each data table is adjusted.
[0157] In a possible implementation manner, determining the standard data items of the type to be adjusted of the field name according to the number of the data items of the type to be adjusted in each data includes:
[0158] Determine the maximum number according to the number of each data item of the type to be adjusted in each data;
[0159] According to the data items of the maximum number of corresponding data, the standard data items of the type to be adjusted of the field name are determined.
[0160] In a possible implementation manner, determining the standard data item of the type to be adjusted of the field name according to the data item corresponding to the maximum number of data includes:
[0161] If there are at least two maximum numbers, then adding data items of data corresponding to the at least two maximum numbers into the instruction and sending it to the preset device;
[0162] A response instruction returned by the preset device is received, and the selected data item carried in the response instruction is determined as a standard data item of the type to be adjusted of the field name.
[0163] In a possible implementation manner, adjusting the data item of the to-be-adjusted type of each data of the field name in each data table includes:
[0164] Obtain each target data table containing the field name; for each data of the field name, obtain the target data item of the data in each target data table, if any target data item obtained is not the standard data item, adjust the target data item corresponding to the data of the field name in each target data table to the standard data item.
[0165] In a possible implementation, the method further includes:
[0166] Acquire each abnormal data in which the data item in each data is not the standard data item;
[0167] Add each abnormal data and each data item of the abnormal data to the abnormal data table;
[0168] The abnormal data table is sent.
[0169] In a possible implementation, it is characterized in that the method further includes:
[0170] Acquire each standard data in which a data item in each data is the standard data item;
[0171] Adding each standard data and the standard data item to the standard data table;
[0172] Send the standard data sheet.
[0173] Since the principle of solving the problem by the above electronic device is similar to that of the data verification method, the implementation of the above electronic device can refer to the embodiment of the method, and the repeated parts will not be repeated.
[0174] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface 502 is used for communication between the above electronic device and other devices. The memory may include a random access memory (RAM) and may also include a non-volatile memory (NVM), such as at least one disk storage. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0175] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it can also be a digital signal processing processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.
[0176] Embodiment 7:
[0177] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, in which a computer program executable by a processor is stored, and when the program is run on the processor, the processor executes any of the data verification methods discussed above. Since the principle of solving the problem by the above-mentioned computer-readable storage medium is similar to that of the data verification method, the implementation of the above-mentioned computer-readable storage medium can refer to the implementation of the method, and the repeated parts will not be repeated.
[0178] Embodiment 8:
[0179] Based on the same inventive concept, the embodiment of the present application also provides a computer program product, which includes: computer program code, when the computer program code is run on a computer, the computer executes any of the data verification methods discussed above. Since the principle of solving the problem by the above computer program product is similar to that of the data verification method, the implementation of the above computer program product can refer to the implementation of the method, and the repeated parts will not be repeated.
[0180] In an embodiment of the present application, the electronic device obtains each data with the same field name in each data table; and obtains the data item of the type to be adjusted for each data in each data table; for each data item obtained, determines the number of data items of the type to be adjusted in each data item; determines the standard data item of the field name according to the number of data items of the type to be adjusted in each data item; based on the standard data item, adjusts the data item of the type to be adjusted for each data in each data table, and adjusts the data item of each data of the field name based on the standard data item, thereby improving the accuracy of the data item, and then facilitating the subsequent execution of other tasks and improving the stable operation of the system.
[0181] The above-mentioned computer-readable storage medium can be any available medium or data storage device that can be accessed by the processor in the electronic device, including but not limited to magnetic storage such as floppy disks, hard disks, magnetic tapes, MO (magneto-optical disks), etc., optical storage such as CD, DVD, BD, HVD, etc., and semiconductor storage such as ROM, EPROM, EEPROM, NAND FLASH (non-volatile memory), SSD (solid state drive), etc.
[0182] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0183] Based on the same technical concept, an embodiment of the present application provides a computer program product, which includes: computer program code, when the computer program code runs on an electronic device, the electronic device implements any of the above embodiments when executing.
[0184] The computer program for performing the operation of the present disclosure may be an assembly instruction, an instruction set architecture (ISA) instruction, a machine instruction, a machine-related instruction, a microcode, a firmware instruction, a state setting data, or a source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.
[0185] The computer program product described herein may be implemented in hardware, software, or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK), etc.
[0186] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0187] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0188] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0189] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. A data verification method, characterized in that: The method comprises: Acquire each data with the same field name in each data table; and acquire the data item of the type to be adjusted of each data in each data table; For each acquired data item, determining the number of data items of the type to be adjusted in each data item that is the data item; Determine the standard data item of the type to be adjusted of the field name according to the number of each data item of the type to be adjusted in each data; Based on the standard data item, the data item of the to-be-adjusted type of each data of the field name in each data table is adjusted.
2. The method according to claim 1, characterized in that The determining, according to the data items of the type to be adjusted in each data being the quantity of each data item, the standard data items of the type to be adjusted of the field name comprises: Determine the maximum number according to the number of each data item of the type to be adjusted in each data; According to the data items of the maximum number of corresponding data, the standard data items of the type to be adjusted of the field name are determined.
3. The method according to claim 2, characterized in that The step of determining the standard data item of the type to be adjusted of the field name according to the data item corresponding to the maximum number of data includes: If there are at least two maximum numbers, then adding data items of data corresponding to the at least two maximum numbers into the instruction and sending it to the preset device; A response instruction returned by the preset device is received, and the selected data item carried in the response instruction is determined as a standard data item of the type to be adjusted of the field name.
4. The method according to claim 1, characterized in that: Adjusting the data item of the to-be-adjusted type for each data of the field name in each data table includes: Obtain each target data table containing the field name; for each data of the field name, obtain the target data item of the data in each target data table, if any target data item obtained is not the standard data item, adjust the target data item corresponding to the data of the field name in each target data table to the standard data item.
5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Acquire each abnormal data in which the data item in each data is not the standard data item; Add each abnormal data and each data item of the abnormal data to the abnormal data table; The abnormal data table is sent.
6. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Acquire each standard data in which a data item in each data is the standard data item; Adding each standard data and the standard data item to the standard data table; Send the standard data sheet.
7. A data verification device, characterized in that: The device comprises: An acquisition module is used to acquire each data with the same field name in each data table; and acquire a data item of a type to be adjusted for each data in each data table; A determination module is used to determine, for each acquired data item, the number of data items of the type to be adjusted in each data item; and determine the standard data items of the type to be adjusted for the field name according to the number of data items of the type to be adjusted in each data item; A processing module is used to adjust the data item of the to-be-adjusted type for each data of the field name in each data table based on the standard data item.
8. The device according to claim 7, characterized in that The determination module is specifically used to determine the maximum number according to the number of data items of the type to be adjusted in each data; and determine the standard data items of the type to be adjusted of the field name according to the data items of the data corresponding to the maximum number.
9. The device according to claim 8, characterized in that The determination module is specifically used to add the data items corresponding to the at least two maximum quantities in the instruction and send it to the preset device if there are at least two maximum quantities; receive the response instruction returned by the preset device, and determine the selected data items carried in the response instruction as the standard data items of the type to be adjusted for the field name.
10. The device according to claim 7, characterized in that The processing module is specifically used to obtain each target data table containing the field name; for each data of the field name, obtain the target data item of the data in each target data table; if any target data item obtained is not the standard data item, then adjust the target data item corresponding to the data of the field name in each target data table to the standard data item.
11. The device according to claim 7, characterized in that The processing module is further used to obtain each abnormal data whose data item in each data is not the standard data item; add each abnormal data and the data item of each abnormal data to the abnormal data table; The abnormal data table is sent.
12. The device according to claim 7, characterized in that The processing module is further used to obtain each standard data whose data item is the standard data item in each data; and add each standard data and the standard data item to the standard data table; Send the standard data sheet.
13. An electronic device, characterized in that: The electronic device comprises at least a processor and a memory, and the processor is used to implement the steps of the data verification method as described in any one of claims 1 to 6 when executing a computer program stored in the memory.
14. A computer-readable storage medium, characterized in that: It stores a computer program, which, when executed by a processor, implements the steps of the data verification method as described in any one of claims 1 to 6.
15. A computer program product, characterized in that The computer program product comprises: a computer program code, which, when executed on a computer, enables the computer to execute the steps of the data verification method as described in any one of claims 1 to 6.