Data management method and system, electronic equipment and computer storage medium
Through automated data management methods, including de-replication, format adjustment and verification, the problem of high human resources consumption in fiscal and tax data management is solved, and efficient and low-cost data processing and storage are achieved.
Patent Information
- Application Number
- CN202411855722.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-16
- Publication Date
- 2025-04-11
AI Technical Summary
The prior art requires a lot of human resources when processing and managing fiscal and tax data, resulting in inefficiency and high cost.
Provide a data management method, including obtaining the data set to be entered, de-replication and processing, adjusting the data format, performing verification, and storing according to the data type. This method reduces manual intervention through automated processes.
By automatically processing fiscal and tax data, we can significantly save human resources, improve data management efficiency, and reduce costs.
Smart Images
Figure CN119807276A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of software engineering technology, and in particular to a data management method, system, electronic device, and computer storage medium. Background Art
[0002] With the development of information technology and the popularization of Internet applications, the informatization and electronicization of all walks of life are developing rapidly, which puts higher requirements on the management of fiscal and tax data. The existing methods for processing fiscal and tax data are mainly used for the generation of fiscal and tax data, and for the generated fiscal and tax data, a large amount of human resources are still needed to manage it.
[0003] Therefore, there is an urgent need for a new data management method to process financial and tax data in order to save human resources. Summary of the invention
[0004] In view of this, embodiments of the present application provide a data management method, system, electronic device and computer storage medium to at least partially solve the above problems.
[0005] An embodiment of the present application provides a data management method, including: obtaining a set of data to be entered; performing deduplication and merging processing on the data to be entered included in the set of data to be entered to obtain preprocessed data; adjusting the format of the data in the preprocessed data to a predetermined format to obtain standard data; verifying the standard data, and storing the verified standard data according to the data type of the standard data.
[0006] In one possible implementation, the de-duplication and merging processing of the data to be entered included in the data set to be entered includes: determining, according to the generation frequency of the data to be entered, a target rule corresponding to the generation frequency from multiple de-duplication and merging rules, wherein the de-duplication and merging rule is related to the generation frequency; and de-duplication and merging processing of the data to be entered based on the de-duplication and merging rule.
[0007] In a possible implementation, the type of generation frequency includes at least one of the following: one-time generation, generation at a periodic frequency, generation at an indefinite frequency, and non-definite generation.
[0008] In a possible implementation, adjusting the format of the data in the preprocessed data to a predetermined format includes: if the data type of the preprocessed data is a financial statement, determining the accounting standard type of the preprocessed data, and adding the determined accounting standard type to the preprocessed data.
[0009] In a possible implementation, the checking of the standard data includes: detecting whether there is abnormal data in the standard data, and if the abnormal data exists, determining whether the abnormal data can be corrected according to the cross-reference relationship; if it can be corrected, correcting the abnormal data according to the cross-reference relationship; if it cannot be corrected, discarding the abnormal data.
[0010] In a possible implementation, obtaining the data set to be entered includes: obtaining the original data of the data to be entered; determining a data extraction rule based on the data source of the original data; the data extraction rule is used to indicate target data to be extracted from the original data; according to the data extraction rule, extracting the target data from the original data to obtain the data to be entered, and obtaining the data set to be entered based on the data to be entered.
[0011] In a possible implementation, the method further includes: determining a data set corresponding to the data item from multiple data sets based on the data item in the verified standard data, the data set corresponding to the data item being a data set for storing the data corresponding to the data item; and storing the data corresponding to the data item in the verified standard data in the data set corresponding to the data item.
[0012] An embodiment of the present application also provides a data management system, including: a data acquisition module, used to acquire a set of data to be entered; a de-duplication and merging module, used to perform de-duplication and merging processing on the data to be entered included in the set of data to be entered, so as to obtain pre-processed data; a standardization module, used to adjust the format of the data in the pre-processed data to a predetermined format, so as to obtain standard data; a verification module, used to verify the standard data; and a storage module, used to store the verified standard data according to the data type of the standard data.
[0013] An embodiment of the present application also provides an electronic device, including: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus; the memory is used to store at least one computer software, and the computer software enables the processor to execute the method described in any of the above embodiments.
[0014] An embodiment of the present application further provides a computer storage medium on which a computer program is stored. When the program is executed by a processor, the method described in any of the above embodiments is implemented.
[0015] In an embodiment of the present application, a data set to be entered can be obtained, and the data to be entered included in the data set to be entered can be de-duplicated and merged to obtain pre-processed data. The format of the data in the pre-processed data is adjusted to a predetermined format to obtain standard data. The standard data is verified, and the verified standard data is stored according to the data type of the standard data. When financial and tax data is used as the data to be entered, the data to be entered can be obtained, and multiple processes such as de-duplication and merging, format adjustment, and verification can be performed in sequence, without the need for manual intervention, which can significantly save human resources. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.
[0017] Figure 1 It is a flowchart of the steps of a data management method provided by an embodiment of the present application;
[0018] Figure 2 This is a method provided by the embodiment of the present application. Figure 1 The step flow chart of S110;
[0019] Figure 3 This is a method provided by the embodiment of the present application. Figure 1 The step flow chart of S140;
[0020] Figure 4 It is a structural block diagram of a data management system provided by an embodiment of the present application;
[0021] Figure 5 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0022] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the embodiments of the present application should fall within the scope of protection of the embodiments of the present application.
[0023] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.
[0024] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".
[0025] The embodiments of the present application provide a data management method to at least partially solve the above problems.
[0026] The data management method provided in the embodiment of the present application is described in detail below with reference to the accompanying drawings.
[0027] like Figure 1 As shown, the embodiment of the present application provides a data management method, including:
[0028] S110: Obtain a data set to be entered.
[0029] In the embodiment of the present application, the data set to be entered can be obtained from a memory storing the data set to be entered, and the data set to be entered can be obtained by collecting the data to be entered. The data to be entered in the data set to be entered can be fiscal and taxation data, or other suitable data. It should be understood that finance and taxation are abbreviations for finance and taxation, and fiscal and taxation data can be understood as fiscal data and taxation data.
[0030] like Figure 2 As shown, as an optional embodiment, obtaining the data set to be entered may include:
[0031] S111: Obtaining the original data to be entered.
[0032] The original data of the data to be entered may be data from different channels. Optionally, the embodiment of the present application can obtain the original data of the data to be entered through RPA robots, ETL (Extract-Transform-Load) tools, API interfaces, etc. Of course, the original data of the data to be entered can also be obtained through other suitable methods, such as batch import methods, etc. The embodiment of the present application does not limit the method of obtaining the original data of the data to be entered. The obtained original data can be data or files in a variety of data formats, such as PDF files, HTML files, table files, text files, JSON data, OFD files, etc.
[0033] S112: Determine a data extraction rule according to the data source of the original data; the data extraction rule is used to indicate the target data to be extracted from the original data.
[0034] The target data may include data items and data corresponding to the data items. Data items are also called data fields. Each data item is used to indicate a feature, such as "company name", "data ID", "generation time", "time period", "amount", etc. The data corresponding to the data item can be understood as the data belonging to the data item, that is, the specific content of the data item. For example, the data corresponding to the data item "company name" can be XX Company, and the data corresponding to the data item "amount" can be 3,000 yuan, etc.
[0035] As an optional implementation, this embodiment can set a corresponding data extraction rule for each data source. The data extraction rule indicates the target data to be extracted from the original data of each data source.
[0036] When extracting data, you can call a data extraction tool to extract data according to the data format of the original data, such as calling an OCR (Optical Character Recognition) recognition tool, a semantic recognition tool, a format conversion tool, etc. to extract data. The specific data extraction process can refer to the relevant technology and will not be described here. The data extraction rule can also be used to indicate the data extraction tool to be called when extracting data.
[0037] S113: extracting target data from the original data according to the data extraction rule, obtaining the data to be entered, and obtaining a data set to be entered based on the data to be entered.
[0038] After determining the data extraction rules according to the data source of the original data, the corresponding data extraction tool can be called to extract the target data indicated by the data extraction rules from the original data to obtain the data to be entered. It should be understood that the data set to be entered is the set of data to be entered, and the data set to be entered can include one or more data to be entered.
[0039] As a feasible implementation method, when extracting target data from raw data, the data items in the raw data can be determined, and the data type of the data to be entered can be identified. The data type of the data to be entered can include invoices, receipts, financial statements, etc. For example, the data type of the data to be entered can be identified based on the text or number in the raw data to be entered. For details, please refer to the relevant technology, which will not be described here.
[0040] After extracting the target data, the data to be entered can be generated based on the extracted target data, and the data type, data source, and data source of the data to be entered can be marked in the data to be entered. The original data of the data to be entered can carry the data source identifier of the original data, and its data source can be determined based on the data source identifier.
[0041] In an embodiment of the present application, a corresponding data extraction rule can be preset for each data source. After the original data of the data to be entered is obtained, the corresponding data extraction rule is determined according to the data source of the original data. According to the data extraction rule, the target data is extracted from the original data to obtain the data to be entered. Therefore, when faced with original data with multiple data sources, the corresponding data extraction rule can be selected according to each data source, and the target data can be automatically extracted from the original data of the data source to obtain the data to be entered without the need for manual classification and sorting, which can effectively save human resources.
[0042] S120: performing de-duplication and merging processing on the data to be entered included in the data set to be entered to obtain pre-processed data.
[0043] In the input data set, there may be duplicate data to be input, and data to be input that can be merged. In the embodiment of the present application, the data to be input included in the input data set can be deduplicated and merged to detect whether there are duplicate data to be input. If so, only one copy of the data to be input can be retained by removing it. If not, the next step can be continued. In addition, it can be detected whether there is data to be input that can be merged. If so, it can be merged. If not, the next step can be continued.
[0044] As an optional embodiment, performing de-duplication and merging processing on the data to be entered included in the data set to be entered includes:
[0045] According to the generation frequency of the data to be entered, a target rule corresponding to the generation frequency is determined from multiple deduplication and merging rules, wherein the deduplication and merging rule is related to the generation frequency, and according to the deduplication and merging rule, the data to be entered is deduplication and merged.
[0046] Different types of generation frequencies of data to be entered may have different types. The generation frequency corresponding to each type of data to be entered may be recorded, thereby determining the generation frequency of the data to be entered according to the data type of the data to be entered.
[0047] When performing de-duplication and merging processing on the data to be input, the generation frequencies of all the data to be input may be determined, and then de-duplication and merging processing may be performed on the data to be input with the same generation frequency.
[0048] In an embodiment of the present application, deduplication and merging rules corresponding to each generation frequency can be set in advance based on various generation frequencies of the data to be entered, and a deduplication and merging rule base including multiple deduplication and merging rules can be constructed. When deduplication and merging is performed on the data to be entered, the deduplication and merging rule corresponding to the generation frequency can be determined from multiple deduplication and merging rules according to the generation frequency of the data to be entered, and the deduplication and merging rule can be determined as the target rule, and according to the target rule, the data to be entered can be deduplicated and merged. By setting the deduplication and merging rules related to the generation frequency, special deduplication and merging rules can be used to perform deduplication and merging for the data to be entered with different generation cycles, which is beneficial to improving the effect of deduplication and merging.
[0049] In some optional embodiments, the type of generation frequency includes at least one of the following: one-time generation, generation at a periodic frequency, generation at an indefinite frequency, and non-definite generation.
[0050] One-time generation can be understood as being generated only once, and data generated once usually only has one version.
[0051] The data to be entered with a one-time generation frequency can be recorded as the first data to be entered. The de-duplication merging rule corresponding to the one-time generation frequency can be: compare the data items of each first data to be entered, if there are at least two first data to be entered with the same data items, then retain the first data to be entered with the latest acquisition time among the at least two first data to be entered, and discard the other first data to be entered among the at least two first data to be entered. If there are no at least two first data to be entered with the same data items, then retain each first data to be entered.
[0052] It should be noted that the time for obtaining a certain data in the embodiment of the present application is the time when the data is obtained during the implementation of the method.
[0053] Generating at a periodic frequency can be understood as generating according to a predetermined period, and the period can be year, month, season, etc.
[0054] The data to be entered that is generated at a periodic frequency can be recorded as the second data to be entered. The data item of the data to be entered can include a data ID and a period. The data ID can be understood as an identifier that is unique to an item of data to be entered, which can be a number, a code, etc.
[0055] The de-duplication and merging rules corresponding to the generation frequency generated by the periodic frequency can be: compare the data corresponding to the data ID of each second data to be entered and the data corresponding to the period to which it belongs. If there are at least two second data to be entered whose data corresponding to the data ID and the data corresponding to the period to which it belongs are the same, then retain the data to be entered with the latest acquisition time among the at least two second data to be entered, and discard the other data to be entered that are the same as the at least two second data to be entered. If there are no at least two second data to be entered whose data corresponding to the data ID and the data corresponding to the period to which it belongs are the same, proceed to the next step. The embodiment of the present application only needs to compare the data ID and the period to which it belongs of the second data to be entered, without having to compare all the data items of the second data to be entered, which can effectively improve the efficiency of de-duplication and merging. It should be understood that the data corresponding to the data ID is the data content of the data ID, and the data corresponding to the period to which it belongs is the data content of the period to which it belongs.
[0056] Generating at an irregular frequency can be understood as generating at irregular intervals or not generating on a regular basis. For example, it can be generated at irregular intervals according to the actual operating conditions of the enterprise.
[0057] The data to be recorded that is generated at an indefinite frequency can be recorded as the third data to be recorded. The data item of the data to be recorded can also include the generation time.
[0058] The de-duplication merging rule corresponding to the generation frequency generated at an indefinite frequency may be: compare the data corresponding to the data ID of each third data to be entered and the data corresponding to the generation time. If there are at least two third data to be entered whose data corresponding to the data ID and the data corresponding to the generation time are the same, then retain the data to be entered with the latest acquisition time among the at least two third data to be entered, and discard the other data to be entered among the at least two third data to be entered. If there are no at least two third data to be entered whose data corresponding to the data ID and the data corresponding to the generation time are the same, proceed to the next step.
[0059] Non-necessarily generated can be understood as not necessarily generated, for example, negative data such as industrial and commercial penalty data that are not necessarily generated and to be entered.
[0060] The data to be entered that is not necessarily generated can be recorded as the fourth data to be entered. The de-duplication and merging rules corresponding to the generation frequency that is not necessarily generated can be similar to the de-duplication and merging rules generated once, that is: compare the data items of each fourth data to be entered, if there are at least two fourth data to be entered with the same data items, then retain the fourth data to be entered with the latest acquisition time among the at least two fourth data to be entered, and discard the other fourth data to be entered in the at least two fourth data to be entered. If there are no at least two fourth data to be entered with the same data items, then retain each fourth data to be entered.
[0061] The type of generation frequency may also include annual generation, which can be understood as annual generation, that is, each item of data to be entered with an annual generation frequency is generated only once each year. It should be noted that if a certain item of data to be entered satisfies both annual generation and periodic frequency generation, the type of generation frequency of the data to be entered is determined to be annual generation; if a certain item of data to be entered satisfies both annual generation and indefinite frequency generation, the type of generation frequency of the data to be entered is determined to be annual generation.
[0062] The data to be entered that is generated on an annual basis can be recorded as the fifth data to be entered. The de-duplication and merging rules corresponding to the generation frequency generated on an annual basis can be: compare the data corresponding to the period to which each fifth data to be entered belongs. If there are at least two fifth data to be entered that have the same data corresponding to the period, the at least two fifth data to be entered can be recorded as the sixth data to be entered. Compare the data items of each sixth data to be entered. If there are at least two sixth data to be entered that have the same data items, the sixth data to be entered that has the latest acquisition time among the at least two sixth data to be entered is retained, and the other sixth data to be entered among the at least two sixth data to be entered is discarded. If there are no at least two sixth data to be entered that have the same data items, each sixth data to be entered is retained.
[0063] In an embodiment of the present application, the types of generation frequencies may include one-time generation, generation at a periodic frequency, generation at an indefinite frequency, and non-definite generation, etc., which can more comprehensively cover the types of generation frequencies of the data to be entered, so as to determine appropriate de-duplication and merging rules according to the generation frequency of the data to be entered.
[0064] S130: Adjust the format of the data in the preprocessed data to a predetermined format to obtain standard data.
[0065] The embodiment of the present application can set predetermined formats corresponding to various data items, so as to adjust the data corresponding to each data item in the preprocessed data to the predetermined format corresponding to the data item.
[0066] As an optional embodiment, adjusting the format of the data in the preprocessed data to a predetermined format includes:
[0067] If the data type of the preprocessed data is a financial statement, the accounting standard type of the preprocessed data is determined, and the determined accounting standard type is added to the preprocessed data.
[0068] The pre-processed data may be financial and tax data. Specifically, the data types of the pre-processed data may include invoices, receipts, financial statements, and the like.
[0069] It should be understood that the data type of the data to be entered, the pre-processed data obtained by de-overlapping the data to be entered, and the data type of the standard data obtained based on the pre-processed data are the same.
[0070] Accounting standards are used to define the data structure and filling requirements of financial statements. Therefore, there is a one-to-one correspondence between the accounting standard type and the data structure of the financial statement. The data structure of the financial statement can be indicated by the accounting standard type, and the accounting standard type of the financial statement can also be determined based on the data structure of the financial statement. The data structure of the financial statement is the composition structure of the financial statement, which can include data items in the financial statement.
[0071] In the embodiment of the present application, the data structure of the preprocessed data can be parsed by a suitable semantic recognition tool to determine the accounting standard type of the financial statement. Alternatively, the accounting standard type of the financial statement can be determined by other suitable methods, and the present application does not limit the method of determining the accounting standard type of the financial statement.
[0072] In an embodiment of the present application, when the data type of the preprocessed data is a financial statement, the accounting standard type of the preprocessed data can be determined, and the determined accounting standard type can be added to the preprocessed data, which can facilitate users to understand the data structure of the financial statement by giving the accounting standard type.
[0073] As a feasible implementation manner, adjusting the format of the data in the preprocessed data to a predetermined format may also include: completing the data in the preprocessed data according to the predetermined format.
[0074] Optionally, the data format corresponding to the period in the preprocessed data can be "year-month-day". If the data corresponding to the period in the preprocessed data is X year X month, X year X month is completed to "X year X month X day", which can be specifically completed to "X year X month 1 day, etc.
[0075] In an embodiment of the present application, the data corresponding to the data items in the preprocessed data can be completed according to the predetermined format corresponding to the data items in the preprocessed data, so as to ensure the integrity of the data corresponding to the data items as much as possible and make the data corresponding to the data items have good consistency.
[0076] S140: Checking the standard data, and storing the checked standard data according to the data type of the standard data.
[0077] When the standard data is checked, it is possible to detect whether there is abnormal data in the standard data, and when the abnormal data is detected, the abnormal data can be processed to ensure the validity of the checked standard data as much as possible.
[0078] The data type of the standard data is the same as the corresponding pre-processed data and the data to be entered, and the data type of the standard data can be determined according to the data type of the data to be entered corresponding to the standard data. It should be understood that the pre-processed data corresponding to the standard data is the pre-processed data of the standard data obtained by adjusting the format of the data to a predetermined format. The data to be entered corresponding to the standard data is the data to be entered that can be obtained by performing de-duplication and merging processing on the pre-processed data corresponding to the standard data.
[0079] In an embodiment of the present application, a data set to be entered can be obtained, and the data to be entered included in the data set to be entered can be de-duplicated and merged to obtain pre-processed data. The format of the data in the pre-processed data is adjusted to a predetermined format to obtain standard data. The standard data is verified, and the verified standard data is stored according to the data type of the standard data. When financial and tax data is used as the data to be entered, the data to be entered can be obtained, and multiple processes such as de-duplication and merging, format adjustment, and verification can be performed in sequence, without the need for manual intervention, which can significantly save human resources.
[0080] like Figure 3 As shown, as an optional embodiment, checking the standard data includes:
[0081] S141: Check whether there is abnormal data in the standard data, and if there is abnormal data, determine whether the abnormal data can be corrected based on the cross-reference relationship.
[0082] It should be understood that each item of data in the standard data has its corresponding data rule or data format. By detecting whether the data in the standard data conforms to the corresponding data rule or data format, it is possible to detect whether the data is abnormal data.
[0083] Exemplarily, the amount of a red ticket in an invoice is usually a negative value. If the data type of a certain standard data is an invoice, and specifically a red ticket, but the amount therein is a positive value, then the amount in the standard data can be determined to be abnormal data. If the predetermined format of amount data such as detailed amount and summary amount is Arabic numerals, but the detailed amount or summary amount in a certain standard data is detected to be Chinese characters, then the detailed amount or summary amount in the standard data can be determined to be abnormal data. If the time length of the period to which a certain standard data belongs should be a whole month, but the time length of the period to which the standard data belongs is detected to be less than one month or more than one month, then the period to which the standard data belongs can be determined to be abnormal data. In addition, when a certain data in the standard data is "empty", "null" or does not exist, the data is also abnormal data.
[0084] It should be noted that the data item in the red ticket includes the "original ticket ID", while the blue ticket in the invoice does not include the "original ticket ID". When the data type of the standard data is invoice, the embodiment of the present application can determine whether the data type of the standard data is a red ticket or a blue ticket by detecting whether the data item includes the "original ticket ID".
[0085] Cross-checking is a common relationship in fiscal and tax data. It means that there is a relationship between related numbers that can be used to check and verify each other. For example, the summary amount is equal to the sum of the related detailed amounts, and the total sales of a product is equal to the product of the number of units sold and the unit price of the product.
[0086] In the embodiment of the present application, a correction rule library can be set up, in which various digital data with cross-reference relationship in the data to be entered or the standard data, and the cross-reference relationship between various digital data are pre-stored. After detecting abnormal data, it can be determined whether the correction rule library includes the abnormal data. If the abnormal data is not included, it can be determined that the abnormal data cannot be corrected; if the abnormal data is included, it can be determined whether the data with cross-reference relationship is normal data. If the data with cross-reference relationship is normal data, it can be determined that the abnormal data can be corrected. If the data with cross-reference relationship is abnormal data, it can be determined that the abnormal data cannot be corrected.
[0087] S142: If correction is possible, the abnormal data is corrected according to the cross-reference relationship.
[0088] For example, if the data type of a certain item of standard data is an invoice, and specifically a red ticket, but the amount therein is a positive value, and it is determined that the amount can be corrected, then the amount can be corrected to a negative value.
[0089] If the predetermined format of the amount data such as the detailed amount and the summary amount is Arabic numerals, but the summary amount in a certain standard data is detected as Chinese characters, and it is determined that the summary amount can be corrected, the detailed amounts summarized as the summary amount can be summed, and the summary amount can be corrected to the Arabic numerals of the summation result. If it is detected that there is a detailed amount in a certain standard data as Chinese characters, and it is determined that the detailed amount can be corrected, the summary amount summarized by the detailed rule and the detailed rules summarized together with the detailed rule as the summary amount can be subtracted, and the detailed amount can be corrected to the Arabic numerals of the subtraction result. It should be understood that when it is detected that there are multiple detailed amounts in a certain standard data as Chinese characters, it can be determined that the multiple detailed amounts in Chinese characters cannot be corrected.
[0090] If the time length of the period of a certain standard data should be a whole month, but it is detected that the time length of the period in the standard data is less than one month or exceeds one month, and it is determined that the period can be corrected, the time length of the period can be adjusted to a whole month.
[0091] S143: If correction is not possible, the abnormal data is discarded.
[0092] As a feasible implementation manner, only the abnormal data may be discarded, or the standard data to which the abnormal data belongs may be directly discarded.
[0093] In the example of the present application, it is possible to detect whether there is abnormal data in the standard data, and when there is abnormal data, it is determined whether the abnormal data can be corrected based on the cross-reference relationship. When the abnormal data can be corrected based on the cross-reference relationship, the abnormal data can be corrected based on the cross-reference relationship. When the abnormal data cannot be corrected based on the cross-reference relationship, the abnormal data is discarded to complete the processing of the abnormal data, thereby reducing invalid data and erroneous data in the standard data.
[0094] In some optional embodiments, the method provided by the present application further includes: reacquiring the discarded abnormal data. The discarded abnormal data can be directly acquired based on the data to be entered corresponding to the standard data to which the discarded abnormal data belongs, or the data to be entered corresponding to the standard data to which the discarded abnormal data belongs can be reacquired, and the discarded abnormal data can be acquired based on the data to be entered. The specific acquisition process can refer to the above embodiment, which will not be described in detail here. In the embodiment of the present application, after the abnormal data is discarded, the abnormal data can be reacquired to ensure the integrity of the stored standard data as much as possible.
[0095] In some optional embodiments, the method provided by the present application further includes: determining a data set corresponding to the data item from multiple data sets according to the data item in the verified standard data, the data set corresponding to the data item being a data set used to store the data corresponding to the data item. The data corresponding to the data item in the verified standard data is stored in the data set corresponding to the data item.
[0096] In the embodiment of the present application, a correspondence between each data item and a data set can be established, so that each data item and the data corresponding to each data item are fixedly stored in one or more corresponding data sets. In addition, the correspondence between each data item and the data set can be stored.
[0097] According to the data items in the verified standard data, the data set corresponding to the data item can be searched from the correspondence between each stored data item and the data set, thereby determining the data set corresponding to the data item.
[0098] As a feasible implementation method, the data set may include a detailed data set, an indicator data set, and a summary data set. The detailed data set corresponds to detailed data items such as detailed amount and invoice details, and is used to store data corresponding to data items such as detailed amount and invoice details. The indicator data set corresponds to indicator data items such as average value and growth rate, and is used to store data corresponding to data items such as average value and growth rate. The summary data set corresponds to summary data items such as summary amount and total sales, and is used to store data corresponding to data items such as summary amount and total sales.
[0099] It should be noted that the present application may set up multiple databases, such as an original database, a unified database, and a data set database, etc. Optionally, the acquired data set to be entered may be stored in the original database. Depending on the data type of the standard data, the verified standard data may be stored in the unified database. Each data set may be stored in the data set database.
[0100] In the embodiments of the present application, multiple data sets can be established to store various data item data respectively according to different application scenarios, and the data set corresponding to the data item can be determined from the multiple data sets based on the data item in the verified standard data, and the data corresponding to the data item in the verified standard data can be stored in the data set corresponding to the data item, which can facilitate unified management of the data corresponding to each data item, so that the user can quickly find all the data corresponding to a data item from the data set corresponding to the data item.
[0101] like Figure 4 As shown, the embodiment of the present application also provides a data management system 400, including: a data acquisition module 410, a data fusion module and a data storage module 450, and the data fusion module may include a deduplication and merging module 420, a standardization module 430 and a verification module 440.
[0102] The data acquisition module 410 is used to acquire a set of data to be entered.
[0103] The de-duplication and merging module 420 is used to perform de-duplication and merging processing on the data to be entered included in the data set to be entered to obtain pre-processed data.
[0104] The standardization module 430 is used to adjust the format of the data in the pre-processed data to a predetermined format to obtain standard data.
[0105] The verification module 440 is used to verify the standard data.
[0106] The storage module is used to store the verified standard data according to the data type of the standard data.
[0107] It should be noted that the data management system provided in the embodiment of the present application and the embodiment of the aforementioned data management method are based on the same inventive concept and can achieve the same effect. The specific implementation process can be found in the description in the embodiment of the aforementioned data management method, which will not be repeated here.
[0108] The present application also provides an electronic device. Figure 5 , shows a schematic diagram of the structure of an electronic device according to an embodiment of the present application. The specific embodiment of the present application does not limit the specific implementation of the electronic device.
[0109] like Figure 5 As shown, the electronic device may include: a processor (processor) 502 , a communication interface (Communications Interface) 504 , a memory (memory) 506 , and a communication bus 508 .
[0110] in:
[0111] The processor 502 , the communication interface 504 , and the memory 506 communicate with each other via a communication bus 508 .
[0112] The communication interface 504 is used to communicate with other electronic devices or servers.
[0113] The processor 502 is used to execute at least one computer software 510, and specifically can execute the relevant steps in the above data management method embodiment.
[0114] The processor 502 may be a CPU, or an application specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.
[0115] The memory 506 is used to store at least one computer software 510. The memory 506 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory.
[0116] The computer software 510 may include multiple computer instructions. The computer software 510 may specifically enable the processor 502 to perform operations corresponding to the data management method described in any of the aforementioned multiple method embodiments through multiple computer instructions. The specific implementation of each step in the computer software 510 can refer to the corresponding descriptions in the corresponding steps and units in the aforementioned method embodiments, and has corresponding beneficial effects, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the aforementioned method embodiments, which will not be repeated here.
[0117] The present application also provides a computer storage medium on which a computer program is stored, and when the program is executed by a processor, the method described in any of the above-mentioned multiple method embodiments is implemented. The computer storage medium includes but is not limited to: a compact disc read-only memory (CD-ROM), a random access memory (RAM), a floppy disk, a hard disk or a magneto-optical disk, etc.
[0118] An embodiment of the present application also provides a computer program product, including computer instructions, which instruct a computing device to execute operations corresponding to the method described in any one of the aforementioned multiple method embodiments.
[0119] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.
[0120] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present application.
[0121] The above implementation methods are only used to illustrate the embodiments of the present application, and are not limitations on the embodiments of the present application. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, all equivalent technical solutions also belong to the scope of the embodiments of the present application. The scope of patent protection of the embodiments of the present application should be limited by the claims.
Claims
1. A data management method, characterized in that: include: Get the data set to be entered; Performing de-duplication and merging processing on the data to be entered included in the data set to be entered to obtain pre-processed data; Adjusting the format of the data in the preprocessed data to a predetermined format to obtain standard data; The standard data is checked, and according to the data type of the standard data, the checked standard data is stored.
2. The data management method according to claim 1, characterized in that: The de-duplication and merging process of the data to be entered included in the data set to be entered includes: According to the generation frequency of the data to be input, determining a target rule corresponding to the generation frequency from a plurality of de-duplication merging rules, wherein the de-duplication merging rule is related to the generation frequency; Based on the deduplication and merging rules, the data to be input is deduplicated and merged.
3. The data management method according to claim 2, characterized in that: The types of generation frequency include at least one of the following: one-time generation, generation at a periodic frequency, generation at an indefinite frequency, and non-definite generation.
4. The data management method according to claim 3, characterized in that: The step of adjusting the format of the data in the preprocessed data to a predetermined format includes: If the data type of the pre-processed data is a financial statement, the accounting standard type of the pre-processed data is determined, and the determined accounting standard type is added to the pre-processed data.
5. The data management method according to claim 4, characterized in that: The checking of the standard data comprises: Detecting whether there is abnormal data in the standard data, and if there is abnormal data, determining whether the abnormal data can be corrected according to the cross-reference relationship; If correction is possible, the abnormal data is corrected according to the cross-reference relationship; If correction is not possible, the abnormal data is discarded.
6. The data management method according to any one of claims 1 to 5, characterized in that: The step of obtaining a set of data to be entered includes: Obtaining original data of the data to be entered; Determining a data extraction rule according to the data source of the original data; the data extraction rule is used to indicate the target data to be extracted from the original data; According to the data extraction rule, the target data is extracted from the original data to obtain the data to be entered, and the data set to be entered is obtained based on the data to be entered.
7. The data management method according to any one of claims 1 to 5, characterized in that: The method further comprises: According to the data item in the verified standard data, determining a data set corresponding to the data item from multiple data sets, the data set corresponding to the data item being a data set used to store the data corresponding to the data item; The data corresponding to the data item in the verified standard data is stored in the data set corresponding to the data item.
8. A data management system, characterized in that: include: A data acquisition module is used to acquire a set of data to be entered; A de-duplication and merging module, used for performing de-duplication and merging processing on the data to be entered included in the data set to be entered to obtain pre-processed data; A standardization module, used for adjusting the format of the data in the preprocessed data to a predetermined format to obtain standard data; A verification module, used for verifying the standard data; The storage module is used to store the verified standard data according to the data type of the standard data.
9. An electronic device, comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one computer software, and the computer software enables the processor to execute the method according to any one of claims 1-7.
10. A computer storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.