A data management method and device

Abstract

The application provides a data management method, comprising: acquiring heterogeneous source data; the heterogeneous source data comprises basic information and power consumption data of an enterprise; performing data adaptation on the heterogeneous source data to obtain first data; performing standardization on the first data to obtain second data; performing quality evaluation on the second data; and generating a quality report according to the result of the quality evaluation. By adapting and standardizing the data, the type of data that can be analyzed is increased, the adaptability of analysis is expanded, the heterogeneous source data is comprehensively analyzed, and the efficiency of data processing is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, in particular to a data management method and device. BACKGROUND

[0002] In recent years, with the power system entering the financial industry, realizing the combination of production and finance, the market competitiveness of financial units is gradually improved, and the ability and profit contribution of financial capital serving the main business are continuously enhanced. In various reporting systems related to power financial institutions, various business data of reporting systems need to be collected to provide rich analysis basis for financial decision-making.

[0003] Because different systems use different technologies and data specifications, different processing is needed for different data. On the one hand, it is difficult to summarize all data, and on the other hand, too much data processing will lead to low data processing efficiency, resulting in a substantial decline in real-time data processing.

[0004] Therefore, how to improve the comprehensiveness of data recognition while improving data processing efficiency is a technical problem that technicians in the field urgently need to solve. SUMMARY

[0005] Therefore, the embodiments of the present application provide a data management method and device, aiming to improve the comprehensiveness of data recognition while improving data processing efficiency.

[0006] In a first aspect, the embodiments of the present application provide a data management method, comprising:

[0007] acquiring heterogeneous source data; the heterogeneous source data includes basic information and power consumption data of an enterprise;

[0008] performing data adaptation on the heterogeneous source data to obtain first data;

[0009] performing standardization on the first data to obtain second data;

[0010] performing quality evaluation on the second data;

[0011] generating a quality report according to the result of the quality evaluation.

[0012] Preferably, after the quality evaluation on the second data, the method further comprises:

[0013] determining whether the result of the quality evaluation meets a standard;

[0014] If not, returning the heterogeneous source data.

[0015] Preferably, after the quality evaluation on the second data, the method further comprises:

[0016] determine whether the result of the quality evaluation meets a criterion;

[0017] if yes, write the second data into a database;

[0018] perform financial business analysis on the second data.

[0019] Preferably, the database comprises a temporary database and an official database.

[0020] Preferably, the basic information and power consumption data of the enterprise comprise financial, production, and logistics information of the enterprise, and power consumption and electricity fee records of the enterprise.

[0021] In a second aspect, an embodiment of the present application provides a data management apparatus, comprising:

[0022] an acquisition module configured to acquire heterogeneous source data; the heterogeneous source data comprising basic information and power consumption data of an enterprise;

[0023] an adaptation module configured to perform data adaptation on the heterogeneous source data to obtain first data;

[0024] a standardization module configured to perform standardization on the first data to obtain second data;

[0025] a quality evaluation module configured to perform quality evaluation on the second data;

[0026] a report generation module configured to generate a quality report according to the result of the quality evaluation.

[0027] Preferably, the apparatus further comprises:

[0028] a determination module configured to determine whether the result of the quality evaluation meets a criterion;

[0029] a return module configured to return the heterogeneous source data if the result of the quality evaluation does not meet the criterion.

[0030] Preferably, the apparatus further comprises:

[0031] a determination module configured to determine whether the result of the quality evaluation meets a criterion;

[0032] a storage module configured to write the second data into a database if the result of the quality evaluation meets the criterion;

[0033] an analysis module configured to perform financial business analysis on the second data.

[0034] In a third aspect, an embodiment of the present application provides an apparatus, comprising a memory and a processor, wherein the memory is configured to store instructions or codes, and the processor is configured to execute the instructions or codes to enable the apparatus to perform the data management method of any one of the preceding first aspect.

[0035] In a fourth aspect, the embodiments of the present application provide a computer storage medium, wherein a code is stored in the computer storage medium, and when the code is executed, an apparatus executing the code implements the data management method according to any one of the first aspect.

[0036] The embodiments of the present application provide a data management method. In the execution of the method, first, heterogeneous source data is acquired, then the heterogeneous source data is data adapted to obtain first data, the first data is then standardized to obtain second data, the second data is then quality evaluated, and finally, a quality report is generated according to the result of the quality evaluation. In this way, data can be collected by using an efficient and widely compatible method, the adaptability of analysis is expanded, heterogeneous source data is comprehensively analyzed, various heterogeneous data is compatible through data adaptation, and a general data specification is quickly standardized, which speeds up data management, solves data quality problems, and the quality of data can be directly reflected by using a report. The type of data that can be analyzed is increased, and the efficiency of data processing is improved. BRIEF DESCRIPTION OF DRAWINGS

[0037] To make the technical solutions in the embodiments or the prior art clearer, the accompanying drawings needed in the embodiments or the prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description are only some embodiments of the present application, and other accompanying drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0038] Figure 1 A method flowchart of the data management method provided by the embodiments of the present application is provided.

[0039] Figure 2 Another method flowchart of the data management method provided by the embodiments of the present application is provided.

[0040] Figure 3 A structural schematic diagram of the data management device provided by the embodiments of the present application is provided. DETAILED DESCRIPTION

[0041] In recent years, with the continuous development of the power system and the financial industry, the concept of power system entering the financial industry and realizing the combination of production and finance is proposed. Through the integration of the two, the market competitiveness of financial units can be improved, and the ability and profit contribution of financial capital serving the main business can be enhanced. In various reporting systems related to power financial institutions, business data of various reporting systems need to be collected to provide rich analysis basis for financial decision-making. Because different systems use different technologies and data specifications, different processing is needed for different data. On the one hand, it is difficult to summarize all the data, and on the other hand, too much data processing will lead to low data processing efficiency, resulting in a significant decline in the real-time performance of data processing.

[0042] The method provided by the embodiment of the application is executed by a computer device, and is used for managing data.

[0043] Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor belong to the scope of protection of the present application.

[0044] Referring to Figure 1 , Figure 1 A method flowchart of the data management method provided by the embodiment of the present application comprises:

[0045] Step S101: acquiring heterogeneous source data.

[0046] The heterogeneous source data is a multi-source heterogeneous data, wherein the multi-source refers to multiple data holders, and the heterogeneous refers to the inconsistency of the types and characteristics of the data. Specifically, the multiple data holders include multiple enterprises; the heterogeneous data is because the acquired data includes enterprise basic information and power consumption data, and the enterprise basic information and power consumption data include enterprise financial, production, and logistics information and enterprise power consumption and electricity fee records, and the types and characteristics corresponding to different data are mostly different, so the types and characteristics are inconsistent.

[0047] Step S102: data adaptation is performed on the heterogeneous source data to obtain first data.

[0048] Data adaptation can be performed by a data adapter. The main purpose of the data adapter is to communicate data between the data source and the data set, and exchange data between the data source and the data set. In many application programs, this means reading data from a database into a data set, and then writing changed data from the data set back to the database. By configuring the adapter, various data can be changed to a data type that can be analyzed.

[0049] Step S103: standardization is performed on the first data to obtain second data.

[0050] The standardization refers to implementing standards, that is, changing the first data into a data format under a unified data standard, so that the data specification conforms to the standards through the standardization processing.

[0051] Step S104: quality evaluation is performed on the second data.

[0052] The quality evaluation is mainly for the integrity, uniqueness, consistency, accuracy, validity and timeliness of the second data.

[0053] The integrity refers to whether each item of the data is recorded and there is no missing; the uniqueness, also known as uniqueness, refers to whether the data is repeated and each piece of data is unique; the consistency refers to whether the data is consistent before and after processing, which can be detected by the data volume of the source table and the data volume of the target table, or by specific field matching detection; the accuracy refers to whether the data can reflect the content of the business, which can be detected by the sum, average, maximum and minimum of some fields of the data; the validity refers to whether the data conforms to the rules, such as whether the data of the date field conforms to the unified date format, whether the numerical range of the amount field is reasonable, whether the data of the ID field is in compliance, etc.; the timeliness refers to which time period the data reflects, which can be detected by the update frequency and time difference of the data, or by setting a timeout alarm for the key task to monitor.

[0054] Step S105: generating a quality report according to the result of the quality evaluation.

[0055] The quality report can record the data quality of the heterogeneous source data, facilitate timely viewing of data information in subsequent use, and facilitate the technical personnel to further adjust according to the data quality.

[0056] In summary, the embodiment can collect data by using an efficient and widely compatible method, expand the adaptability of analysis, comprehensively analyze heterogeneous source data, adapt various heterogeneous data through data adaptation, and quickly standardize the general data specification, thereby increasing the data management speed, solving the data quality problem, using the report mode to intuitively reflect the data quality situation, increasing the types of data that can be analyzed, and improving the efficiency of data processing.

[0057] In the embodiment of the present application, the above Figure 1 The steps can have various possible implementation manners, which will be introduced below. It should be noted that the implementation manners given in the following introduction are only exemplary and do not represent all implementation manners of the embodiment of the present application.

[0058] Referring to Figure 2FIG. 2 is another method flow diagram of the data management method provided by the present embodiment.

[0059] Step S201: Obtain heterogeneous source data.

[0060] The heterogeneous source data is multi-source heterogeneous data, wherein the multi-source refers to multiple data holders, and the heterogeneous refers to inconsistency in types and characteristics of the data. Specifically, the multiple data holders include multiple enterprises; the heterogeneous data is because the obtained data includes enterprise basic information and power consumption data, and the enterprise basic information and power consumption data includes enterprise financial, production, and logistics information and enterprise power consumption and electricity charge records, and the types and characteristics corresponding to different data are mostly different, thus leading to inconsistency in types and characteristics.

[0061] Step S202: Data adaptation is performed on the heterogeneous source data to obtain first data.

[0062] The data adaptation can be performed by a data adapter. The main purpose of the data adapter is to perform data communication between a data source and a data set, and exchange data between the data source and the data set. In many application programs, this means reading data from a database into a data set, and then writing changed data from the data set back to the database. By configuring the adapter, various types of data can be changed into an analyzable data type.

[0063] Step S203: Standardization is performed on the first data to obtain second data.

[0064] The standardization refers to implementing a standard, i.e., changing the first data into a data format under a unified data standard. Through the standardization processing, the data specification can conform to the standard.

[0065] Step S204: Quality evaluation is performed on the second data.

[0066] The quality evaluation is mainly for the completeness, uniqueness, consistency, accuracy, effectiveness, and timeliness of the second data.

[0067] The integrity refers to whether each item of the evaluation data is recorded without missing; the uniqueness, also referred to as the distinctness, refers to whether the evaluation data is repeated, and whether each piece of data is unique; the consistency refers to whether the evaluation data is consistent before and after processing, which can be detected by the data volume of the source table and the data volume of the target table, or by specific field matching detection; the accuracy refers to whether the data can reflect the content of the business, which can be detected by the sum, average, maximum and minimum of some fields of the data; the validity refers to whether the data meets the rules, such as whether the data of the date field meets the unified date format, whether the numerical range of the amount field is reasonable, and whether the data of the ID field is in compliance, etc.; and the timeliness refers to which time period the data reflects, which can be detected by the update frequency and time difference of the data, or by setting a timeout alarm for a key task to monitor.

[0068] Step S205: judging whether the result of the quality evaluation meets the standard.

[0069] A certain standard can be set in advance to judge the result of the quality evaluation. For example, if the second data has five of the six aspects of integrity, uniqueness, consistency, accuracy, validity and timeliness, it can be considered that the result of the quality evaluation meets the standard. If yes, go to step S206; if no, go to step S208.

[0070] Step S206: if yes, write the second data into the database.

[0071] If the result of the quality evaluation meets the standard, the second data is saved in two copies, one of which is stored in the temporary database, and the other of which is stored in the formal database. The temporary database can be a 7-day / 1-month / 3-month backup database. Writing the second data into the database can provide business support for other businesses, facilitate the analysis of financial businesses, help enterprises to conduct financial credit and financial investment, and solve the problem of financing difficulty on the supply chain of small and medium-sized enterprises.

[0072] Step S207: performing financial business analysis on the second data.

[0073] As a possible implementation, the financial business analysis on the second data can include financial customer portrait, financial product recommendation, investment accounting, etc. The financial customer portrait includes basic enterprise portrait, financial product transaction portrait, customer product preference portrait, customer risk portrait, customer supply chain portrait, etc.; the financial product recommendation includes financial product search, financial product analysis and comparison, financial product review; the investment accounting includes transaction display and transaction query, the transaction display is used to display the transaction process between the enterprise and the fund party, and the transaction query is used to query the financial service transaction data.

[0074] Specifically, the basic enterprise portrait is used to provide basic portrait information of a financial customer, the financial product transaction portrait is used to provide transaction portrait information of the financial customer, the customer product preference portrait is used to provide customer financial product preference portrait information, the customer risk portrait is used to provide customer financial risk information, and the customer supply chain portrait is used to show customer related industry supply chain portrait information. The financial product search is used for a user to search for various financial product information, the financial product analysis and comparison is used for the user to analyze and compare various financial products according to various financial product information, and the financial product comment is used to input and display comment information of various financial products.

[0075] Step S208: returning the heterogeneous source end data.

[0076] If the result of the quality evaluation does not meet the standard, the heterogeneous source end data needs to be returned to the enterprise, and after the enterprise receives the returned data, the data can be adjusted and re-uploaded. In this way, the data can be returned in time during the data quality evaluation, and re-executed, avoiding the time consumption of massive data into the warehouse, causing the increase of work cost and storage cost, increasing the correction speed, and solving the problem of data quality.

[0077] Step S209: generating a quality report according to the result of the quality evaluation.

[0078] The quality report can record the data quality of the heterogeneous source end data, facilitate timely viewing of data information in subsequent use, and facilitate technicians to further adjust according to the quality of the data.

[0079] In summary, the embodiment can collect data by using an efficient and widely compatible method, adapt various heterogeneous data through data adaptation, and quickly convert to a general data specification, thereby increasing the speed of data management, solving the problem of data quality, discovering unqualified data and returning it in time, avoiding the time consumption of massive data into the warehouse, causing high work cost and storage cost, and directly discovering the quality of the data through the report. At the same time, it is convenient for financial analysis and generates additional financial value.

[0080] The above is some specific implementation manners of the data management method provided by the embodiment of the application, and based on this, the application also provides a corresponding device. The device provided by the embodiment of the application will be introduced from the perspective of functional modularization.

[0081] Referring to Figure 3 The structure schematic diagram of the data management device 300 is shown in the figure, and the device 300 includes an acquisition module 301, an adaptation module 302, a standard penetration module 303, a quality evaluation module 304, and a report generation module 305.

[0082] An acquisition module is configured to acquire heterogeneous source data; the heterogeneous source data comprises basic information and power consumption data of an enterprise;

[0083] An adaptation module is configured to perform data adaptation on the heterogeneous source data to obtain first data.

[0084] A standardization module is configured to perform standardization on the first data to obtain second data.

[0085] A quality evaluation module is configured to perform quality evaluation on the second data.

[0086] A report generation module is configured to generate a quality report according to a result of the quality evaluation.

[0087] A judgment module is configured to judge whether the result of the quality evaluation meets a standard.

[0088] A return module is configured to return the heterogeneous source data if the result of the quality evaluation does not meet the standard.

[0089] A storage module is configured to write the second data into a database if the result of the quality evaluation meets the standard.

[0090] An analysis module is configured to perform financial business analysis on the second data.

[0091] The database comprises a temporary database and an official database.

[0092] The basic information and power consumption data of the enterprise comprise financial, production and logistics information of the enterprise and power consumption and electricity charge records of the enterprise.

[0093] Embodiments of the present application further provide corresponding devices and computer storage media for implementing the schemes provided by the embodiments of the present application.

[0094] The device comprises a memory and a processor, the memory is configured to store instructions or codes, and the processor is configured to execute the instructions or codes to enable the device to perform the method for improving first screen performance according to any of the embodiments of the present application.

[0095] In practical application, the computer storage medium can adopt any combination of one or more computer media. The computer medium can be a computer signal medium or a computer storage medium. The computer storage medium may, for example, but is not limited to, an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (non-exhaustive list) of the computer storage medium include: an electric connection with one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present embodiment, the computer storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or component.

[0096] The computer signal medium can include a data signal propagated in a baseband or as a part of a carrier wave, in which a computer program code is borne. Such a propagated data signal can take on many forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. The computer signal medium can also be any computer medium that can send, propagate or transport a program for use by or in connection with an instruction execution system, device or component.

[0097] The program code contained on the computer medium can be transmitted in any suitable medium, including but not limited to wireless, wire line, optical cable, RF, etc., or any suitable combination thereof.

[0098] The computer program code for carrying out operations of the present application can be written in one or more programming languages or combinations of languages including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages such as "C" or the like. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0099] The data management method provided by the application can be used in the financial field or other fields, for example, can be used in the application scenario of APP payment verification in the financial field. The other fields are any fields except the financial field, for example, the field of data processing. The above are only examples, and do not limit the application field of the application provided by the application.

[0100] It should also be noted that the relational terms herein, such as first and second, are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any such actual relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements recited, but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without more limitations, an element defined by the statement "comprising a" does not exclude the existence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0101] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, it is described more simply, and the relevant part can be referred to the part of the method embodiment. According to the actual needs, some or all of the modules can be selected to achieve the purpose of the embodiment. Those skilled in the art can understand and implement it without creative labor.

[0102] The above is only one specific embodiment of the present application, but the protection scope of the present application is not limited to this. Any skilled person in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A data management method, characterized in that, The method includes: Acquire heterogeneous source data; the heterogeneous source data includes basic information about the enterprise and electricity consumption data. By configuring a data adapter, the heterogeneous source data is adapted to obtain the first data. The data adapter is used to change various types of data into analyzable data types. The first data is standardized and converted into a data format under a unified data standard to obtain the second data. The second data is evaluated for quality across six dimensions: completeness, uniqueness, consistency, accuracy, validity, and timeliness. Consistency refers to whether the data remains consistent before and after processing, detected by comparing the data volume of the source and target tables, or by matching specific fields. Timeliness refers to the time period the data reflects, including testing the data update frequency and time difference, or monitoring by setting timeout alarms for critical tasks. Accuracy refers to whether the data reflects the business content, including testing the sum, average, maximum, and minimum values ​​of certain fields to ensure they match the business situation. Validity assesses whether the data conforms to rules, including whether date fields conform to a unified date format, whether the range of monetary values ​​is reasonable, and whether ID card field data is compliant. If the second data meets five of the six aspects of completeness, uniqueness, consistency, accuracy, validity, and timeliness, the quality assessment result is considered to meet the standard. Two copies of the second data are saved: one in a temporary database and one in a formal database. The temporary database is a backup database for a set period. If the quality assessment results do not meet the standards, the heterogeneous source data will be returned to the relevant enterprise so that the enterprise can adjust the data and re-upload it after receiving the returned data. Based on the results of the quality assessment, a quality report is generated, which is used to support subsequent corporate financial customer profiling, financial product recommendations, and investment accounting analysis.

2. The method according to claim 1, characterized in that, Following the quality assessment of the second data, the following is also included: Determine whether the results of the quality assessment meet the standards; If not, return the heterogeneous source data.

3. The method according to claim 1, characterized in that, Following the quality assessment of the second data, the following is also included: Determine whether the results of the quality assessment meet the standards; If so, write the second data into the database; The second data is used for financial business analysis.

4. The method according to claim 3, characterized in that, The database includes a temporary database and a formal database.

5. The method according to claim 1, characterized in that, The company's basic information and electricity consumption data include the company's financial, production, and logistics information, as well as the company's electricity consumption and electricity bill records.

6. A data management device, characterized in that, The device includes: The acquisition module is used to acquire heterogeneous source data; the heterogeneous source data includes the enterprise's basic information and electricity consumption data. An adaptation module is used to adapt the heterogeneous source data by configuring a data adapter to obtain first data. The data adapter is used to change various types of data into analyzable data types. The standardization module is used to standardize the first data, changing the first data into a data format under a unified data standard to obtain the second data. The quality assessment module is used to assess the quality of the second data from six dimensions: completeness, uniqueness, consistency, accuracy, validity, and timeliness. Consistency refers to whether the data is consistent before and after processing, detected by comparing the data volume of the source and target tables, or by matching specific fields. Timeliness refers to the time period reflected by the data, including detecting the data update frequency and time difference, or monitoring by setting timeout alarms for critical tasks. Accuracy refers to whether the data reflects the business content, including detecting whether the sum, average, maximum, and minimum values ​​of certain fields match the business situation. The validity assessment evaluates whether the data conforms to the rules, including whether the date field conforms to the unified date format, whether the value range of the amount field is reasonable, and whether the ID card field data is compliant. If the second data meets five of the six aspects of completeness, uniqueness, consistency, accuracy, validity, and timeliness, the quality assessment result is considered to meet the standard. Two copies of the second data are saved, one in a temporary database and one in the official database. The temporary database serves as a backup database for a set period. If the quality assessment result does not meet the standard, the heterogeneous source data is returned to the enterprise so that the enterprise can adjust the data and re-upload it after receiving the returned data. The report generation module is used to generate a quality report based on the results of the quality assessment. The quality report is used to support subsequent corporate financial customer profiling, financial product recommendations, and investment accounting analysis.

7. The apparatus according to claim 6, characterized in that, The device further includes: The judgment module is used to determine whether the results of the quality assessment meet the standards. The return module is used to return the heterogeneous source data if no.

8. The apparatus according to claim 6, characterized in that, The device further includes: The judgment module is used to determine whether the results of the quality assessment meet the standards. The data entry module is used to write the second data into the database if the condition is met. The analysis module is used to perform financial business analysis on the second data.

9. A device, characterized in that, The device includes a memory and a processor, the memory being used to store instructions or code, and the processor being used to execute the instructions or code to cause the device to perform the data management method according to any one of claims 1 to 5.

10. A computer storage medium, characterized in that, The computer storage medium stores code for executing the data management method according to any one of claims 1 to 5.

Images