Data access method, system, device and computer readable medium

Abstract

Disclosed are a data access method and a data access system, wherein the data access method comprises: before accessing the data to be accessed, performing multi-level verification on the data to be accessed, wherein the multi-level verification comprises: judging whether the data to be accessed is duplicate data according to historical access records; judging whether the data to be accessed is duplicate data according to master-slave relationship information; judging whether the data to be accessed is duplicate data according to metadata. The present invention can avoid duplicate access of data and reduce waste of resources.

Description

Technical Field

[0001] The present invention relates to the field of computer and software technology, and in particular to a data access method, system, device and computer-readable medium. Background Art

[0002] In the process of building a data warehouse or migrating data from different information systems, a large amount of data needs to be imported from various data sources. There are data quality problems such as data entry errors or the same object being represented in different forms in different data sources, which will affect the quality of information services. Therefore, an important task is to convert the data into a consistent form through data cleansing to ensure the correctness of the data. Data cleansing mainly involves data mapping, data matching and data merging / purging. Through mapping, the data format is standardized; through matching, duplicate records are found; through merging, a complete record is retained or generated. The core of data cleaning activities is the identification of near-duplicate data. The so-called near-duplicate data refers to data with different forms but the same semantics. From a narrow perspective, if the values ​​of two records in certain fields are equal or similar enough, the two records are considered to be near-duplicates of each other, which we call duplicate records or duplicate data. In order to improve data quality, it is necessary to find and delete duplicate data. This is also called data duplication checking or data deduplication.

[0003] Current Internet systems are faced with the problem of duplicate data access. For the same type of business, such as orders, coupons, etc., online production data is usually a set, that is, the same department produces data, and different business departments use the data to calculate the indicators they are concerned about. When different business parties are preparing to use data, they do not care whether the platform already has data. Usually, according to business needs, they directly choose to access the data through the data access platform to the data warehouse or Kafka.

[0004] It can be seen that in the existing technical solutions, there is no systematic judgment on duplicate access to online production data. Depending on the business situation, the impact of duplicate data may be large or small. Duplicate data access usually brings the following problems:

[0005] 1) Due to reliance on manual approval and identification, data is repeatedly accessed to the data warehouse or Kafka, wasting a lot of storage space;

[0006] 2) When performing big data analysis, multiple tasks extract and process multiple copies of duplicate data, resulting in a waste of cluster resources;

[0007] 3) Repeated access to data will also cause certain reading pressure on the online system. Summary of the invention

[0008] In view of the above problems, the purpose of the present invention is to provide a data access method, which performs multi-level verification before data access, thereby reducing the probability of repeated data access, reducing the waste of storage resources and computing resources, and improving data reading efficiency.

[0009] To achieve the above object, according to a first aspect of an embodiment of the present invention, a data access method is provided, wherein before accessing the data to be accessed, a multi-level check is performed on the data to be accessed, and the multi-level check includes:

[0010] Determining whether the data to be accessed is duplicate data according to historical access records;

[0011] Determining whether the data to be accessed is duplicate data according to the master-slave relationship information;

[0012] It is determined whether the data to be accessed is duplicate data according to the metadata.

[0013] Optionally, in the multi-level verification step, the judgment in the next step is performed only when the previous step judges that the data is not duplicate data.

[0014] Optionally, the method further includes: acquiring configuration information of the data to be accessed.

[0015] Optionally, if the data to be accessed is library data, the configuration information includes the IP or host name of the data to be accessed; if the data to be accessed is table data, the configuration information includes the IP or host name of the data to be accessed and the table name.

[0016] Optionally, judging whether the data to be accessed is duplicate data according to the master-slave structure relationship includes:

[0017] Get master-slave relationship information;

[0018] Retrieving the historical access record according to the master-slave relationship information and the configuration information; and

[0019] Determine whether the data is duplicate based on the search results.

[0020] Optionally, obtain the master-slave relationship information according to the calling interface or command line method.

[0021] Optionally, the method further includes: acquiring configuration information of the data to be accessed through a user interface.

[0022] Optionally, judging whether the data to be accessed is duplicate data according to the metadata includes:

[0023] Acquire the prefix name of the accessed data according to the metadata; and

[0024] Generate an alias for the data to be accessed according to the prefix name.

[0025] According to a second aspect of an embodiment of the present invention, there is provided a data access system, including: a verification module, the verification module performing multi-level verification on the data to be accessed, the verification module including:

[0026] A first verification unit, used to determine whether the data to be accessed is duplicate data according to historical access records;

[0027] A second checking unit, used for judging whether the data to be accessed is duplicate data according to the master-slave relationship information;

[0028] The third verification unit is used to determine whether the data to be accessed is duplicate data according to the metadata.

[0029] Optionally, the second verification unit is executed after the first verification unit verifies the data to be accessed and passes the verification, and the third verification unit is executed after the second verification unit verifies the data to be accessed and passes the verification.

[0030] According to a third aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed, any of the above-mentioned data access methods is implemented.

[0031] According to a fourth aspect of an embodiment of the present invention, a data access device is provided, including:

[0032] Memory, for storing computer instructions;

[0033] A processor is coupled to the memory, and the processor is configured to execute any of the above-mentioned data access methods based on computer instructions stored in the memory.

[0034] The embodiments of the present invention have the following advantages or beneficial effects: multi-level verification is performed on the data to be accessed through historical access records, master-slave relationship information and metadata, thereby reducing the probability of repeated data access.

[0035] The preferred embodiment of the present invention has the following advantages or beneficial effects: when multi-level verification is performed on the access data through historical access records, master-slave relationship information and metadata, the verification of the next step is performed only after the verification of the previous step is passed, that is, the verification with a high probability of error is performed first, thereby improving the verification efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0037] Figure 1 A schematic diagram showing a data access system according to an embodiment of the present invention is shown;

[0038] Figure 2 A flow chart showing a data access method according to an embodiment of the present invention is shown;

[0039] Figure 3 A flowchart of obtaining master-slave relationship information according to an embodiment of the present invention is shown;

[0040] Figure 4 is a structural diagram of a data access device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The present invention is described below based on embodiments, but the present invention is not limited to these embodiments. In the detailed description of the present invention below, some specific details are described in detail. It is possible for a person skilled in the art to fully understand the present invention without the description of these details. In order to avoid confusing the essence of the present invention, known methods, processes, and procedures are not described in detail. In addition, the drawings are not necessarily drawn to scale.

[0042] The specific implementation of the present invention is further described in detail below in conjunction with the drawings and examples.

[0043] like Figure 1 As shown, Figure 1 The schematic diagram of the data access system of the embodiment of the present invention is shown. The data access system 10 includes a main control module 200 and a verification module 100. When the main control module 200 accesses the data to be accessed to the data warehouse or Kafka, it calls the verification module 100 to perform multi-level verification, and the data to be accessed that passes the verification is finally accessed to the data warehouse or Kafka.

[0044] A data warehouse is a strategic collection of all types of data to support decision-making processes at all levels of an enterprise. It is a single data store created for analytical reporting and decision support purposes. It provides guidance for business process improvement, monitoring time, cost, quality, and control for enterprises that need business intelligence.

[0045] Kafka is a high-throughput distributed publish-subscribe messaging system that can handle all the action streams in consumer-scale websites. This action (web browsing, searching, and other user actions) is a key factor in many social functions on the modern web. This data is usually solved by processing logs and log aggregation due to throughput requirements. For log data and offline analysis systems like Hadoop, but with the constraints of real-time processing, this is a viable solution. Kafka aims to unify online and offline message processing through Hadoop's parallel loading mechanism, and also to provide real-time messages through clusters.

[0046] The verification module 100 includes a first verification unit 101 , a second verification unit 102 and a third verification unit 103 .

[0047] The first verification unit 101 reads the historical access record 301, and determines whether the data to be accessed is already in the historical access record based on the configuration information of the data to be accessed, which means that the data to be accessed has been accessed, and the verification fails. The configuration information of the data to be accessed can be obtained by entering the configuration file or configuration interface. Generally, the data access system will provide a visual interface, and the user can obtain the configuration information of the data to be accessed through the visual interface. The configuration information of the data to be accessed varies depending on the type of data to be accessed. If the data to be accessed is a database, the configuration information to be obtained includes the IP address or host name of the server where the database to be accessed is located, the port, and the database name. If the data to be accessed is a database table, the configuration information to be obtained includes the IP address or host name of the server where the database of the database table to be accessed is located, the port, the database name, and the database table name.

[0048] The second verification unit 102 obtains the master-slave relationship information 302, and determines whether the data to be accessed is duplicate access data based on the master-slave relationship information 302 and the historical access record 301. In an online database system, a master server and a slave server are generally used to store data, and the master server and the slave server have the same data structure and store the same data. The master-slave relationship will lead to duplicate access to data. For example, the IP address of the master server has been used to access the data, and now it is submitted to access the same data using the IP address of the slave server. The second verification unit 102 determines whether the database of the data to be accessed is a master library or a slave library, that is, if the database of the data to be accessed is a master library, the slave library information of the data to be accessed is obtained, and if the database of the data to be accessed is a slave library, the master library of the data to be accessed is obtained, and then, according to the corresponding master library or slave library and other configuration information, the historical access record 301 is retrieved, and whether there is corresponding duplicate data is determined based on the search results.

[0049] The third verification unit 103 reads metadata 303 and verifies the data to be accessed according to metadata 303. Metadata 303 comes from the metadata management system. Since a new database table name is generated for each database table after the data is accessed to the data warehouse or kafka, in order to prevent the same database table name, the metadata is retrieved according to the new database table name to ensure that there is no duplicate name. In general, the new database table name is a prefix added to the original database table name. For example, if the database table name accessed last time is bdm_library name_table name, then fdm_library name_table name is used this time to avoid duplication. This verification is only a fuzzy verification to avoid creating the same database table name in hive in the future. Hive is a data warehouse tool based on Hadoop, which can map structured data files to a database table, and provide simple sql query functions, which can convert sql statements into MapReduce tasks for running. Its advantage is that the learning cost is low, simple MapReduce statistics can be quickly implemented through SQL-like statements, and there is no need to develop a special MapReduce application, which is very suitable for statistical analysis of data warehouses.

[0050] In summary, the embodiments of the present invention perform multi-level verification on the data to be accessed through historical access records, master-slave relationship information, and metadata, thereby reducing the probability of repeated data access.

[0051] In one embodiment, the second verification unit 102 obtains the master-slave relationship information 302 according to the input configuration information. In the online database system, there are relational databases that are uniformly operated and maintained by the group and relational databases that are not uniformly operated and maintained by the group. The relational database that is uniformly operated and maintained by the group provides interface access, and the interface can be called to obtain the master database information of the IP or domain name filled in by the user. For the relational database that is not uniformly operated and maintained by the group, no interface is provided, and the command line must be used to connect to the database, and the master-slave relationship information of the database must be retrieved in the database. Accordingly, the configuration information needs to indicate that the data to be accessed is a relational database that is uniformly operated and maintained by the group and a relational database that is not uniformly operated and maintained by the group.

[0052] In one embodiment, the second verification unit 102 is executed after the first verification unit 101 verifies the data to be connected, and the third verification unit 103 is executed after the second verification unit 102 verifies the data to be connected. Verification steps with high error probability are executed first, thereby improving verification efficiency.

[0053] It should be noted that the data to be accessed can be offline data or real-time data. The access of offline data requires the database IP or domain name to be filled in, while the access of real-time data, because of the method of obtaining online binlog, can only fill in the database IP. However, the verification processing methods of the two are basically the same.

[0054] Based on the above data access system, the present invention provides a data access method. Figure 2 This is a flow chart of a data access method according to an embodiment of the present invention, which specifically includes the following steps:

[0055] Step S201: The user fills in the configuration information of the data to be accessed on the interface of the data access system;

[0056] Step S202: The configuration information of the data to be accessed filled in by the user is compared with the IP, library, and table in the historical access record;

[0057] Step S203: Determine whether it is a complete match, if yes, execute step S204, if no, execute step S205;

[0058] Step S204: Return to the data access system and display "Already accessed, cannot access again";

[0059] Step S205: Obtain corresponding master-slave relationship information according to the configuration that cannot completely match the IP, library, and table in the historical access record, and identify whether the data to be accessed is the same library or table of the same master-slave relationship;

[0060] Step S206: Determine whether the information is completely matched, if yes, execute step 204, if no, execute step S207;

[0061] Step S207: comparing the data to be accessed that are not in the same library or the same table under the same master-slave relationship with the library or table in the metadata;

[0062] Step S208: Determine whether the information is completely matched, if yes, execute step S204, if no, execute step S209;

[0063] Step S209: The data to be accessed that has passed verification is accessed into the system.

[0064] The comparison with the historical access records in the above steps S202 and S203 is a complete match, which can effectively exclude repeated access of the same IP, library, and table, greatly reducing repeated access of data.

[0065] The above steps S205 and S206 are to prevent users from using different slave or master databases to access the same master-slave relationship, thereby causing duplicate data access. Furthermore, in the above steps S205 and S206, different methods are used to obtain the master-slave relationship information according to the different operation and maintenance groups selected by the user. After obtaining the master-slave relationship information of the database to be accessed, it is compared with the master database of all IPs in the system. If there is no identical master-slave architecture, the library and table are matched. If there is an identical master-slave architecture, it returns that it has been accessed and cannot be accessed repeatedly.

[0066] In the above steps S207 and S208, since the table names generated by the unified process are regular, i.e., bdm_library name_table name, or bdm_library name_table name_da, fdm_library name_table name_chain, or fdm_library name_table name, it is necessary to ensure that the data to be accessed cannot be named with the same prefix. This check is a fuzzy check, the purpose of which is to avoid creating the same table name in Hive later.

[0067] In one embodiment, for the above data access method, when the user fills in the configuration information of the data to be accessed, it is necessary to include various information such as database type, database IP, database name, table name, port number, etc. When filling in, the client can verify the filled in configuration information based on predetermined rules.

[0068] In the above steps S205 and S206, due to different selections of the operation and maintenance group by the user, different ways of obtaining the master database relationship information are formed. Figure 3 The illustrated embodiment will introduce in detail the specific steps of obtaining the master-slave relationship information.

[0069] Figure 3 A flow chart of obtaining master-slave relationship information according to an embodiment of the present invention is shown, including:

[0070] Step S301: Obtain the IP address or domain name in the configuration information;

[0071] Step S302: Determine the operation and maintenance group by IP address. If it is a unified operation and maintenance of the group, execute step S303; if it is not a unified operation and maintenance of the group, execute step S304;

[0072] Step S303: Obtain the main database information of the IP address or domain name filled in by the user through the interface;

[0073] Step S304: After connecting to the database through the user name and password, obtain the master database information of the database through commands;

[0074] Step S305: Identify the main database information in the data source management system;

[0075] Step S306: comparing the master database information identified in step S303 or S304 with the master database information in step S305;

[0076] Step S307: Determine whether there is the same library or table of the same master and slave; if yes, execute step S308; if no, execute step S309;

[0077] Step S308: Return to the data access system and display "Already accessed, cannot access again";

[0078] Step S309: proceed to the next step of judgment.

[0079] In the above steps S303 and S304, different steps are executed according to the operation and maintenance group selected by the user, but their ultimate goal is still to obtain the master library information of the data and compare it with the master library information in the data source management system in step S306 to determine whether the data to be accessed is a duplicate access.

[0080] The data access method provided by the present invention performs three checks on the filled configuration information, which can not only completely match the accessed information, but also dynamically identify the database master library information, ensure the maximum degree of repetitive access restrictions, and improve the accuracy and efficiency of data deduplication.

[0081] Figure 4 4 is a structural diagram of an apparatus for constructing a search model according to an embodiment of the present invention. Figure 4 The device shown is only an example and should not constitute any limitation to the functions and scope of use of the embodiments of the present invention.

[0082] refer to Figure 4 , the device includes a processor 401, a memory 402 and an input / output device 403 connected by a bus. The memory 402 includes a read-only memory (ROM) and a random access memory (RAM). The memory 402 stores various computer instructions and data required for executing system functions. The processor 401 reads various computer instructions from the memory 402 to perform various appropriate actions and processes. The input / output device includes an input part such as a keyboard and a mouse; an output part such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage part such as a hard disk; and a communication part such as a network interface card such as a LAN card and a modem. The memory 402 also stores the following computer instructions to complete the operation specified by the device of the embodiment of the present invention: before accessing the data to be accessed, the data to be accessed is subjected to a multi-level check, and the multi-level check includes: judging whether the data to be accessed is duplicate data according to the historical access record; judging whether the data to be accessed is duplicate data according to the master-slave relationship information; judging whether the data to be accessed is duplicate data according to the metadata.

[0083] Accordingly, an embodiment of the present invention provides a computer-readable storage medium, which stores computer instructions, and the computer instructions implement the operations specified in the above method when executed.

[0084] The flowcharts and block diagrams in the accompanying drawings illustrate the possible system frameworks, functions and operations of the systems, methods and devices of the embodiments of the present invention. The boxes on the flowcharts and block diagrams can represent a module, a program segment or just a piece of code, and the modules, program segments and codes are all executable instructions for implementing the specified logical functions. It should also be noted that the executable instructions for implementing the specified logical functions can be recombined to generate new modules and program segments. Therefore, the boxes and the order of the boxes in the accompanying drawings are only used to better illustrate the process and steps of the embodiments, and should not be used as a limitation on the invention itself.

[0085] Each module or unit of the system can be implemented by hardware, firmware or software. Software, for example, includes a coding program formed by various programming languages ​​such as JAVA, C / C++ / C#, SQL, etc. Although the steps and the order of the steps of the embodiment of the present invention are given in the method and the method legend, the executable instructions for the steps to implement the specified logical functions can be recombined to generate new steps. The order of the steps should not be limited to the order of the steps in the method and the method legend, and can be adjusted at any time according to the needs of the function. For example, some of the steps are executed in parallel or in reverse order.

[0086] The system and method according to the present invention can be deployed on a single or multiple servers. For example, different modules can be deployed on different servers to form dedicated servers. Alternatively, the same functional units, modules or systems can be deployed in a distributed manner on multiple servers to reduce load pressure. The servers include but are not limited to multiple PCs, PC servers, blade computers, supercomputers, etc. in the same local area network and connected via the Internet.

[0087] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data access method, characterized in that: Before accessing the data to be accessed, a multi-level check is performed on the data to be accessed, and the multi-level check includes: Judging whether the data to be accessed is duplicate data according to the historical access records includes: obtaining configuration information of the data to be accessed; searching the historical access records according to the configuration information; and determining whether the data is duplicate data according to the search result; Determining whether the data to be accessed is duplicate data according to the master-slave relationship information includes: acquiring the master-slave relationship information; retrieving the historical access record according to the master-slave relationship information and the configuration information; and determining whether it is duplicate data according to the retrieval result; Determining whether the data to be accessed is duplicate data according to the metadata includes: acquiring a prefix name of the accessed data according to the metadata; and generating an alias after access for the data to be accessed according to the prefix name, Among them, in the steps of the multi-level verification, the judgment of the next step is performed only when the previous step judges that it is not duplicate data.

2. The data access method according to claim 1, characterized in that: If the data to be accessed is library data, the configuration information includes the IP or host name of the data to be accessed; if the data to be accessed is table data, the configuration information includes the IP or host name of the data to be accessed and the table name.

3. The data access method according to claim 2, characterized in that: Obtain master-slave relationship information by calling the interface or command line.

4. The data access method according to claim 3, characterized in that: Also includes: The configuration information of the data to be accessed is obtained through the user interface.

5. A data access system, characterized in that: include: A verification module, wherein the verification module performs multi-level verification on the access data, and the verification module comprises: The first verification unit is used to determine whether the data to be accessed is duplicate data according to the historical access record, including: obtaining configuration information of the data to be accessed; searching the historical access record according to the configuration information; and determining whether it is duplicate data according to the search result; The second verification unit is used to determine whether the data to be accessed is duplicate data according to the master-slave relationship information, including: obtaining the master-slave relationship information; retrieving the historical access record according to the master-slave relationship information and the configuration information; and determining whether it is duplicate data according to the retrieval result; The third verification unit is used to determine whether the data to be accessed is duplicate data according to the metadata, including: obtaining a prefix name of the accessed data according to the metadata; and generating an alias after access for the data to be accessed according to the prefix name. The second verification unit is executed after the first verification unit verifies the data to be accessed and passes the verification, and the third verification unit is executed after the second verification unit verifies the data to be accessed and passes the verification.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed, the data access method according to any one of claims 1 to 4 is implemented.

7. A data access device, characterized in that: include: Memory, for storing computer instructions; A processor is coupled to the memory, and the processor is configured to execute the data access method according to any one of claims 1 to 4 based on computer instructions stored in the memory.

Images