Data collection method, device, electronic device and storage medium

Abstract

The present invention discloses a data collection method, device, electronic device and storage medium. The method is applied to a data collection engine, and includes: monitoring the rule configuration information of business data; when the rule configuration information changes, updating the rule configuration file and storage physical table based on the changed rule configuration information to obtain a target rule configuration file and a target storage physical table; performing quality inspection on the business data based on the target rule configuration file to obtain inspection results; storing the inspection results in the corresponding target storage physical table to obtain target governance data. The present invention monitors the changes of the rule configuration information through the data collection engine, and updates the rule configuration file and storage physical table, thereby realizing the automatic update of the rule configuration file and storage physical table, reducing the configuration time of the rule configuration file and storage physical table, and improving the efficiency of data collection.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data acquisition method, device, electronic equipment and storage medium. Background Art

[0002] With the maturity of technologies such as data marts and data warehouses, data applications in various business fields have developed rapidly, and data governance of business data has become an indispensable part.

[0003] At present, in the process of data governance, hard-coding is usually used to govern data. During the process of collecting data to be governed, this method can only adjust rules and other information by manually editing source code and recompiling executable files, and the collection efficiency is low. Summary of the invention

[0004] The present invention provides a data collection method, device, electronic equipment and storage medium to improve the efficiency of data collection.

[0005] According to one aspect of the present invention, there is provided a data collection method, which is applied to a data collection engine, comprising:

[0006] Monitoring business data rule configuration information;

[0007] In the case where the rule configuration information is changed, the rule configuration file and the storage physical table are updated based on the changed rule configuration information to obtain a target rule configuration file and a target storage physical table;

[0008] Performing quality inspection on the business data based on the target rule configuration file to obtain a inspection result;

[0009] The detection results are stored in the corresponding target storage physical table to obtain target management data.

[0010] According to another aspect of the present invention, there is provided a data acquisition device, which is applied to a data acquisition engine and comprises:

[0011] Information monitoring module, used to monitor the rule configuration information of business data;

[0012] An information updating module, used for updating the rule configuration file and the storage physical table based on the changed rule configuration information when the rule configuration information is changed, to obtain a target rule configuration file and a target storage physical table;

[0013] A data detection module, used to perform quality detection on the business data based on the target rule configuration file to obtain a detection result;

[0014] The data storage module is used to store the detection results in the corresponding target storage physical table to obtain target management data.

[0015] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data collection method described in any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data collection method described in any embodiment of the present invention when executed.

[0020] The technical solution of the embodiment of the present invention monitors the rule configuration information of business data through a data acquisition engine. When a change is detected in the rule configuration information, the data acquisition engine updates the rule configuration file and the storage physical table according to the changed rule configuration information, thereby obtaining an updated target rule configuration file and a target storage physical table, thereby realizing automatic updating of the rule configuration file and the storage physical table; further, the updated target rule configuration file can be used to perform quality inspection on the business data, and the inspection results of the quality inspection can be stored in the corresponding target storage physical table to obtain target governance data, thereby solving the problem of low collection efficiency of manually changing the rule configuration file and the storage physical table, reducing the configuration time of the rule configuration file and the storage physical table, and improving the efficiency of data collection.

[0021] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1is a flow chart of a data collection method provided according to Embodiment 1 of the present invention;

[0024] Figure 2 is a flow chart of a data collection method provided according to Embodiment 2 of the present invention;

[0025] Figure 3 This is a collection and assembly initialization flow chart provided according to the second embodiment of the present invention;

[0026] Figure 4 This is a collection scheduling flow chart provided according to the second embodiment of the present invention;

[0027] Figure 5 is a flow chart of a data collection method provided according to Embodiment 3 of the present invention;

[0028] Figure 6 This is a flow chart of data collection and tracing provided according to Embodiment 3 of the present invention;

[0029] Figure 7 is a structural schematic diagram of a data acquisition device provided according to a fourth embodiment of the present invention;

[0030] Figure 8 It is a structural schematic diagram of an electronic device for implementing the data collection method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "target", "original", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] Embodiment 1

[0034] Figure 1 A flowchart of a data acquisition method is provided for the first embodiment of the present invention. This embodiment is applicable to the case where a data acquisition engine automatically acquires data. The method can be executed by a data acquisition device. The data acquisition device can be implemented in the form of hardware and / or software. The data acquisition device can be configured in an electronic device, such as a server. Figure 1 As shown, the method includes:

[0035] S110: Monitoring rule configuration information of business data.

[0036] S120: When the rule configuration information is changed, the rule configuration file and the storage physical table are updated based on the changed rule configuration information to obtain a target rule configuration file and a target storage physical table.

[0037] S130: Perform quality detection on the business data based on the target rule configuration file to obtain a detection result.

[0038] S140. Store the detection results in the corresponding target storage physical table to obtain target management data.

[0039] In an embodiment of the present invention, the data collection engine may be an engine device for automatic data collection, and the data collection engine may monitor the rule configuration information of the business data to determine whether the rule configuration information has changed. The business data may be information generated during the business processing of the user, such as bank storage business data, credit business data, etc. The business data may include information such as customer number, customer name, and transaction amount. The rule configuration information may be a rule for performing operations such as data collection and detection, that is, information that complies with the rule may be collected through the rule configuration information.

[0040] Specifically, the rule configuration information can be monitored in a variety of ways. In some embodiments, the data acquisition engine obtains the rule configuration information at every preset time interval to determine whether the rule configuration information has changed. In some embodiments, the data acquisition engine is triggered to obtain the rule configuration information based on a preset trigger event to determine whether the rule configuration information has changed. For example, the trigger event may be the detection of an information input action on the rule configuration interface, which triggers the data acquisition engine to obtain the rule configuration information input on the rule configuration interface.

[0041] Furthermore, in the process of monitoring the configuration information of the rules, if the configuration information of the rules changes, it indicates that the collection or detection rules of the business data have changed, and it is necessary to update the rule configuration file and the storage physical table to ensure the accuracy of data collection. Among them, the rule configuration file may include one or more of the data collection rules and data detection rules, and the rule configuration file may be used to process the business data to obtain the data that needs to be governed. The data detection rules and data collection rules may be some rules set by means of page configuration, and the data detection rules may be used for data quality detection, such as the certificate number in the customer information cannot be empty, etc. The data collection rules may be used for data collection, that is, data collection is performed according to the preset collection rules, which may include but are not limited to the fields to be detected by the data detection rules, the data items to be collected and recorded, and the metadata index information, etc. After the data collection rules and data detection rules are set, they can be stored in a preset relational database, and the relational database may include Oracle, MySQl, Hive and other databases. The storage physical table refers to a physical table used to store data in the database, and the physical table may include multiple data items, for example, the physical table may include data items such as name, transaction time and transaction volume.

[0042] Specifically, the current rule configuration information is compared with the previous rule configuration information. If it is found that the current rule configuration information is different from the previous rule configuration information, it indicates that the rule configuration information has changed, and the rule configuration file and the storage physical table need to be updated to meet the current data collection needs and ensure the accuracy of data collection. It should be noted that the rule configuration file corresponds to the storage physical table one by one, that is, both are updated at the same time each time they are updated.

[0043] On the basis of the above-mentioned embodiment, the rule configuration file and the storage physical table are updated based on the changed rule configuration information to obtain the target rule configuration file and the target storage physical table, including: extracting the key identifier of the changed rule configuration information, dividing the key identifier by attributes, and constructing the target storage physical table based on the key identifier after the attribute division; taking the target storage physical table as a unit, compiling the changed rule configuration information according to the preset file configuration rules to obtain the target rule configuration file.

[0044] Among them, the key identifier refers to the extractable key information in the changed rule configuration information. Typically, the key identifier may include but is not limited to information such as key value, field name and field type. The purpose of attribute division is to classify the key identifiers. It can be understood that the key identifiers may include attribute information of multiple different types and levels. The key identifiers can be arranged or hierarchically divided according to the attribute information such as type and level. For example, the key identifiers of the same type are arranged at the same level; different types of key identifiers are arranged separately; key identifiers of different levels are arranged in layers, etc. Further, the target storage physical table can be constructed based on the arrangement or hierarchical division relationship of the key identifiers. The target rule configuration file is compiled based on the target storage physical table as a unit, which can ensure that each storage physical table has a corresponding target rule configuration file. In other words, the target rule configuration file is generated at the same time as the target storage physical table, so that the subsequent data can be stored accordingly.

[0045] Exemplarily, the key value, field name, field type and other information are extracted from the changed rule configuration information, and the extracted information is spliced ​​through SQL and other database statements to obtain the target storage physical table; further, the changed rule configuration information can be imported into the back-end business processing system in units of physical tables, and the business processing system normalizes, converts the format and other operations on the changed rule configuration information according to preset file configuration rules to obtain the target rule configuration file corresponding to the target storage physical table.

[0046] Further, the updated target rule configuration file is used to perform quality detection to obtain the detection result. The detection result is the result obtained by performing quality detection on the business data, that is, the data to be governed. Specifically, after the business data is detected by the data detection rules in the target rule configuration file, the data to be governed can be obtained.

[0047] It should be noted that data quality is the basis for ensuring data governance. The data detection rules for data quality detection mainly include four aspects: completeness, consistency, accuracy and timeliness. Completeness refers to whether there is any missing business data. The missing data may be the missing of the entire business data record or the missing record of a certain field information in the business data. The value of incomplete data will be greatly reduced, and it is also the most basic evaluation standard for data quality. Consistency refers to whether the business data follows a unified specification and whether the data set maintains a unified format. Accuracy refers to whether there are anomalies or errors in the information recorded in the data. Timeliness refers to the time interval from the generation of business data to the time it can be viewed, also called the delay time of the data. Through the above four aspects of quality detection of business data, the data to be governed that meets the data detection rules is obtained.

[0048] On the basis of the above embodiment, the quality detection of the business data based on the target rule configuration file includes: comparing the business data based on the data detection rules in the target rule configuration file, and using the comparison result as the detection result; wherein the data detection rule includes at least one of a data interval detection rule, a dictionary range detection rule, a list range detection rule and a function detection rule.

[0049] Specifically, the business data is compared with the data detection rules. If the business data meets the data detection rules, the business data that meets the data detection rules will be used as the detection result, that is, the business data that meets the data detection rules will be used as the data to be governed. Among them, the data interval detection rule refers to the detection rule of the numerical range, which can be used to determine whether the business data is within the preset numerical range; the dictionary range detection rule can be used to determine whether the data object or item in the business data is within the preset dictionary range; the list range detection rule can be used to determine whether the list of items in the business data is within the preset list range, and the function detection rule can be used to determine whether the function of the business data is complete or the format is correct.

[0050] Furthermore, the target governance data is stored in the form of a physical table, that is, the detection results are stored in the corresponding target storage physical table for subsequent calls, and the data can also be distinguished through the physical table.

[0051] An embodiment of the present invention provides a data collection method, which monitors the rule configuration information of business data through a data collection engine. When a change is detected in the rule configuration information, the data collection engine updates the rule configuration file and the storage physical table according to the changed rule configuration information, thereby obtaining an updated target rule configuration file and a target storage physical table, thereby realizing automatic updating of the rule configuration file and the storage physical table; further, the updated target rule configuration file can be used to perform quality inspection on the business data, and the inspection result of the quality inspection can be stored in the corresponding target storage physical table to obtain target governance data, thereby solving the problem of low collection efficiency caused by manually changing the rule configuration file and the storage physical table, reducing the configuration time of the rule configuration file and the storage physical table, and improving the efficiency of data collection.

[0052] Embodiment 2

[0053] Figure 2A flowchart of a data collection method provided for Embodiment 2 of the present invention, the embodiment of the present invention can be combined with each optional solution in the above embodiment. In the embodiment of the present invention, optionally, after updating the rule configuration file and the storage physical table based on the changed rule configuration information to obtain the target rule configuration file and the target storage physical table, the method further includes: disassembling the target rule configuration file, storing the disassembled configuration information in a relational database, and generating index information of the disassembled configuration information.

[0054] like Figure 2 As shown, the method includes:

[0055] S210: Monitoring rule configuration information of business data.

[0056] S220: When the rule configuration information is changed, the rule configuration file and the storage physical table are updated based on the changed rule configuration information to obtain a target rule configuration file and a target storage physical table.

[0057] S230: Decompose the target rule configuration file, store the decomposed configuration information in a relational database, and generate index information of the decomposed configuration information.

[0058] S240: Perform quality detection on the business data based on the target rule configuration file to obtain a detection result.

[0059] S250: Store the detection results in the corresponding target storage physical table to obtain target management data.

[0060] Among them, the target rule configuration file may include configuration rules for multiple objects. If the target rule configuration file is directly called to perform quality inspection on business data, the adaptability is poor. The target rule configuration file is disassembled by object, and the disassembled configuration information is stored in a relational database. The disassembled configuration information can be retrieved according to actual needs and assembled into a target rule configuration file, which can improve the adaptability of the target rule configuration file. A relational database refers to a database that uses a relational model to organize data and can store one or more target rule configuration files. The disassembled configuration information can be quickly retrieved through index information.

[0061] For example, Figure 3As shown, the data collection engine may include a collection assembly initialization engine, which monitors the status of the data configuration information. When the data configuration information changes, it will trigger the collection assembly initialization job to reinitialize the collection configuration information. The initialization work includes updating the version management of the stored information and data, and generating new rule configuration files, storage physical tables and other static data. After the assembly initialization job is completed, the update status and index information are marked.

[0062] On the basis of the above embodiment, before performing quality inspection on the business data based on the target rule configuration file, the method also includes: when the collection task meets the preset judgment conditions, obtaining the index information of the disassembled configuration information; matching the index information in the relational database to obtain the disassembled configuration information corresponding to the index information; assembling the disassembled configuration information corresponding to the index information to obtain the target rule configuration file.

[0063] For example, Figure 4 As shown, the preset judgment condition can be used to judge whether the collection task meets the scheduling requirements, wherein the preset judgment condition includes any one of the preset time, preset time interval or preset number of task executions. It can be understood that the collection task scheduling can be a timed task, and the collection task can be automatically executed when the preset time, preset time interval or preset number of task executions are met. Further, when the collection task needs to be scheduled, the index information of the disassembled configuration information is obtained, and the index information is matched with the static data (i.e., the configuration information after the rule configuration file is disassembled, etc.) in the relational database. If the configuration information after the rule configuration file is matched, the disassembled configuration information is assembled to obtain the target rule configuration file. Further, the rule engine component is called, such as drools, OpenL Tablets, Easy Rules, etc., and the data quality is detected using the matching rule configuration file, and the data to be governed is written into the matching target storage physical table, and the relevant scheduling log information is recorded, and then the next collection task is judged.

[0064] It should be noted that the index information is matched in the relational database and whether the match is successful is determined. The purpose of this setting is to prevent the use of unupdated rule configuration files for data quality detection during transmission of the rule configuration files, which can effectively improve the quality of data detection.

[0065] An embodiment of the present invention provides a data collection method, which disassembles a target rule configuration file, stores the disassembled configuration information in a relational database, and generates index information of the disassembled configuration information, so that the target rule configuration file has higher adaptability when assembled.

[0066] Embodiment 3

[0067] Figure 5 A flowchart of a data collection method provided for Embodiment 3 of the present invention, the embodiment of the present invention can be combined with the various optional schemes in the above embodiments. In the embodiment of the present invention, optionally, after storing the detection results in the target storage physical table to obtain the target governance data, the method further includes: extracting the target data item of the target governance data; using the target data item as the child node information of the metadata, determining the parent node information of the metadata according to the parent-child node relationship of the metadata; generating a traceability analysis report based on the child node information and parent node information of the metadata.

[0068] like Figure 5 As shown, the method includes:

[0069] S310: Monitoring rule configuration information of business data.

[0070] S320: When the rule configuration information is changed, the rule configuration file and the storage physical table are updated based on the changed rule configuration information to obtain a target rule configuration file and a target storage physical table.

[0071] S330: Perform quality detection on the business data based on the target rule configuration file to obtain a detection result.

[0072] S340: Store the detection result in the corresponding target storage physical table to obtain target management data.

[0073] S350, extracting target data items of the target governance data;

[0074] S360, taking the target data item as child node information of metadata, determining the parent node information of the metadata according to the parent-child node relationship of the metadata;

[0075] S370: Generate a traceability analysis report based on the child node information and parent node information of the metadata.

[0076] The target data item refers to the data item to be governed in the target governance data, and the target data item can be extracted from the target governance data by methods such as keyword extraction. Metadata is data that describes business data, or is used to provide structural data with association relationships, and the association relationship may include parent-child node relationships, etc. For example, metadata for business data may include information such as customer number, customer name, gender, and age.

[0077] Specifically, the target governance data may include multiple data items, from which one or more target data items can be extracted, and the target data items are used as child node information of metadata, and data tracing is performed based on the parent-child node relationship of metadata, and the parent node information of metadata is obtained. The child node information and parent node information of metadata can be used as key statistical information to form a traceability analysis report as a reference for data governance. Optionally, the traceability analysis report can also include system information that generates the target governance data, which is convenient for tracking the source system of the target governance data and realizing data traceability analysis.

[0078] For example, Figure 6 As shown, this embodiment can perform metadata configuration, collection rule configuration and detection rule configuration. After the configuration is completed, it can be stored in the corresponding relational database, which includes a metadata database, a detection rule library and a collection rule library. The data acquisition engine can retrieve the rule configuration information from the relational database to form a rule configuration file and a storage physical table. Furthermore, the data acquisition engine implements data quality detection through collection scheduling to obtain the data to be governed (i.e., the target governance data) and business integration data, among which the data to be governed can use data items and the parent-child node relationship of metadata for data traceability analysis.

[0079] An embodiment of the present invention provides a data collection method, which extracts target data items of target governance data, takes the target data items as child node information of metadata, and determines the parent node information of metadata according to the parent-child node relationship of metadata, thereby generating a traceability analysis report for the child node information and parent node information of metadata, thereby realizing automatic traceability analysis of the data to be governed.

[0080] Embodiment 4

[0081] Figure 7 This is a structural diagram of a data acquisition device provided by Embodiment 4 of the present invention. Figure 7 As shown, the device comprises:

[0082] Information monitoring module 410, used to monitor rule configuration information of business data;

[0083] An information updating module 420 is used to update the rule configuration file and the storage physical table based on the changed rule configuration information when the rule configuration information is changed, so as to obtain a target rule configuration file and a target storage physical table;

[0084] A data detection module 430 is used to perform quality detection on the business data based on the target rule configuration file to obtain a detection result;

[0085] The data storage module 440 is used to store the detection results in the corresponding target storage physical table to obtain target management data.

[0086] An embodiment of the present invention provides a data acquisition device, which monitors the rule configuration information of business data through a data acquisition engine. When a change is detected in the rule configuration information, the data acquisition engine updates the rule configuration file and the storage physical table according to the changed rule configuration information, thereby obtaining an updated target rule configuration file and a target storage physical table, thereby realizing automatic updating of the rule configuration file and the storage physical table; further, the updated target rule configuration file can be used to perform quality inspection on the business data, and the inspection result of the quality inspection can be stored in the corresponding target storage physical table to obtain target governance data, thereby solving the problem of low collection efficiency caused by manually changing the rule configuration file and the storage physical table, reducing the configuration time of the rule configuration file and the storage physical table, and improving the efficiency of data acquisition.

[0087] Based on any optional technical solution in the embodiment of the present invention, optionally, the information updating module 420 may also be used to:

[0088] Extracting the key identifier of the changed rule configuration information, dividing the key identifier by attributes, and constructing a target storage physical table based on the key identifier after the attribute division;

[0089] Taking the target storage physical table as a unit, the changed rule configuration information is compiled according to a preset file configuration rule to obtain a target rule configuration file.

[0090] Based on any optional technical solution in the embodiments of the present invention, optionally, the device further includes:

[0091] The information disassembly module is used to disassemble the target rule configuration file, store the disassembled configuration information in a relational database, and generate index information of the disassembled configuration information.

[0092] Based on any optional technical solution in the embodiments of the present invention, optionally, the device is further used for:

[0093] When the collection task meets the preset judgment condition, obtaining the index information of the disassembled configuration information;

[0094] Matching the index information in the relational database to obtain the disassembled configuration information corresponding to the index information;

[0095] The disassembled configuration information corresponding to the index information is assembled to obtain a target rule configuration file.

[0096] Based on any optional technical solution in the embodiments of the present invention, optionally, the preset judgment condition is used to judge whether the acquisition task meets the scheduling requirements, wherein the preset judgment condition includes any one of a preset time, a preset time interval or a preset number of task executions.

[0097] Based on any optional technical solution in the embodiments of the present invention, optionally, the data detection module 430 may also be used to:

[0098] Comparing the business data based on the data detection rule in the target rule configuration file, and using the comparison result as the detection result;

[0099] The data detection rule includes at least one of a data interval detection rule, a dictionary range detection rule, a list range detection rule and a function detection rule.

[0100] Based on any optional technical solution in the embodiments of the present invention, optionally, the device is further used for:

[0101] Extracting target data items of the target governance data;

[0102] Taking the target data item as the child node information of the metadata, determining the parent node information of the metadata according to the parent-child node relationship of the metadata;

[0103] A traceability analysis report is generated based on the child node information and parent node information of the metadata.

[0104] The data acquisition device provided in the embodiment of the present invention can execute the data acquisition method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0105] Embodiment 5

[0106] Figure 8A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0107] like Figure 8 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0108] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0109] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as a data acquisition method.

[0110] In some embodiments, the data collection method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the data collection method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the data collection method in any other suitable manner (e.g., by means of firmware).

[0111] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0112] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0113] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0114] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0115] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0116] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0117] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0118] Embodiment 6

[0119] Embodiment 6 of the present invention further provides a computer-readable storage medium, wherein the computer-executable instructions are used to execute a data collection method when executed by a computer processor, the method comprising:

[0120] Monitoring business data rule configuration information;

[0121] In the case where the rule configuration information is changed, the rule configuration file and the storage physical table are updated based on the changed rule configuration information to obtain a target rule configuration file and a target storage physical table;

[0122] Performing quality inspection on the business data based on the target rule configuration file to obtain a inspection result;

[0123] The detection results are stored in the corresponding target storage physical table to obtain target management data.

[0124] The computer-readable storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0125] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0126] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0127] Computer program code for performing the operation of embodiments of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0128] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A data collection method, characterized in that: Applied to data collection engines, including: Monitoring business data rule configuration information, wherein the rule configuration information is the rules for collecting and detecting data; In the case where the rule configuration information is changed, the rule configuration file and the storage physical table are updated based on the changed rule configuration information to obtain a target rule configuration file and a target storage physical table, wherein the rule configuration file includes data collection rules and data detection rules, and the storage physical table is a physical table for storing data in a database; Decomposing the target rule configuration file, storing the decomposed configuration information in a relational database, and generating index information of the decomposed configuration information; When the collection task meets the preset judgment condition, obtaining the index information of the disassembled configuration information, wherein the preset judgment condition is used to judge whether the collection task meets the scheduling requirement, wherein the preset judgment condition includes any one of a preset time, a preset time interval or a preset number of task executions; Matching the index information in the relational database to obtain the disassembled configuration information corresponding to the index information; Assembling the disassembled configuration information corresponding to the index information to obtain a target rule configuration file; Performing quality inspection on the business data based on the target rule configuration file to obtain a inspection result; The detection results are stored in the corresponding target storage physical table to obtain target management data.

2. The method according to claim 1, characterized in that The updating of the rule configuration file and the storage physical table based on the changed rule configuration information to obtain the target rule configuration file and the target storage physical table includes: Extracting the key identifier of the changed rule configuration information, dividing the key identifier by attributes, and constructing a target storage physical table based on the key identifier after the attribute division; Taking the target storage physical table as a unit, the changed rule configuration information is compiled according to a preset file configuration rule to obtain a target rule configuration file.

3. The method according to claim 1, characterized in that The performing quality detection on the business data based on the target rule configuration file includes: Comparing the business data based on the data detection rule in the target rule configuration file, and using the comparison result as the detection result; The data detection rule includes at least one of a data interval detection rule, a dictionary range detection rule, a list range detection rule and a function detection rule.

4. The method according to any one of claims 1 to 3, characterized in that After storing the detection result in the target storage physical table to obtain the target governance data, the method further includes: Extracting target data items of the target governance data; Taking the target data item as the child node information of the metadata, determining the parent node information of the metadata according to the parent-child node relationship of the metadata; A traceability analysis report is generated based on the child node information and parent node information of the metadata.

5. A data acquisition device, characterized in that: Applied to data collection engines, including: An information monitoring module is used to monitor the rule configuration information of business data, wherein the rule configuration information is the rules for collecting and detecting data; An information updating module, used for updating a rule configuration file and a storage physical table based on the changed rule configuration information when the rule configuration information is changed, to obtain a target rule configuration file and a target storage physical table, wherein the rule configuration file includes a data collection rule and a data detection rule, and the storage physical table is a physical table for storing data in a database; An information disassembly module, used to disassemble the target rule configuration file, store the disassembled configuration information in a relational database, and generate index information of the disassembled configuration information; When the collection task meets the preset judgment condition, obtaining the index information of the disassembled configuration information, wherein the preset judgment condition is used to judge whether the collection task meets the scheduling requirement, wherein the preset judgment condition includes any one of a preset time, a preset time interval or a preset number of task executions; Matching the index information in the relational database to obtain the disassembled configuration information corresponding to the index information; Assembling the disassembled configuration information corresponding to the index information to obtain a target rule configuration file; A data detection module, used to perform quality detection on the business data based on the target rule configuration file to obtain a detection result; The data storage module is used to store the detection results in the corresponding target storage physical table to obtain target management data.

6. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the data collection method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data collection method according to any one of claims 1 to 4 when executed.

Images