Data exploration method and apparatus, electronic device, and storage medium

Abstract

The present disclosure relates to a data exploration method, device and equipment and storage medium, and relates to the technical field of data processing. In the present disclosure, a target exploration item for a target data table is determined, the target exploration item including an exploration item for a table attribute of the target data table and an exploration item for a field attribute of a field in the target data table. Each exploration item is respectively provided with a corresponding exploration rule. According to the exploration rule corresponding to the target exploration item, the target data table is explored to obtain an exploration result corresponding to the target exploration item. The exploration result is used to represent the corresponding exploration situation of the target data table on the table attribute and the field attribute. According to the exploration result of the target exploration item, data intelligence information for the target data table is generated. The method can explore the target data table in a more fine-grained manner, so that a user object can comprehensively obtain intelligence information of the target data table.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a data exploration method, apparatus, electronic device, and storage medium. Background Technology

[0002] Currently, an increasing number of applications and services are built on data, highlighting its undeniable importance. Furthermore, data quality is the foundation for the validity and accuracy of data analysis and data mining conclusions, and a prerequisite for all data-driven decision-making. Therefore, ensuring data quality, accuracy, and usability is of paramount importance.

[0003] In related technologies, it is often only possible to explore general information such as the lifecycle, permissions, and business domain of a data table, but it is not possible to explore the data table in a more granular way, which makes it impossible for users to fully obtain the intelligence information of the data table. Summary of the Invention

[0004] This disclosure provides a data exploration method, apparatus, device, and storage medium that can explore data tables with finer granularity, enabling users to comprehensively obtain intelligence information from the data tables.

[0005] The technical solution of this disclosure is as follows: According to a first aspect of the present disclosure, a data exploration method is provided. The method includes: determining target exploration items for a target data table, the target exploration items including exploration items for table attributes of the target data table and exploration items for field attributes of fields in the target data table; wherein each exploration item is preset with a corresponding exploration rule; exploring the target data table according to the exploration rule corresponding to the target exploration item to obtain exploration results corresponding to the target exploration item; the exploration results are used to characterize the exploration status of the target data table in terms of table attributes and field attributes; and generating data table intelligence information for the target data table based on the exploration results of the target exploration items.

[0006] Optionally, each exploration item has a pre-defined exploration result processing rule. Based on the exploration result of the target exploration item, data table intelligence information for the target data table is generated, including: converting the exploration result corresponding to each target exploration item into a standard exploration result that meets the pre-defined display conditions according to the pre-defined exploration result processing rule for each target exploration item; generating data table intelligence information in a pre-defined format, which is used to display the standard exploration result corresponding to each target exploration item.

[0007] Optionally, after generating the data table information in a preset format, the above method further includes: sending the data table information to the terminal device where the target user account is located, wherein the target user account is an account that has a preset association with the target data table.

[0008] Optionally, the target data table can be explored according to the exploration rules corresponding to the target exploration item, including: if the target data table does not exist in the database, the target data table can be explored according to the exploration rules corresponding to the target exploration item.

[0009] Optionally, the above method also includes: if the target data table contains data table information in the database, sending the data table information to the terminal device.

[0010] Optionally, the target data table is explored according to the exploration rules corresponding to the target exploration item to obtain the exploration results corresponding to the target exploration item, including: identifying redundant fields in the target data table as the first field; determining the ratio of the first field in the target data table to the total number of fields in the target data table; and determining the exploration results corresponding to the target exploration item based on the ratio of the number of the first field to the total number of fields.

[0011] Optionally, if the target probe item includes field format, the target data table is probed according to the probe rules corresponding to the target probe item to obtain the probe results corresponding to the target probe item, including: identifying the fields with abnormal format in the target data table as second fields; determining the ratio of the number of second fields in the target data table to the total number of fields in the target data table; and determining the probe results corresponding to the target probe item based on the ratio of the number of second fields to the total number of fields.

[0012] Optionally, when the target probe item includes a partition attribute, the target data table is probed according to the probe rules corresponding to the target probe item to obtain the probe results corresponding to the target probe item, including: when the target data table is partitioned by date, the partition continuity in the target data table is determined according to the date corresponding to each partition in the target data table; and the probe results corresponding to the target probe item are determined according to the ratio of the number of target partitions in the target data table to the total number of partitions in the target data table, wherein the target partition is a partition with sub-partitions.

[0013] Optionally, the target probe item for the target data table includes: receiving a request sent by the terminal device for probing the target data table, the request including a first probe item specified by the user account on the terminal device side for the target data table; and determining the first probe item as the target probe item.

[0014] Optionally, the target exploration item for the target data table includes: when a target data table that meets the preset exploration conditions is generated, the preset second exploration item is determined as the target exploration item.

[0015] Optionally, the preset detection conditions are: usage frequency greater than preset usage frequency and / or preset priority level higher than preset level.

[0016] According to a second aspect of the present disclosure, a data probing apparatus is provided, comprising: a first determining unit configured to determine target probing items for a target data table, the target probing items including probing items for table attributes of the target data table and probing items for field attributes of fields in the target data table; wherein each probing item is preset with a corresponding probing rule; a second determining unit configured to perform probing on the target data table according to the probing rule corresponding to the target probing item, and obtain probing results corresponding to the target probing item; the probing results are used to characterize the corresponding probing situation of the target data table in terms of table attributes and field attributes; and an information generating unit configured to generate data table intelligence information for the target data table according to the probing results of the target probing items.

[0017] Optionally, each exploration item has a corresponding exploration result processing rule preset. The information generation unit is specifically configured to perform the following: according to the exploration result processing rule preset for each target exploration item, convert the exploration result corresponding to each target exploration item into a standard exploration result that meets the preset display conditions; generate a data table intelligence information in a preset format, which is used to display the standard exploration result corresponding to each target exploration item.

[0018] Optionally, after generating the data table information in a preset format, the information generation unit is also configured to: send the data table information to the terminal device where the target user account is located, where the target user account is an account with a preset association with the target data table.

[0019] Optionally, the second determining unit is specifically configured to perform the following: if the target data table does not exist in the database, to probe the target data table according to the probe rules corresponding to the target probe item.

[0020] Optionally, the information generation unit is also configured to perform the following: if data table intelligence information for the target data table exists in the database, send the data table intelligence information to the terminal device.

[0021] Optionally, the second determining unit is configured to perform: determining redundant fields in the target data table as first fields; determining the ratio of the first field in the target data table to the total number of fields in the target data table; and determining the exploration result corresponding to the target exploration item based on the ratio of the number of first fields to the total number of fields.

[0022] Optionally, when the target probe item includes field format, the second determining unit is specifically configured to perform: determining the fields with abnormal format in the target data table as second fields; determining the ratio of the number of second fields in the target data table to the total number of fields in the target data table; and determining the probe result corresponding to the target probe item based on the ratio of the number of second fields to the total number of fields.

[0023] Optionally, when the target probe item includes a partition attribute, the second determining unit is specifically configured to perform: when the target data table is partitioned by date, determine the partition continuity in the target data table based on the date corresponding to each partition in the target data table; and determine the probe result corresponding to the target probe item based on the ratio of the number of target partitions in the target data table to the total number of partitions in the target data table, wherein the target partition is a partition with sub-partitions.

[0024] Optionally, the target probe item for the target data table includes: receiving a request sent by the terminal device for probing the target data table, the request including a first probe item specified by the user account on the terminal device side for the target data table; and determining the first probe item as the target probe item.

[0025] Optionally, the target exploration item for the target data table includes: when a target data table that meets the preset exploration conditions is generated, the preset second exploration item is determined as the target exploration item.

[0026] Optionally, the preset detection conditions are: usage frequency greater than preset usage frequency and / or preset priority level higher than preset level.

[0027] According to a third aspect of the present disclosure, an electronic device is provided, which may include: a processor and a memory for storing processor-executable instructions; wherein the processor is configured to execute instructions to implement any of the optional data probing methods of the first aspect described above.

[0028] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which instructions are stored, such that when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform any of the optional data probing methods of the first aspect described above.

[0029] According to a fifth aspect of the present disclosure, a computer program product is provided, the computer program product including computer instructions that, when executed on an electronic device, cause the electronic device to perform a data probing method as optionally implemented in the first aspect.

[0030] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure.

[0031] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects: Based on any of the above, this disclosure defines target probe items for a target data table. These target probe items include probe items for table attributes of the target data table and probe items for field attributes of fields in the target data table. Each probe item has a pre-defined probe rule. The target data table is probed according to the probe rule corresponding to the target probe item, yielding probe results for each target probe item. These probe results characterize the probe status of the target data table in terms of table attributes and field attributes. Based on the probe results of the target probe items, data table intelligence information for the target data table is generated. This method allows for more granular probing of the target data table, enabling users to comprehensively obtain intelligence information about the target data table. Attached Figure Description

[0032] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0033] Figure 1 A schematic diagram of the structure of a data probing system provided in an embodiment of this disclosure is shown; Figure 2 This diagram illustrates a data probing architecture provided in an embodiment of the present disclosure. Figure 3 A flowchart illustrating a data probing method provided in an embodiment of this disclosure is shown; Figure 4 This illustration shows a schematic diagram of the interface of a terminal device provided in an embodiment of the present disclosure; Figure 5 A schematic diagram of the structure of a data probing device provided in an embodiment of this disclosure is shown; Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure is shown. Detailed Implementation

[0034] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0035] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0036] It should also be understood that the term "comprising" indicates the presence of the described feature, whole, step, operation, element and / or component, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements and / or components.

[0037] The data disclosed herein may be data authorized by the user or fully authorized by all parties.

[0038] Currently, an increasing number of applications and services are built on data, highlighting its undeniable importance. Furthermore, data quality is the foundation for the validity and accuracy of data analysis and data mining conclusions, and a prerequisite for all data-driven decision-making. Therefore, ensuring data quality, accuracy, and usability is of paramount importance.

[0039] In related technologies, it is often only possible to explore general information such as the lifecycle, permissions, and business domain of a data table, but it is not possible to explore the data table in a more granular way, which makes it impossible for users to fully obtain the intelligence information of the data table.

[0040] Based on this, this disclosure provides a data exploration method, comprising: determining target exploration items for a target data table, wherein the target exploration items include exploration items for table attributes of the target data table and exploration items for field attributes of fields in the target data table; wherein each exploration item is preset with a corresponding exploration rule. The target data table is explored according to the exploration rule corresponding to the target exploration item to obtain the exploration result corresponding to the target exploration item; the exploration result is used to characterize the exploration status of the target data table in terms of table attributes and field attributes. Based on the exploration result of the target exploration item, data table intelligence information for the target data table is generated. This method can explore the target data table with finer granularity, enabling users to comprehensively obtain intelligence information about the target data table.

[0041] The following provides illustrative examples of application scenarios for the data probing method provided in this disclosure: Figure 1 This is a schematic diagram of a data exploration system provided in an embodiment of the present disclosure, such as... Figure 1As shown, the data exploration system may include a server 110 and a terminal device 120. The server 110 can establish a connection with the terminal device 120 via a wired network or a wireless network.

[0042] The server 110 can be used to receive a request sent by the terminal device 120 to probe a target data table. The request includes a first probe item specified by the user account on the terminal device side for the target data table. The user can determine the specific content of the first probe item through the terminal device 120.

[0043] In some embodiments, server 110 may be a single server, or it may be a server cluster consisting of multiple servers (or microservers). The server cluster may also be a distributed cluster. This disclosure does not limit the specific implementation of server 110.

[0044] Terminal device 120 can be used to send a request to the server to explore the target data table, and at the same time receive data table information of the target data table returned by server 110.

[0045] In some embodiments, the terminal device can be a mobile phone, tablet computer, desktop computer, laptop computer, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, as well as cellular phone, personal digital assistant (PDA), augmented reality (AR) / virtual reality (VR) device, etc., which can install and use content community applications (such as Kuaishou). This disclosure does not impose any special restrictions on the specific form of the terminal. It can interact with users through one or more methods such as keyboard, touchpad, touch screen, remote control, voice interaction, or handwriting device.

[0046] Optionally, the above Figure 1 In the data exploration system shown, server 110 can be connected to at least one terminal device 120. This disclosure does not limit the number or type of terminal devices 120.

[0047] The data probing method provided in this disclosure can be applied to the aforementioned... Figure 1 The server shown is 110.

[0048] In some embodiments, the entity executing the data probing method provided in this disclosure may be a data probing device, which may be built into the server 110 described above.

[0049] In some embodiments, the data probing apparatus described above includes a data probing architecture 130. Figure 2 This is a schematic diagram illustrating a data probing architecture exemplified in this disclosure, such as... Figure 2 As shown, the data exploration architecture 130 includes a database 131, a computing engine 132, and a data table acquisition node 133.

[0050] The computing engine 132 is used to receive a request sent from the terminal device 120 corresponding to the user account, which requests to perform data exploration on the first exploration item of the target data table; and to obtain the corresponding preset exploration rules from the database 131 according to the target data table and the corresponding first exploration item.

[0051] Database 131 is used to send the target data table and the preset exploration rules corresponding to the first exploration item to the calculation engine 132.

[0052] The computing engine 132 is also used to probe the target data table according to the probe rules corresponding to the first probe item, obtain the data table intelligence information of the target data table, return the data table intelligence information of the target data table to the terminal device 120 corresponding to the user account, and send the data table intelligence information of the target data table to the database 131 for storage.

[0053] The data table acquisition node 133 is used to retrieve data tables from the database 131, determine the data tables that meet the preset exploration conditions as target data tables, and send the target data tables and the preset second exploration items to the computing engine 132.

[0054] It should be noted that the data table acquisition node 133 can retrieve data tables from the database 131 at regular intervals. For example, it can retrieve them once a day or once a week. The preset search conditions can be data tables whose usage frequency exceeds the preset usage frequency, or data tables whose search priority of the user account's pre-installed device is higher than the preset level.

[0055] In some embodiments, when a data table meets the preset exploration conditions, the database 131 is also used to send the preset exploration rules corresponding to the target data table and the second exploration item to the computing engine 132.

[0056] The calculation engine 132 is used to obtain the data table intelligence information of the target data table according to the preset exploration rules corresponding to the target data table and the second exploration item, and send the data table intelligence information of the target data table to the database 131 for storage.

[0057] Figure 3 A flowchart of a data probing method provided in this disclosure embodiment is shown below. Figure 3 As shown, when a data probing method is applied to a server or terminal device, the data probing method may include: S301. Determine the target probe items for the target data table. The target probe items include probe items for the table attributes of the target data table and probe items for the field attributes of the fields in the target data table. Each probe item has a corresponding preset probe rule.

[0058] In one implementation, targeting a target data table includes: receiving a request from a terminal device to probe the target data table, the request including a first probe item specified by a user account on the terminal device side for the target data table, and identifying the first probe item as the target probe item.

[0059] In the above implementation, the request is initiated by the user account on the corresponding terminal device. The user account can select the target data table and the corresponding first probe item on the terminal device. See also Figure 4 , Figure 4 This illustration shows a schematic diagram of the interface of a terminal device according to an embodiment of the present disclosure. The interface includes multiple data table controls, multiple probe item controls, and a task creation control. A user account can select the data table they wish to probe according to their needs, i.e., determine the target data table from multiple data tables. In this embodiment, the target data table selected by the user is data table B. Then, a first probe item is determined from the multiple probe items. In this embodiment, the probe items selected by the user account are probe item A, probe item B, probe item H, and probe item I. Each probe item has a corresponding preset probe rule. After the user determines the selected target data table and the corresponding first probe item, they select the task creation control to generate a request for probing data table B, and send the request to the server through the terminal device.

[0060] Specifically, table attributes include at least one of the table's partitioning attribute and the total number of fields in the table; field attributes include at least one of the following: field null value rate, field length, field uniqueness, field value range, field dictionary code, and field format. Specifically, the null value probe indicates the ratio of the number of fields with null values ​​to the total number of fields in the table; the field length probe indicates the maximum and minimum field lengths for each field in the table; the field enumeration probe indicates the ratio of the number of fields with duplicate data to the total number of fields in the table; the field length probe indicates the maximum and minimum field ranges for each field in the table; the field dictionary code probe indicates the dictionary code for each field in the table; and the field format probe indicates the ratio of the number of fields with abnormal formats to the total number of fields in the table. By probing multiple parameters in the table attributes and field attributes of the target data table, a more granular exploration of the data table can be achieved, enabling users to obtain comprehensive information about the data table.

[0061] S302. Based on the exploration rules corresponding to the target exploration items, explore the target data table to obtain the exploration results corresponding to the target exploration items; the exploration results are used to characterize the corresponding exploration status of the target data table in terms of table attributes and field attributes.

[0062] Specifically, each exploration item has a pre-set corresponding exploration rule, which is stored in the database. The server can retrieve the pre-set exploration rule corresponding to each exploration item in the target exploration item from the database based on the request to explore the target data table. S303, Generate data table intelligence information for the target data table based on the exploration results of the target exploration item.

[0063] In one implementation, each probe item has a pre-defined probe result processing rule, and the above S303 includes: Based on the preset detection result processing rules for each target detection item, the detection results corresponding to each target detection item are converted into standard detection results that meet preset display conditions. A preset format data table intelligence information is generated, which is used to display the standard detection results corresponding to each target detection item.

[0064] Specifically, since the exploration results include multiple parameters, such as field null value rate, field duplication rate, format anomaly rate, maximum and minimum field length, maximum and minimum field value range, and at least one of the dictionary codes corresponding to each field, the exploration results corresponding to each target exploration item are converted into standard exploration results that meet the preset display conditions through exploration result processing rules, generating data table intelligence information in a preset format, and storing the data table intelligence information in the database.

[0065] In some embodiments, the data table information is further configured to display the corresponding item in red in the data table information when any of the following exceeds a preset threshold: field null value rate, field duplication rate, format anomaly rate, maximum and minimum field length, maximum and minimum field value range, or dictionary code corresponding to each field; and when it does not exceed the preset threshold, display the corresponding item in green in the data table information.

[0066] As can be seen from the above, by using the exploration result processing rules to convert the exploration results corresponding to each target exploration item into standard exploration results that meet the preset display conditions, the corresponding data table intelligence information is obtained. User accounts can view the corresponding data table intelligence information, which allows for intuitive and comprehensive acquisition of the target data table intelligence information, thereby improving the usability and accuracy of the data.

[0067] As shown in S301-S303, by determining the target probe items for the target data table, which include probe items for table attributes and probe items for field attributes of the target data table, each probe item has a corresponding preset probe rule. Based on the probe rule corresponding to the target probe item, the target data table is probed, yielding the probe results corresponding to the target probe item. The probe results characterize the probe status of the target data table in terms of table attributes and field attributes. Based on the probe results of the target probe items, data table intelligence information for the target data table is generated. This allows for more granular probing of the target data table, enabling users to comprehensively obtain intelligence information about the target data table.

[0068] In one implementation, S302 includes: identifying redundant fields in the target data table as first fields; determining the ratio of the first field in the target data table to the total number of fields in the target data table; and determining the exploration result corresponding to the target exploration item based on the ratio of the number of first fields to the total number of fields.

[0069] In the above implementation, when the target probe item includes the field null value rate, the first field can be a field containing null values. For example, the total number of fields in the target data table is 100. According to the probe rules, it is determined whether each field contains null values. When a field contains null values, that field is designated as the first field. Since the number of first fields in the current target data table is 20, and the ratio can be a percentage, the percentage of the number of first fields in the target data table is 20%. Therefore, the probe result corresponding to this probe item is that the field null value rate of the target data table is 20%.

[0070] In the data development process, when building new models, user accounts cannot assess the presence of null values ​​in various fields of the data table. Even if the null values ​​are manually verified, the process is cumbersome and inefficient. The technical solution provided in this disclosure automatically probes the null value rate of fields in the data table through preset probing rules, enabling more granular data probing of the target data table so that user accounts can have a comprehensive understanding of the data table's information.

[0071] In the above implementation, when the target probe item includes the duplication rate, the first field can be a field with duplicate content. The ratio of the number of the first field in the target data table to the total number of fields in the target data table is determined. Based on the ratio of the number of the first field to the total number of fields, the probe result corresponding to the probe item is determined.

[0072] For example, the total number of fields in the target data table is 100. According to the probing rules, it is determined whether there is duplicate data in each field. When there is duplicate data in a certain field, the field is determined as the first field. Since the number of first fields in the target data table is 30, the number of first fields accounts for 30% of the total number of fields. At this time, the probing result of this probing item is that the duplication rate of the target data table is 30%.

[0073] As can be seen from the above, by automatically probing the fields with duplicate data in the data table through preset probing rules, it is possible for user accounts to understand the duplication rate of the fields with duplicate data in the data table, thereby realizing data probing of the target data table with finer granularity, so that user accounts can have a comprehensive understanding of the information in the data table.

[0074] In one implementation, when the target probe item includes a field length, the above S302 includes: determining the field length corresponding to each field in the target data table, and determining the probe result corresponding to the probe item based on the maximum and minimum values ​​of the field lengths corresponding to each field.

[0075] As can be seen from the above, by automatically probing the field length in the data table through preset probing rules, it is possible for user accounts to understand the maximum and minimum values ​​of the field length in the data table, thereby realizing data probing of the target data table with finer granularity, so that user accounts can have a comprehensive understanding of the information in the data table.

[0076] In some embodiments, when the target probe item includes a range of field values, S302 above includes: determining the range of values ​​corresponding to each field in the target data table; and determining the probe result corresponding to the probe item based on the maximum and minimum values ​​in the range of values ​​corresponding to each field.

[0077] As can be seen from the above, by automatically probing the numerical range of fields in a data table through preset probing rules, users can easily understand the maximum and minimum values ​​of the numerical range of fields in the data table. This enables more granular data probing of the target data table, allowing users to have a comprehensive understanding of the information in the data table.

[0078] In some embodiments, when the target probe includes a probe for field dictionary codes, the above S302 includes: determining the dictionary code corresponding to each field in the target data table, and determining the probe result corresponding to the probe based on the dictionary code corresponding to each field.

[0079] As can be seen from the above, by automatically probing the dictionary code corresponding to each field in the data table through preset probing rules, it is possible for user accounts to understand the dictionary code corresponding to each field in the data table. This enables more granular data probing of the target data table, allowing user accounts to have a comprehensive understanding of the information in the data table.

[0080] In one implementation, when the target probe includes probes for field format, S302 includes: identifying the field with abnormal format in the target data table as the second field; determining the ratio of the number of second fields in the target data table to the total number of fields in the target data table; and determining the probe result corresponding to the target probe based on the ratio of the number of second fields to the total number of fields.

[0081] For example, the target data table has a total of 100 fields. According to the probing rules, it is determined whether there is a format anomaly in each field. When a field has a format anomaly, it is identified as the second field. The number of second fields in the current target data table is 10. The number of second fields accounts for 10% of the total number of fields. At this time, the probing result is that the format anomaly rate of the target data table is 10%.

[0082] As described above, by using preset probing rules to determine the number of fields with formatting errors in a data table, users can easily understand the ratio of fields with formatting errors to the total number of fields, thus obtaining the corresponding formatting error rate. This enables more granular data probing of the target data table, allowing users to gain a comprehensive understanding of the data table's information.

[0083] In some embodiments, when the target probe includes an attribute indicating whether the representation table is a partitioned table, the above-mentioned S302 includes: If the target data table is a partitioned data table, determine the corresponding data type of the target data table.

[0084] As can be seen from the above, probing the type of data table through preset probing rules can facilitate users to evaluate data quality at a more granular level, understand data attributes, improve data usability and accuracy, and increase efficiency.

[0085] In some embodiments where the target probe includes an attribute indicating whether the table is a partitioned table, S302 above includes: determining the date corresponding to each partition when the target data table is partitioned by date; determining the maximum value, minimum value, and whether the dates are consecutive.

[0086] As can be seen from the above, by probing the dates corresponding to each partition in the partitioned data table using preset probing rules, determining the maximum and minimum values ​​of the dates corresponding to each partition, and judging whether the dates corresponding to each partition are consecutive, it is possible for user accounts to evaluate the data with more granularity, understand the data attributes, improve the usability and accuracy of the data, and improve efficiency.

[0087] In some embodiments, when the target probe item includes an attribute indicating whether the table is a partitioned table, the above S302 includes: determining a partition in the target data table that has sub-partitions as a target partition; determining the number of target partitions in the target data table; and determining a first probe result based on the ratio of the number of target partitions to the total number of fields.

[0088] As can be seen from the above, by probing each partition in the partitioned data table using preset probing rules, the number of partitions with sub-partitions can be determined. This allows user accounts to probe the data with more granularity, understand the data attributes, improve the usability and accuracy of the data, and increase efficiency.

[0089] In some embodiments, when the target probe includes the total number of fields in the table, the above-mentioned S302 includes: determining the total number of fields in the target data table.

[0090] As can be seen from the above, by probing each partition in the partitioned data table using preset probing rules, the number of partitions with sub-partitions can be determined. This allows user accounts to evaluate data with more granularity, understand data attributes, improve data usability and accuracy, and increase efficiency.

[0091] In one implementation, S302 includes: when there is no data table intelligence information for the target data table in the database, probing the target data table according to the probing rules corresponding to the target probing item.

[0092] In another implementation, the above method further includes: if the target data table's data table information exists in the database, sending the target data table's data table information to the terminal device.

[0093] As described above, by storing data table intelligence information in the database, when multiple users need to explore the same target data table, if one user has already completed the exploration and obtained the corresponding data table intelligence information, since the data table intelligence information is stored in the database, other users can directly retrieve the corresponding data table intelligence information from the database when they need to obtain it. This avoids multiple user accounts conducting multiple data explorations of the same target data table, thus preventing a waste of computing resources.

[0094] In one implementation, the target probe item for the target data table includes: when a target data table that meets preset probe conditions is generated, a preset second probe item is determined as the target probe item. The preset probe conditions are: usage frequency greater than a preset usage frequency and / or preset priority level higher than a preset level.

[0095] In the above implementation, the server crawls data tables. When the usage frequency of a crawled data table exceeds a preset usage frequency, that data table is designated as the target data table, and a preset second probe item is designated as the target probe item. The target data table is then probed according to the probe rules corresponding to the preset second probe item. The task of probing the target data table is automatically placed in a scheduling queue for execution. This task includes the preset second probe item, which includes probe items targeting table attributes of the target data table and / or probe items targeting field attributes of fields in the target data table.

[0096] In some embodiments, user accounts pre-assign priority levels to data tables stored in the database. When the server fetches data tables, if the priority level of the fetched data table is greater than the preset level, the data table is identified as the target data table, and a preset second probe item is identified as the target probe item. The target data table is then probed according to the probe rules corresponding to the preset second probe item. The task of probing the target data table is automatically placed in a scheduling queue to await execution. This task includes the preset second probe item, which includes probe items targeting table attributes of the target data table and / or probe items targeting field attributes of fields in the target data table.

[0097] In some embodiments, once the data table information of the target data table is determined, it is automatically or periodically sent to the terminal device where the target user account resides via a message. The target user account is an account with a preset association with the target data table. For example, the target user account can be a user account with editing and modification permissions for the target data table, or a user account with management permissions for the target data table. The target user account can also use the corresponding terminal device to view the corresponding data table information.

[0098] The server retrieves data tables that meet preset exploration conditions from the database, designates these tables as target data tables, and pre-explores the target data tables according to preset second exploration items. The obtained data table intelligence information is then stored in the database. When a user needs to obtain the data table intelligence information, they can directly retrieve the corresponding data table intelligence information, avoiding the waiting process of exploring the target data table, saving exploration time, effectively conserving computing resources, and improving data exploration efficiency.

[0099] It is understood that, in actual implementation, the terminal / server of this disclosure embodiment may include one or more hardware structures and / or software modules for implementing the aforementioned corresponding data probing methods, and these hardware structures and / or software modules may constitute an electronic device. Those skilled in the art should readily recognize that, based on the algorithmic steps of the various examples described in conjunction with the embodiments disclosed herein, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is implemented in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0100] Based on this understanding, this disclosure also provides a data probing device that can be applied to terminal devices. Figure 5A schematic diagram of the structure of the data probing device provided in an embodiment of this disclosure is shown. Figure 5 As shown, the data exploration device may include: a first determining unit 610, a second determining unit 620, and an information generation unit 630.

[0101] The first determining unit 610 is configured to determine target exploration items for a target data table. The target exploration items include exploration items for table attributes of the target data table and exploration items for field attributes of fields in the target data table. Each exploration item has a corresponding exploration rule. The second determining unit 620 is configured to explore the target data table according to the exploration rules corresponding to the target exploration items to obtain exploration results corresponding to the target exploration items. The exploration results are used to characterize the exploration status of the target data table in terms of table attributes and field attributes. The information generation unit 630 is configured to generate data table intelligence information for the target data table based on the exploration results of the target exploration items.

[0102] Optionally, each exploration item has a corresponding exploration result processing rule preset. The information generation unit 630 is specifically configured to perform the following: according to the exploration result processing rule preset for each target exploration item, convert the exploration result corresponding to each target exploration item into a standard exploration result that meets the preset display conditions; generate a data table intelligence information in a preset format, which is used to display the standard exploration result corresponding to each target exploration item.

[0103] Optionally, after generating the data table information in a preset format, the information generation unit is also configured to: send the data table information to the terminal device where the target user account is located, where the target user account is an account with a preset association with the target data table.

[0104] Optionally, the second determining unit 620 is specifically configured to perform the following: if the target data table does not exist in the database, to explore the target data table according to the exploration rules corresponding to the target exploration item.

[0105] Optionally, the information generation unit 630 is also configured to perform the following: if data table intelligence information for the target data table exists in the database, send the data table intelligence information to the terminal device.

[0106] Optionally, the second determining unit 620 is configured to perform: determining redundant fields in the target data table as first fields; determining the ratio of the first field in the target data table to the total number of fields in the target data table; and determining the exploration result corresponding to the target exploration item based on the ratio of the number of first fields to the total number of fields.

[0107] Optionally, when the target probe item includes field format, the second determining unit 620 is specifically configured to perform: determining the field with abnormal format in the target data table as the second field; determining the ratio of the number of second fields in the target data table to the total number of fields in the target data table; and determining the probe result corresponding to the target probe item based on the ratio of the number of second fields to the total number of fields.

[0108] Optionally, when the target probe item includes a partition attribute, the second determining unit 620 is specifically configured to perform: when the target data table is partitioned by date, determining the partition continuity in the target data table based on the date corresponding to each partition in the target data table; and determining the probe result corresponding to the target probe item based on the ratio of the number of target partitions in the target data table to the total number of partitions in the target data table, wherein the target partition is a partition with sub-partitions.

[0109] Optionally, the target probe item for the target data table includes: receiving a request sent by the terminal device for probing the target data table, the request including a first probe item specified by the user account on the terminal device side for the target data table; and determining the first probe item as the target probe item.

[0110] Optionally, the target exploration item for the target data table includes: when a target data table that meets the preset exploration conditions is generated, the preset second exploration item is determined as the target exploration item.

[0111] Optionally, the preset detection conditions are: usage frequency greater than preset usage frequency and / or preset priority level higher than preset level.

[0112] As described above, the embodiments of this disclosure can divide the electronic device into functional modules according to the above method examples. The integrated modules can be implemented in hardware or as software functional modules. Furthermore, it should be noted that the module division in these embodiments is illustrative and represents only one logical functional division; in actual implementation, other division methods may be used. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into a single processing module.

[0113] The specific methods by which each module performs its operations and the beneficial effects of the data exploration device in the above embodiments have been described in detail in the foregoing method embodiments, and will not be repeated here.

[0114] This disclosure also provides an electronic device. Figure 6A schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure is shown. This electronic device may be a data probing device and may include at least one processor 71, a communication bus 72, a memory 73, and at least one communication interface 74.

[0115] Processor 71 can be a central processing unit (CPU), a microprocessor unit, an ASIC, or one or more integrated circuits used to control the execution of programs according to the present disclosure. As an example, combined with... Figure 5 The functions implemented by the first determining unit 610, the second determining unit 620, and the information generating unit 630 in the electronic device are the same as those of... Figure 6 The processor 71 in it performs the same function.

[0116] The communication bus 72 may include a path for transmitting information between the aforementioned components.

[0117] Communication interface 74 uses any transceiver-like device for communicating with other devices or communication networks, such as servers, Ethernet, radio access networks (RAN), wireless local area networks (WLAN), etc. As an example, The memory 73 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory may exist independently and be connected to the processing unit via a bus. The memory may also be integrated with the processing unit.

[0118] The memory 73 stores the application code that executes the present invention, and its execution is controlled by the processor 71. The processor 71 executes the application code stored in the memory 73 to implement the functions of the method of the present invention.

[0119] In a specific implementation, as one example, processor 71 may include one or more CPUs, for example... Figure 6 CPU0 and CPU1 in the CPU.

[0120] In a specific implementation, as one example, an electronic device may include multiple processors, for example... Figure 6 Processors 71 and 75 are included. Each of these processors may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. A processor here may refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0121] In a specific implementation, as one embodiment, the electronic device may further include an input device 76 and an output device 77. The input device 76 and output device 77 communicate and can accept user input in various ways. For example, the input device 76 may be a mouse, keyboard, touchscreen device, or sensing device. The output device 77 communicates with the processor 71 and can display information in various ways. For example, the output device 71 may be a liquid crystal display (LCD), a light-emitting diode (LED) display device, etc.

[0122] Those skilled in the art will understand that Figure 6 The structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0123] This disclosure also provides an electronic device. This electronic device can be a data probing device. The electronic device can vary significantly due to differences in configuration or performance, and may include one or more processors and one or more memories. The memory stores at least one instruction, which is loaded and executed by the processor to implement the data probing methods provided in the above-described method embodiments. Of course, the electronic device may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The electronic device may also include other components for implementing device functions, which will not be elaborated here.

[0124] This disclosure also provides a computer-readable storage medium including instructions stored thereon. When the instructions in the computer-readable storage medium are executed by a processor of a computer device, the computer is able to perform the data probing method provided in the embodiments described above. For example, the computer-readable storage medium can be a memory 73 including instructions, which can be executed by a processor 71 of a terminal to complete the above method. As another example, the computer-readable storage medium can be a memory including instructions, which can be executed by a processor of an electronic device to complete the above method. Optionally, the computer-readable storage medium can be a non-transitory computer-readable storage medium, such as a ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage device.

[0125] This disclosure also provides a computer program product including computer instructions that, when executed on an electronic device, cause the electronic device to perform the above-described actions. Figures 1-4 The data exploration method shown in any of the attached figures.

[0126] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0127] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A data exploration method, characterized in that, The method includes: The target probe items for the target data table are determined. The target probe items include probe items for table attributes of the target data table and probe items for field attributes of fields in the target data table. Each probe item is preset with a corresponding probe rule. The target data table is explored according to the exploration rules corresponding to the target exploration item to obtain the exploration results corresponding to the target exploration item; the exploration results are used to characterize the exploration status of the target data table on the table attributes and the field attributes; Based on the exploration results of the target exploration items, generate data table intelligence information for the target data table; The process of identifying target exploration items for the target data table includes: The terminal device receives a request to probe the target data table, the request including a first probe item specified by the user account on the terminal device side for the target data table, and identifies the first probe item as the target probe item; and / or, When a target data table that meets the preset exploration conditions is generated, the preset second exploration item is determined as the target exploration item, wherein the preset exploration conditions include at least one of the following: the usage frequency of the target data table is greater than the preset usage frequency, and the priority level of the target data table is higher than the preset level. In the case where the second exploration item is determined to be the target exploration item, the task of exploring the target data table according to the exploration rules corresponding to the target exploration item is automatically placed into the scheduling queue to wait for execution. Specifically, probing the target data table according to the probing rules corresponding to the target probing item includes: if the target data table does not exist in the database, probing the target data table according to the probing rules corresponding to the target probing item, and storing the probing results corresponding to the target probing item in the database; if the target data table exists in the database, directly retrieving the data table information from the database and sending it to the terminal device.

2. The data exploration method according to claim 1, characterized in that, Each of the exploration items has a corresponding exploration result processing rule. The step of generating data table intelligence information for the target data table based on the exploration results of the target exploration item includes: According to the preset detection result processing rules for each target detection item, the detection result corresponding to each target detection item is converted into a standard detection result that meets the preset display conditions; Generate a data table intelligence information with a preset format, which is used to display the standard exploration results corresponding to each of the target exploration items.

3. The data exploration method according to claim 2, characterized in that, After generating the data table information in the preset format, the method further includes: The data table information is sent to the terminal device where the target user account is located. The target user account is an account that has a preset association with the target data table.

4. The data exploration method according to claim 1, characterized in that, The step of probing the target data table according to the probing rule corresponding to the target probing item to obtain the probing result corresponding to the target probing item includes: The redundant field in the target data table is designated as the first field; Determine the ratio of the first field in the target data table to the total number of fields in the target data table; The exploration result corresponding to the target exploration item is determined based on the ratio of the number of the first field to the total number of fields.

5. The data exploration method according to claim 4, characterized in that, When the target probe includes a probe for the field's null value rate, the step of probing the target data table according to the probe rule corresponding to the target probe to obtain a first probe result includes: The field in the target data table that contains null values ​​is identified as the first field; Determine the ratio of the number of the first field in the target data table to the total number of fields in the target data table; The first exploration result is determined based on the ratio of the number of the first field to the total number of fields.

6. The data exploration method according to claim 4, characterized in that, When the target probe includes a probe for the field length, the step of probing the target data table according to the probe rule corresponding to the target probe to obtain a first probe result includes: Determine the field length corresponding to each field in the target data table; The first exploration result is determined based on the maximum and minimum values ​​of the field length corresponding to each field.

7. The data exploration method according to claim 4, characterized in that, When the target probe includes a probe for the uniqueness of the field, the step of probing the target data table according to the probe rule corresponding to the target probe to obtain a first probe result includes: The field in the target data table containing duplicate data is identified as the second field; Determine the ratio of the number of the second field in the target data table to the total number of fields in the target data table; The first exploration result is determined based on the ratio of the number of the second field to the total number of the fields.

8. The data exploration method according to claim 4, characterized in that, When the target probe item includes a probe item targeting the numerical range of the field, the step of probing the target data table according to the probe rule corresponding to the target probe item to obtain a first probe result includes: Determine the numerical range corresponding to each field in the target data table; The first exploration result is determined based on the maximum and minimum values ​​within the numerical range corresponding to each field.

9. The data exploration method according to claim 4, characterized in that, When the target probe includes a probe for the field dictionary code, the step of probing the target data table according to the probe rule corresponding to the target probe to obtain a first probe result includes: Determine the dictionary code corresponding to each field in the target data table; The first exploration result is determined based on the dictionary code corresponding to each field.

10. The data exploration method according to claim 4, characterized in that, When the target probe item includes a probe item for the field format, the step of probing the target data table according to the probe rule corresponding to the target probe item to obtain a first probe result includes: The field with an abnormal format in the target data table is identified as the third field; Determine the ratio of the number of the third field in the target data table to the total number of fields in the target data table; The first exploration result is determined based on the ratio of the number of the third field to the total number of fields.

11. The data exploration method according to claim 4, characterized in that, When the target probe item includes the partition attribute of the table, the step of probing the target data table according to the probe rule corresponding to the target probe item to obtain the first probe result includes: When the target data table is partitioned by date, determine the date corresponding to each partition in the target data table, as well as the maximum and minimum values ​​of the dates; Based on the date corresponding to each partition in the target data table, determine whether the partitions in the target data table are consecutive.

12. The data exploration method according to claim 4, characterized in that, When the target probe item includes the partition attribute of the table, the step of probing the target data table according to the probe rule corresponding to the target probe item to obtain the first probe result further includes: Determine the target partition in the target data table, wherein the target partition is a partition that has sub-partitions; The first exploration result is determined based on the ratio of the number of target partitions in the target data table to the total number of partitions in the target data table.

13. The data exploration method according to claim 4, characterized in that, When the target probe item includes the total number of fields in the table, the step of probing the target data table according to the probe rule corresponding to the target probe item to obtain the first probe result includes: determining the total number of fields in the target data table.

14. A data probing device, characterized in that, The device includes: The first determining unit is configured to determine target probe items for a target data table. The target probe items include probe items for table attributes of the target data table and probe items for field attributes of fields in the target data table. Each probe item is preset with a corresponding probe rule. The second determining unit is configured to perform an exploration of the target data table according to the exploration rule corresponding to the target exploration item, and obtain the exploration result corresponding to the target exploration item; the exploration result is used to characterize the exploration status of the target data table on the table attribute and the field attribute; The information generation unit is configured to generate data table intelligence information for the target data table based on the exploration results of the target exploration item; The first determining unit is further configured to execute: The terminal device receives a request to probe the target data table, the request including a first probe item specified by the user account on the terminal device side for the target data table, and identifies the first probe item as the target probe item; and / or, When a target data table that meets the preset exploration conditions is generated, the preset second exploration item is determined as the target exploration item, wherein the preset exploration conditions include at least one of the following: the usage frequency of the target data table is greater than the preset usage frequency, and the priority level of the target data table is higher than the preset level. In the case where the second exploration item is determined to be the target exploration item, the task of exploring the target data table according to the exploration rules corresponding to the target exploration item is automatically placed into the scheduling queue to wait for execution. The second determining unit is further configured to perform the following: if the target data table does not exist in the database, to probe the target data table according to the probe rules corresponding to the target probe item, and to store the probe results corresponding to the target probe item in the database; if the target data table exists in the database, to directly obtain the data table information from the database and send it to the terminal device.

15. The data probing device according to claim 14, characterized in that, After probing the target data table according to the probing rule corresponding to the target probing item to obtain the first probing result, the second determining unit is further configured to: Based on the first exploration results, a data exploration report of the target data table is generated; The data exploration report of the target data table is returned to the terminal device, and the data exploration report of the target data table is stored in the database.

16. The data probing device according to claim 14, characterized in that, The first determining unit is further configured to: Receive a probe result query request sent by the terminal device, the probe result query request being used to request the probe results of the target data table; Based on the exploration results query request, retrieve the data exploration report of the target data table from the database; The data exploration report of the target data table is sent to the terminal device.

17. The data probing device according to claim 15, characterized in that, The table attributes include at least one of the table's partition attributes and the total number of fields in the table; the field attributes include at least one of the following: field null value rate, field length, field uniqueness, field value range, field dictionary code, and field format.

18. The data probing device according to claim 17, characterized in that, When the target probe includes a probe for the field null value rate, the second determining unit is specifically used for: The field in the target data table that contains null values ​​is identified as the first field; Determine the ratio of the number of the first field in the target data table to the total number of fields in the target data table; The first exploration result is determined based on the ratio of the number of the first field to the total number of fields.

19. The data probing device according to claim 17, characterized in that, When the target probe includes a probe for the field length, the second determining unit is specifically used for: Determine the field length corresponding to each field in the target data table; The first exploration result is determined based on the maximum and minimum values ​​of the field length corresponding to each field.

20. The data probing device according to claim 17, characterized in that, When the target probe includes a probe for the uniqueness of the field, the second determining unit is specifically used for: The field in the target data table containing duplicate data is identified as the second field; Determine the ratio of the number of the second field in the target data table to the total number of fields in the target data table; The first exploration result is determined based on the ratio of the number of the second field to the total number of the fields.

21. The data probing device according to claim 17, characterized in that, When the target probe includes probes targeting the numerical range of the field, the second determining unit is specifically used for: Determine the numerical range corresponding to each field in the target data table; The first exploration result is determined based on the maximum and minimum values ​​within the numerical range corresponding to each field.

22. The data probing device according to claim 17, characterized in that, When the target probe includes a probe for the field dictionary code, the second determining unit is specifically used for: Determine the dictionary code corresponding to each field in the target data table; The first exploration result is determined based on the dictionary code corresponding to each field.

23. The data probing device according to claim 17, characterized in that, When the target probe includes a probe for the field format, the second determining unit is specifically used for: The field with an abnormal format in the target data table is identified as the third field; Determine the ratio of the number of the third field in the target data table to the total number of fields in the target data table; The first exploration result is determined based on the ratio of the number of the third field to the total number of fields.

24. The data probing device according to claim 17, characterized in that, When the target probe includes the partition attribute of a table, the second determining unit is configured to: When the target data table is partitioned by date, determine the date corresponding to each partition in the target data table, as well as the maximum and minimum values ​​of the dates; Based on the date corresponding to each partition in the target data table, determine whether the partitions in the target data table are consecutive.

25. The data probing device according to claim 23, characterized in that, When the target probe includes the partition attribute of a table, the second determining unit is further configured to: Determine the target partition in the target data table, wherein the target partition is a partition that has sub-partitions; The first exploration result is determined based on the ratio of the number of target partitions in the target data table to the total number of partitions in the target data table.

26. The data probing device according to claim 17, characterized in that, If the target probe includes the total number of fields in the table, the second determining unit is further configured to determine the total number of fields in the target data table.

27. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the data probing method as described in any one of claims 1-13.

28. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the data probing method as described in any one of claims 1-13.

Images