Data processing method, device, electronic device, storage medium and program product

Abstract

The present disclosure provides a data processing method, device, electronic device, storage medium and program product, which relate to the fields of artificial intelligence technology such as big data, feature mining, and machine learning. The method includes: determining a core database table in the candidate database table; using the core database table as the starting point of traversal and the primary key of the core database table as the starting associated element, and determining the associated database table associated with the core database table layer by layer through a breadth-first traversal algorithm; using the core database table as the starting point of traversal and the primary key of the associated database table of the previous layer as the screening condition for screening out associated data from the associated database of the next layer, and extracting the associated data through a depth-first traversal algorithm. This method can automatically mine target data that can be used to generate data features from the database by performing breadth-first traversal and depth-first traversal in sequence, which is more convenient and efficient.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, specifically to the field of artificial intelligence technology such as big data, feature mining, and machine learning, and in particular to a data processing method, device, electronic device, computer-readable storage medium, and computer program product. Background Art

[0002] Feature mining usually takes up more than half of the time in the entire machine learning project, and the effect of feature mining also determines the effect of the entire model.

[0003] Conventional feature mining mainly includes the following steps: determining all original data sets used for feature mining, filtering the data contained in the original data sets, confirming the filtered data types, and generating features according to the confirmed data types. Summary of the invention

[0004] The embodiments of the present disclosure provide a data processing method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] In a first aspect, an embodiment of the present disclosure proposes a data processing method, including: determining a core database table in the candidate database tables; using the core database table as a traversal starting point and the primary key of the core database table as a starting associated element, and determining the associated database table associated with the core database table layer by layer through a breadth-first traversal algorithm; using the core database table as a traversal starting point and the primary key of the associated database table of the previous layer as a screening condition for screening associated data from the associated database of the next layer, and extracting the associated data through a depth-first traversal algorithm.

[0006] In a second aspect, an embodiment of the present disclosure proposes a data processing device, including: a core database table determination unit, configured to determine a core database table in an alternative database table; a breadth-first traversal unit, configured to use the core database table as a traversal starting point and the primary key of the core database table as a starting associated element, and to determine the associated database table associated with the core database table layer by layer through a breadth-first traversal algorithm; a depth-first traversal unit, configured to use the core database table as a traversal starting point and the primary key of the associated database table of the previous layer as a screening condition for screening out associated data from the associated database of the next layer, and to extract the associated data through a depth-first traversal algorithm.

[0007] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can implement the data processing method described in the first aspect when executing.

[0008] In a fourth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, which are used to enable a computer to implement the data processing method described in the first aspect when executed.

[0009] In a fifth aspect, an embodiment of the present disclosure provides a computer program product comprising a computer program, which, when executed by a processor, can implement the data processing method as described in the first aspect.

[0010] The data processing scheme provided by the present disclosure, for each database table constituting the database, first determines the core database table as the core in each candidate database table, then uses the core database table as the starting point of traversal and gradually uses it as the starting associated element, confirms the associated database table that is directly or indirectly associated with the core database table layer by layer through breadth-first traversal, then uses the primary key of the associated database table of the previous layer as the screening condition for screening the associated data from the associated database of the next layer, and screens the target data from the associated database tables of each layer through depth-first traversal. That is, by successively performing breadth-first traversal and depth-first traversal, the target data that can be used to generate data features can be automatically mined from the database, which is more convenient and efficient.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Other features, objects and advantages of the present disclosure will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0013] Figure 1 is an exemplary system architecture in which the present disclosure may be applied;

[0014] Figure 2 A flowchart of a data processing method provided by an embodiment of the present disclosure;

[0015] Figure 3a A schematic diagram of a process for determining an associated database table through breadth-first traversal provided in an embodiment of the present disclosure;

[0016] Figure 3b A schematic diagram of a process for filtering target data from database tables at each level through depth-first traversal provided in an embodiment of the present disclosure;

[0017] Figure 4 A flowchart of a method for determining the data type of target data by combining field statistics and field semantics provided in an embodiment of the present disclosure;

[0018] Figure 5 A flowchart of a method based on maximum probability and determination of target data type provided by an embodiment of the present disclosure;

[0019] Figure 6 A corresponding schematic diagram of a processing method for performing different feature mining on different data types provided in an embodiment of the present disclosure;

[0020] Figure 7 A structural block diagram of a data processing device provided in an embodiment of the present disclosure;

[0021] Figure 8 A schematic diagram of the structure of an electronic device suitable for executing a data processing method provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description. It should be noted that the embodiments in the present disclosure and the features in the embodiments may be combined with each other without conflict.

[0023] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0024] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the data processing method, apparatus, electronic device, and computer-readable storage medium of the present disclosure can be applied.

[0025] like Figure 1As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0026] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various applications for realizing information communication between the terminal devices 101, 102, 103 and server 105 can be installed, such as data screening applications, data transmission applications, feature mining applications, etc.

[0027] Terminal devices 101, 102, 103 and server 105 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc.; when terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above, which can be implemented as multiple software or software modules, or as a single software or software module, and no specific limitation is made here. When server 105 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server; when the server is software, it can be implemented as multiple software or software modules, or as a single software or software module, and no specific limitation is made here.

[0028] The server 105 can provide various services through various built-in applications. Taking a feature mining application that can provide a data screening service in a database for feature mining as an example, the server 105 can achieve the following effects when running the feature mining application: first, the core database table selection information transmitted by the terminal devices 101, 102, and 103 is received through the network 104, so as to determine the core database table in the candidate database table; then, the core database table is used as the starting point of the traversal and the primary key of the core database table is used as the starting association element, and the associated database table associated with the core database table is determined layer by layer through the breadth-first traversal algorithm; next, the core database table is used as the starting point of the traversal and the primary key of the associated database table of the upper layer is used as the screening condition for screening out the associated data from the associated database of the next layer, and the associated data is extracted through the depth-first traversal algorithm.

[0029] It should be noted that, in addition to being temporarily obtained from the terminal devices 101, 102, 103 through the network 104, the selected information of the core database table can also be pre-stored locally in the server 105 in various ways. Therefore, when the server 105 detects that the data has been stored locally (for example, starting to process the tasks to be processed before), it can choose to directly obtain the data locally. In this case, the exemplary system architecture 100 may also not include the terminal devices 101, 102, 103 and the network 104.

[0030] Since the database contains a large amount of data in the form of tables, it takes more computing resources and stronger computing power to process it. Therefore, the data processing methods provided in the subsequent embodiments of the present disclosure are generally performed by the server 105 with stronger computing power and more computing resources. Accordingly, the data processing device is generally also set in the server 105. However, it should also be pointed out that when the terminal devices 101, 102, and 103 also have computing power and computing resources that meet the requirements, the terminal devices 101, 102, and 103 can also complete the above-mentioned operations that are handed over to the server 105 through the feature mining applications installed thereon, and then output the same results as the server 105. Especially in the case where there are multiple terminal devices with different computing powers at the same time, but when the feature mining application determines that the terminal device has stronger computing power and more remaining computing resources, the terminal device can be allowed to perform the above-mentioned operations, thereby appropriately reducing the computing pressure of the server 105. Accordingly, the data processing device can also be set in the terminal devices 101, 102, and 103. In this case, the exemplary system architecture 100 may also not include the server 105 and the network 104 .

[0031] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0032] Please refer to Figure 2 , Figure 2 A flowchart of a data processing method provided by an embodiment of the present disclosure, wherein process 200 includes the following steps:

[0033] Step 201: Determine a core database table from among the candidate database tables;

[0034] This step is intended to be performed by the execution entity of the data processing method (e.g. Figure 1 The server 105 shown determines a core table, ie, a core database table, from among the candidate database tables.

[0035] Among them, the number of alternative database tables is usually multiple, which can be all or part of the database tables constituting the target database, or all or part of the database tables constituting a sub-database under the target database; the core table is usually a database table that uses the starting information that the user wants to perform or instructs to perform feature mining as the primary key of the database table, that is, the core table most directly contains the starting information that the user wants to perform feature mining.

[0036] In practice, the above-mentioned execution entity may first receive the incoming or acquired core table indication information, and then determine the corresponding core database table from a number of alternative database tables based on the core table indication information. The core table indication information should at least include relevant information that enables the above-mentioned execution entity to determine the core table. The relevant information may be selected or set according to actual conditions, and is not specifically limited here.

[0037] For ease of understanding, the present disclosure also provides a specific implementation method including but not limited to the following steps:

[0038] First, based on the external input information received or the information obtained, the feature mining scenario and feature mining requirements are determined; then, the target database corresponding to the feature mining scenario is determined, and the database tables (all by default) that constitute the target database are determined as candidate database tables; next, the starting information for feature mining is determined according to the feature mining requirements; finally, the candidate database table with the starting information as the primary key is determined as the core database table.

[0039] That is, this specific implementation method first determines the target database that meets the feature mining scenario among multiple databases, and then uses the feature mining requirements to determine the core database tables among the candidate database tables that constitute the target database. That is, it provides an implementation plan for first determining the mining scenario and then determining the mining starting information. This is beneficial for narrowing the scope of database table screening and improving the efficiency of determining the core database tables when there are multiple databases in different fields and scenarios as candidates.

[0040] Specifically, the scenarios represented by this feature mining scenario can be: insurance policy information mining scenario, contract information mining scenario, holiday travel hotspot information mining scenario, e-commerce platform purchase and sales product information mining scenario, etc., which can correspond to the insurance policy database, contract database, travel database, and e-commerce platform product database respectively.

[0041] Step 202: using the core database table as the traversal starting point and the primary key of the core database table as the starting association element, and determining the associated database table associated with the core database table layer by layer through a breadth-first traversal algorithm;

[0042] Based on step 201, this step aims to have the execution subject start from the core database table and determine all database tables that are directly or indirectly associated with the core database table (hereinafter collectively referred to as associated database tables) layer by layer through a breadth-first traversal algorithm.

[0043] Among them, Breadth First Search (BFS) originally refers to a graph search method, which means starting from an untraversed node in the graph, first traversing the adjacent nodes of this node, and then traversing the adjacent nodes of each adjacent node in turn. In the present disclosure, the core database table is used as the starting node for traversal, and then other database tables containing the primary key of the core database table are used as its adjacent nodes, and then other databases containing the primary key of the database table of each adjacent node are used as the adjacent nodes of the adjacent nodes in turn... and so on until no new adjacent nodes are found in the last layer.

[0044] Taking mining insurance policy information as an example, see Figure 3a The traversal process diagram shown:

[0045] Figure 3a The sample table in the example is the core database table, and its primary key is the ID number. This means that other tables must be associated with it through the ID number, so the table associated with the sample table must contain the ID number field. According to this restriction, the remaining data tables are traversed to find the data tables that meet the conditions and add them to the current relationship. From the figure, we can see that the sample table and the insurance policy table can be associated through the ID number.

[0046] After completing the exploration starting from the sample table, the breadth-first traversal algorithm enters the second-level traversal. The second level includes the policy table. The primary key of the policy table is the policy number, so the table associated with it must contain the policy number. According to this association, the insured table and the claim table can be found.

[0047] The search for deeper levels is repeated in this way until the maximum depth requirement is met or no new associations can be found, and then the search stops.

[0048] Step 203: Using the core database table as the traversal starting point and the primary key of the associated database table of the previous layer as the screening condition for screening associated data from the associated database of the next layer, the associated data is extracted through a depth-first traversal algorithm.

[0049] On the basis of step 202, this step aims to extract all data associated with the primary key of the core database table (hereinafter collectively referred to as associated data) from the associated database tables of each layer from the core database table by the execution subject starting from the core database table through a depth-first traversal algorithm, according to the primary key of the associated database table of the previous layer as the screening condition for screening out associated data from the associated database of the next layer.

[0050] Taking mining insurance policy information as an example, see Figure 3b The traversal process diagram shown:

[0051] The data table associated with the sample table often stores all the data. However, if all of the data is extracted to construct features, it will not only affect the performance of feature construction, but also make it impossible to complete the feature construction work when machine resources are insufficient. Therefore, it is necessary to extract the data that can be used to construct features from the data table associated with the sample table according to the primary key of the sample table.

[0052] like Figure 3b As shown, the extraction of associated data is performed using a depth-first traversal algorithm, starting with the sample table as the starting node. All of its subtables (policy table) are filtered according to the primary key (ID number) in the sample table. After the filtering is completed, the traversal is started with the policy table as the new starting node, and all of its subtables (insured person table, claim table) are filtered according to the primary key (policy number) of the policy table. After reaching the insured person table and the claim table, the entire data extraction process is completed.

[0053] The data processing method provided by the embodiment of the present disclosure, for each database table constituting the database, first determines the core database table as the core in each candidate database table, then uses the core database table as the starting point of traversal and gradually uses it as the starting associated element, confirms the associated database table that is directly or indirectly associated with the core database table layer by layer through breadth-first traversal, then uses the primary key of the associated database table of the previous layer as the screening condition for screening the associated data from the associated database of the next layer, and screens the target data from the associated database tables of each layer through depth-first traversal. That is, by performing breadth-first traversal and depth-first traversal in sequence, the target data that can be used to generate data features can be automatically mined from the database, which is more convenient and efficient.

[0054] After the associated data is extracted from each associated database table by breadth-first traversal and depth-first traversal in the above embodiment, in order to complete the purpose of feature mining, the data type to which the associated data belongs can be determined immediately afterwards, and then the associated data is mined using a feature mining method corresponding to the data type, that is, feature mining is performed on associated data of different data types using a feature mining method that matches the data type, and usually different types of data correspond to different feature mining methods.

[0055] Specifically, for some associated data that can be further split into multiple words or characters, the data type of the associated data can also be determined based on the data types of the multiple words split from the associated data. For example, the data type of each word or character constituting the associated data is first counted, and then the data type with the largest number of statistics is determined as the data type of the associated data.

[0056] For example, a certain associated data can be split into 7 words, and these 7 words are respectively determined to belong to 4 data types, of which 4 words are determined to belong to data type A, 2 words are determined to belong to data type B, and the remaining 2 words are respectively determined to belong to data types C and D, so data type A should be determined as the data type to which the associated data belongs. Of course, in addition to the solution of determining the data type according to the maximum number given in this embodiment, in different application scenarios, other implementation solutions that meet the requirements can also be formulated according to actual conditions, which will not be listed here one by one.

[0057] On the basis of the above embodiment, in order to deepen the understanding of how to determine the data type to which the associated data belongs, this embodiment further Figure 4 An implementation scheme for determining the data type of target data by combining field statistics and field semantics is shown, and the process 400 includes the following steps:

[0058] Step 401: Determine the field statistics and field semantics of the associated data;

[0059] This step aims to determine the field statistics and field semantics of the associated data by the above-mentioned execution subject. For each field that constitutes the associated data, a statistical analysis of the field is first performed to calculate the coverage rate, number of unique values, proportion of top 10 values, entropy value, minimum value, mean value, variance, maximum value, minimum value, entropy value, mean value, variance, proportion of top 10 values, etc. of the field.

[0060] Among them, coverage rate refers to the quotient of the number of covered samples and the total number of records; the number of unique values ​​refers to the number of field values ​​after deduplication after counting; the proportion of top 10 values ​​refers to the number of times each value in the field appears, sorting in reverse order, accumulating the number of top 10 values, and finally taking the quotient of the accumulated value and the total number of records; the maximum value, minimum value, entropy value, mean, variance, and proportion of top 10 values ​​of value length first count the length of each value in the field, and then count the maximum value, minimum value, entropy value, mean, variance, and proportion of top 10 values ​​of value length.

[0061] The entropy value can be calculated using the following formula:

[0062]

[0063] In the formula, X represents a random variable, x i represents the possible values ​​of the random variable X, p(x i ) indicates that the random variable X has a specific value x i H(X) represents the entropy value of the random variable X.

[0064] Among them, field semantics is usually determined by semantic judgment of the field name, because the field name can also identify the field type in many cases. For example, field names containing date, time, date, and time are generally date types, and field names ending with ID, no, and number are mostly ID (number) type variables; fields ending with quantity, amount, and price are mostly continuous variables, and fields ending with status, code, code, flag, flag, code, status, etc. are generally categorical variables.

[0065] It can be seen that field statistics or field semantics are associated with field types. In order to more accurately determine the data type of each associated data, the present disclosure uses field statistics and field semantics in combination to combine the judgment results obtained by the two judgment methods, thereby improving the accuracy of the results.

[0066] Step 402: determining a first probability that the corresponding associated data belongs to a different data type according to the field statistics;

[0067] Step 403: determining a second probability that the corresponding associated data belongs to a different data type according to the field semantics;

[0068] Step 402 and step 403 are for the execution subject to determine the probability (ie, the first probability and the second probability) that the corresponding associated data belongs to different data types according to the field statistics and the field semantics respectively.

[0069] Step 404: Determine the target data type to which the corresponding associated data belongs by combining the first probability and the second probability.

[0070] On the basis of step 402 and step 403, this step aims at determining the target data type to which the corresponding associated data belongs by the above-mentioned execution subject combining the first probability and the second probability.

[0071] Specifically, the comprehensive approach may include: dynamic weighting of the equal distribution ratio and the unequal distribution ratio, and adding correction values ​​or correction coefficients on this basis.

[0072] An implementation including but not limited to can be found in Figure 5 The flowchart shown in FIG. 5 includes the following steps:

[0073] Step 501: Determine a first weight corresponding to a first probability and a second weight corresponding to a second probability;

[0074] This step aims to enable the execution subject to first determine a first weight corresponding to the first probability and a second weight corresponding to the second probability.

[0075] The first weight and the second weight can be preset, or the preset basic value can be fine-tuned based on the current actual situation to better meet the actual situation. The fine-tuning method and tendency can be determined according to the actual situation. For example, the actual data category is more accurately judged from the perspective of field semantics, so the value of the second weight can be appropriately increased.

[0076] Step 502: weighting the first probability using a first weight, and weighting the second probability using a second weight;

[0077] On the basis of step 501, this step aims at having the above-mentioned execution entities weight their corresponding probabilities respectively using corresponding weights, so as to obtain the weighted first probability and second probability.

[0078] Step 503: According to the weighted probability sum of the first probability and the second probability, the data type with the maximum probability sum is determined as the target data type.

[0079] On the basis of step 502, this step aims to have the above-mentioned execution subject add the weighted first probability and the second probability that each associated data is judged to belong to the same data type to obtain the probability sum of the data type, and then perform horizontal comparison at each data type level based on the size of the probability sum, and finally determine the data type with the largest probability sum as the target data type.

[0080] To deepen understanding, this disclosure also specifically combines the insurance policy scenario and provides a complete implementation solution on how to implement feature mining from this scenario:

[0081] The input of feature mining is structured data, which is mainly expressed in relational databases such as MySQL, Oracle, and Hive (three different relational databases). Through the data relationship extraction module, the data table associated with the sample table can be automatically obtained. After manual confirmation and correction, the modeling data can be automatically extracted. Then, the data field type inference tool is used to determine the type of the field, and different feature construction methods are applied to each field type. Finally, the feature file is exported. The following is an expanded explanation of several important parts:

[0082] 1. Data relationship extraction:

[0083] Since the extraction of data relations is mainly for a large number of database tables, it is necessary to first find out the data tables related to the sample table, and then infer the relationship between each table and the sample table at the same time.

[0084] In the original database, the relationship between the sample table and other tables is unknown, and it is necessary to manually associate other tables to convert them into data tables with association relationships. In this embodiment, the implementation method of the breadth-first traversal algorithm is used. For details, please refer to Figure 2 In the embodiment shown, step 202 is combined with Figure 3a An example is given to illustrate the above, which will not be repeated here.

[0085] 2. Related data extraction:

[0086] The data table associated with the sample table often stores all the data. If all of the data is extracted to construct features, the performance of feature construction will be affected. At the same time, if machine resources are insufficient, feature construction may not even be completed. Therefore, it is necessary to extract the data that can be used to construct features from the data table associated with the sample table.

[0087] This embodiment is implemented by using a depth-first traversal algorithm. For details, please refer to Figure 2 In the embodiment shown, step 203 is combined with Figure 3b An example is given to illustrate the above, which will not be repeated here.

[0088] 3. Automatic judgment of data table fields:

[0089] In the feature mining process, it is necessary to determine the type of each data, which is mainly divided into ID (number) type variables, categorical variables, continuous variables, and date type variables. The feature mining methods for different types of variables are also different.

[0090] In this embodiment, both data field statistics and field name semantics are used as criteria for category judgment. For each field, field statistics analysis is first performed to calculate the field statistics of the field (for details, see Figure 4 The corresponding embodiment of the step 401 is described in detail), and then the semantic information of the field is determined according to the name of the field. The specific steps may be:

[0091] 1) Use manually marked field types, that is, multiple field types are manually pre-set;

[0092] 2) Perform word segmentation on the field names, count the number of times each word appears in the four categories, filter by the total number of times, and finally count the probability of each word appearing in each category.

[0093] If multiple words are hit at the same time, a weighted average is taken according to the frequency of word occurrence. After obtaining the statistical value of the field value and the probability of the field name belonging to each category, the type of the field is determined by combining the two pieces of information.

[0094] 4. Automatic feature generation:

[0095] Features need to be generated based on the type of data field and the relationship between the data and the sample table. For a sample, first obtain all the data associated with it, and then construct features according to the category of the data.

[0096] See also Figure 6 The schematic diagram shown:

[0097] The ID type variable will calculate the total number of IDs and the number of unique values ​​of the data associated with the sample. The total number here refers to the number of all IDs involved, and the number of unique values ​​here refers to the number of all IDs that appear only once.

[0098] Categorical variables calculate the total number of values ​​in each category in the data associated with the sample, the number of unique values, the proportion, the most frequently occurring category value, and the entropy value. The total number and the number of unique values ​​here are for each category. The total number refers to the total number of different values ​​under each type, and the number of unique values ​​refers to the number that appears only once under the category.

[0099] For continuous variables, the maximum, minimum, mean, mode, variance, median, and sum of the values ​​in the data associated with the sample are calculated;

[0100] Date type variables will calculate the year, month, date, weekday, weekend, and the difference between two time category variables in the sample associated data.

[0101] In addition to the feature mining method based on single data, we will also use statistics on the value of a certain data under the category feature value for some more concerned categorical variables, such as different types of insurance in the insurance industry and different commodity categories in the e-commerce industry. For example, we can count the total amount of digital product type orders of a certain user, the largest amount of digital product orders, the number of digital product orders, and other features.

[0102] For ID type variables, two types of features will be constructed. The first type is to verify the consistency of IDs in different links. For example, in the insurance anti-fraud scenario, the mobile phone number corresponding to the insurance policy, the mobile phone number of the reporter, the mobile phone number of the claim adjuster, the mobile phone number of the insured, and the mobile phone number of the insurance beneficiary will be obtained. The consistency of these mobile phone numbers will be extracted as features. The second type of feature is that different IDs of users, such as the user's mobile phone number, bank card number, ID number, device number, etc., will be associated with these IDs as features, such as the number of ID numbers corresponding to the user's mobile phone number, the number of device numbers corresponding to the mobile phone number, the number of bank card numbers corresponding to the mobile phone number, etc.

[0103] Further references Figure 7 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a data processing device, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0104] like Figure 7 As shown, the data processing device 700 of this embodiment may include: a core database table determination unit 701, a breadth-first traversal unit 702, and a depth-first traversal unit 703. The core database table determination unit 701 is configured to determine a core database table from the candidate database table; the breadth-first traversal unit 702 is configured to use the core database table as a traversal starting point and the primary key of the core database table as a starting associated element, and to determine the associated database table associated with the core database table layer by layer through a breadth-first traversal algorithm; the depth-first traversal unit 703 is configured to use the core database table as a traversal starting point and the primary key of the associated database table of the previous layer as a screening condition for screening associated data from the associated database of the next layer, and to extract the associated data through a depth-first traversal algorithm.

[0105] In this embodiment, in the data processing device 700, the specific processing of the core database table determination unit 701, the breadth-first traversal unit 702, and the depth-first traversal unit 703 and the technical effects thereof can be referred to in Figure 2 The relevant descriptions of steps 201 - 203 in the corresponding embodiment are not repeated here.

[0106] In some optional implementations of this embodiment, the core database table determining unit 701 may be further configured to:

[0107] Determine feature mining scenarios and feature mining requirements;

[0108] Determine a target database corresponding to the feature mining scenario, and determine a database table constituting the target database as the candidate database table;

[0109] Determine the starting information of feature mining according to the feature mining requirement;

[0110] The candidate database table using the starting information as the primary key is determined as the core database table.

[0111] In some optional implementations of this embodiment, the data processing device 700 may further include:

[0112] a data type determination unit, configured to determine the data type to which the associated data belongs;

[0113] The feature mining unit is configured to mine features of the associated data using a feature mining method corresponding to the data type.

[0114] In some optional implementations of this embodiment, the data type determination unit may include:

[0115] A field statistics and semantics determination subunit, configured to determine the field statistics and field semantics of the associated data;

[0116] A first probability determination subunit is configured to determine a first probability that the corresponding associated data belongs to a different data type according to the field statistical value;

[0117] A second probability determination subunit is configured to determine a second probability that the corresponding associated data belongs to a different data type according to the field semantics;

[0118] The target data type determination subunit is configured to determine the target data type to which the corresponding associated data belongs by combining the first probability and the second probability.

[0119] In some optional implementations of this embodiment, the data type determination unit may include:

[0120] The split word data type determination subunit is configured to, in response to the fact that the associated data can be split into multiple words, determine the data type of the associated data according to the data types to which the multiple words split from the associated data belong.

[0121] In some optional implementations of this embodiment, the split word data type determination subunit may be further configured to:

[0122] Counting the data type to which each word constituting the associated data belongs;

[0123] The data type with the largest statistical quantity is determined as the data type to which the associated data belongs.

[0124] In some optional implementations of this embodiment, the target data type determination subunit may be further configured to:

[0125] Determine a first weight corresponding to the first probability and a second weight corresponding to the second probability;

[0126] weighting the first probability using the first weight, and weighting the second probability using the second weight;

[0127] According to the probability sum of the weighted first probability and the second probability, a data type having a maximum probability sum is determined as the target data type.

[0128] In some optional implementations of this embodiment, the field statistics include at least one of the following:

[0129] Coverage rate, number of unique values, percentage of TopN values, entropy value, maximum value, mean value, variance, and value length; where N is a positive integer greater than 1.

[0130] In some optional implementations of this embodiment, the feature mining unit may be further configured to:

[0131] For the associated data of the number type, the total number of data associated with the associated data and the number of unique values ​​are used to construct features;

[0132] For categorical associated data, the number of different category values, the number of unique values, the proportion, the most frequently occurring category value, and the entropy value in the data associated with the associated data are used to construct features;

[0133] For continuous associated data, the maximum value, minimum value, mean, mode, variance, and median of the values ​​in the data associated with the associated data are used to construct features;

[0134] For the associated data of date type, features are constructed by using the year, month, date, weekday, weekend or not, and the difference between two time category variables in the data associated with the associated data.

[0135] This embodiment exists as a device embodiment corresponding to the above method embodiment. The data processing device provided by this embodiment, for each database table constituting the database, first determines the core database table as the core in each candidate database table, and then uses the core database table as the starting point of traversal and gradually uses it as the starting association element, and confirms the associated database table that is directly or indirectly associated with the core database table layer by layer through breadth-first traversal, and then uses the primary key of the associated database table of the previous layer as the screening condition for screening the associated data from the associated database of the next layer, and screens the target data from the associated database tables of each layer through depth-first traversal. That is, by performing breadth-first traversal and depth-first traversal in sequence, the target data that can be used to generate data features can be automatically mined from the database, which is more convenient and efficient.

[0136] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can implement the data processing method described in any of the above embodiments when executing.

[0137] According to an embodiment of the present disclosure, the present disclosure further provides a readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to implement the data processing method described in any of the above embodiments when executed.

[0138] According to an embodiment of the present disclosure, the present disclosure further provides a computer program product, which can implement the data processing method described in any of the above embodiments when executed by a processor.

[0139] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0140] like Figure 8As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0141] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0142] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as data processing methods. For example, in some embodiments, the data processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the data processing method in any other appropriate manner (e.g., by means of firmware).

[0143] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0144] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0145] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0146] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0147] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0148] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.

[0149] According to the technical solution of the embodiment of the present disclosure, for each database table constituting the database, first determine the core database table as the core in each candidate database table, then use the core database table as the starting point of traversal and gradually use it as the starting associated element, and confirm the associated database table that is directly or indirectly associated with the core database table layer by layer through breadth-first traversal, and then use the primary key of the associated database table of the previous layer as the screening condition for screening the associated data from the associated database of the next layer, and screen the target data from the associated database tables of each layer through depth-first traversal. That is, by performing breadth-first traversal and depth-first traversal in sequence, the target data that can be used to generate data features can be automatically mined from the database, which is more convenient and efficient.

[0150] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0151] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A data processing method, comprising: Determine the core database table among the candidate database tables; Taking the core database table as the traversal starting point and the primary key of the core database table as the starting association element, determining the associated database table associated with the core database table layer by layer through a breadth-first traversal algorithm; Taking the core database table as the traversal starting point and the primary key of the associated database table of the previous layer as the screening condition for screening associated data from the associated database of the next layer, extracting the associated data through a depth-first traversal algorithm; Determine the field statistics and field semantics of the associated data; determine a first probability that the corresponding associated data belongs to a different data type according to the field statistics; determine a second probability that the corresponding associated data belongs to a different data type according to the field semantics; and determine the data type to which the corresponding associated data belongs by combining the first probability and the second probability; The associated data is subjected to feature mining using a feature mining method corresponding to the data type to which it belongs.

2. The method according to claim 1, wherein: The step of determining the core database table from the candidate database tables includes: Determine feature mining scenarios and feature mining requirements; Determine a target database corresponding to the feature mining scenario, and determine a database table constituting the target database as the candidate database table; Determine the starting information of feature mining according to the feature mining requirement; The candidate database table using the starting information as the primary key is determined as the core database table.

3. The method according to claim 1, further comprising: In response to the associated data being divisible into a plurality of words, the data type to which the associated data belongs is determined according to the data types to which the plurality of words divided from the associated data belong.

4. The method according to claim 3, wherein: The determining the data type of the associated data according to the data types of the multiple words split out of the associated data includes: Counting the data type to which each word constituting the associated data belongs; The data type with the largest statistical quantity is determined as the data type to which the associated data belongs.

5. The method according to claim 1, wherein: The step of combining the first probability and the second probability to determine the data type to which the corresponding associated data belongs includes: Determine a first weight corresponding to the first probability and a second weight corresponding to the second probability; weighting the first probability using the first weight, and weighting the second probability using the second weight; According to the probability sum of the weighted first probability and the second probability, the data type with the maximum probability sum is determined as the data type to which the corresponding associated data belongs.

6. The method according to claim 1, wherein: The field statistics include at least one of the following: Coverage rate, number of unique values, percentage of TopN values, entropy value, maximum value, mean value, variance, and value length; where N is a positive integer greater than 1.

7. The method according to any one of claims 1 to 6, wherein: The mining of features of the associated data using a feature mining method corresponding to the data type to which it belongs includes: For the associated data of the number type, the total number of data associated with the associated data and the number of unique values ​​are used to construct features; For categorical associated data, the number of different category values, the number of unique values, the proportion, the most frequently occurring category value, and the entropy value in the data associated with the associated data are used to construct features; For continuous associated data, the maximum value, minimum value, mean, mode, variance, and median of the values ​​in the data associated with the associated data are used to construct features; For the associated data of date type, features are constructed by using the year, month, date, weekday, weekend or not, and the difference between two time category variables in the data associated with the associated data.

8. A data processing device, comprising: A core database table determination unit, configured to determine a core database table from among the candidate database tables; A breadth-first traversal unit is configured to use the core database table as a traversal starting point and the primary key of the core database table as a starting association element, and determine the associated database table associated with the core database table layer by layer through a breadth-first traversal algorithm; A depth-first traversal unit is configured to use the core database table as a traversal starting point, use the primary key of the associated database table of the previous layer as a screening condition for screening associated data from the associated database of the next layer, and extract the associated data through a depth-first traversal algorithm; a data type determination unit configured to determine field statistics and field semantics of the associated data; determine a first probability that the corresponding associated data belongs to a different data type according to the field statistics; determine a second probability that the corresponding associated data belongs to a different data type according to the field semantics; and determine the data type to which the corresponding associated data belongs by combining the first probability and the second probability; The feature mining unit is configured to mine features of the associated data using a feature mining method corresponding to the data type.

9. The device according to claim 8, wherein: The core database table determination unit is further configured to: Determine feature mining scenarios and feature mining requirements; Determine a target database corresponding to the feature mining scenario, and determine a database table constituting the target database as the candidate database table; Determine the starting information of feature mining according to the feature mining requirement; The candidate database table using the starting information as the primary key is determined as the core database table.

10. The device according to claim 8, wherein: The data type determination unit further includes: The split word data type determination subunit is configured to, in response to the fact that the associated data can be split into multiple words, determine the data type of the associated data according to the data types to which the multiple words split from the associated data belong.

11. The device according to claim 10, wherein: The split word data type determination subunit is further configured to: Counting the data type to which each word constituting the associated data belongs; The data type with the largest statistical quantity is determined as the data type to which the associated data belongs.

12. The device according to claim 8, wherein: The data type determination unit includes a target data type determination subunit configured to determine the data type to which the corresponding associated data belongs by combining the first probability and the second probability, and the target data type determination subunit is further configured to: Determine a first weight corresponding to the first probability and a second weight corresponding to the second probability; weighting the first probability using the first weight, and weighting the second probability using the second weight; According to the probability sum of the weighted first probability and the second probability, a data type having a maximum probability sum is determined as the target data type.

13. The device according to claim 8, wherein: The field statistics include at least one of the following: Coverage rate, number of unique values, percentage of TopN values, entropy value, maximum value, mean value, variance, and value length; where N is a positive integer greater than 1.

14. The device according to any one of claims 8 to 13, wherein: The feature mining unit is further configured to: For the associated data of the number type, the total number of data associated with the associated data and the number of unique values ​​are used to construct features; For categorical associated data, the number of different category values, the number of unique values, the proportion, the most frequently occurring category value, and the entropy value in the data associated with the associated data are used to construct features; For continuous associated data, the maximum value, minimum value, mean, mode, variance, and median of the values ​​in the data associated with the associated data are used to construct features; For the associated data of date type, features are constructed by using the year, month, date, weekday, weekend or not, and the difference between two time category variables in the data associated with the associated data.

15. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the data processing method according to any one of claims 1 to 7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the data processing method according to any one of claims 1 to 7.

17. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the steps of the data processing method according to any one of claims 1 to 7.

Images