Data processing method, apparatus, and electronic device

Abstract

This application discloses a data processing method, apparatus, and electronic device. Relating to the field of big data, the method includes: acquiring multiple data tables in a target database, acquiring fields located in the multiple data tables, obtaining M fields; acquiring feature information of the M fields, clustering the M fields based on the feature information, obtaining multiple target field clusters; acquiring the number of identical fields in each target field cluster, obtaining a count array for each target field cluster; determining a baseline field in each target field cluster based on the count array for each target field cluster, and updating non-baseline fields in the target database that are located in the same field cluster as the baseline field, obtaining an updated target database. This application solves the problem in related technologies where inconsistencies exist in data fields related to the same business content in the database, leading to low accuracy in processing business data.

Description

Technical Field

[0001] This application relates to the field of big data, and more specifically, to a data processing method, apparatus, and electronic device. Background Technology

[0002] Currently, financial institutions have a large number of applications, such as mobile banking and credit card application platforms. Each application contains a large amount of business data, which is uniformly stored in the database corresponding to the application and recorded in the database in the form of tables and fields.

[0003] However, as business data continues to increase, there may be situations where data fields for the same business content are located in different tables, and the data fields recorded in different tables are inconsistent. For example, user A handles business A, and user B also handles business A, but when entering the business information, user A enters the name field of business A in uppercase, while user B enters the name field of business A in lowercase. Consequently, when the business data needs to be processed, the inconsistent data fields will lead to errors in the data processing results.

[0004] There is currently no effective solution to the problem of inconsistent data fields for the same business content in databases, which leads to low accuracy in processing business data. Summary of the Invention

[0005] The main objective of this application is to provide a data processing method, apparatus, and electronic device to solve the problem in the related art where inconsistencies exist in data fields for the same business content in the database, resulting in low accuracy in processing business data.

[0006] To achieve the above objectives, according to one aspect of this application, a data processing method is provided. The method includes: obtaining multiple data tables in a target database, and obtaining fields located in the multiple data tables to obtain M fields, where M is a positive integer; obtaining feature information of the M fields, and clustering the M fields according to the feature information to obtain multiple target field clusters; obtaining the number of identical fields in each target field cluster to obtain a count array for each target field cluster, wherein the count array includes the number of identical fields in each group; determining a baseline field in each target field cluster based on the count array for each target field cluster, and updating non-baseline fields in the target database that are located in the same field cluster as the baseline field, to obtain an updated target database.

[0007] Optionally, obtaining the feature information of M fields includes: obtaining the indicator features required to perform clustering operations, and determining the feature values ​​under each indicator feature based on the field information of each field; encoding the feature values ​​under each field to obtain the feature information of each field.

[0008] Optionally, clustering the M fields based on their feature information to obtain multiple target field clusters includes: using a density-based clustering algorithm to cluster the M fields based on their feature information to obtain N first field clusters, and identifying the first candidate field cluster with the largest number of fields among the N first field clusters, where N is a positive integer; setting the K value of the initial clustering model based on the value of N and the first candidate field clusters to obtain a target clustering model, and using the target clustering model to cluster the M fields based on their feature information to obtain H second field clusters, and identifying the second field cluster with the largest number of fields among the H second field clusters. Second candidate field cluster; determine whether the second candidate field cluster meets the preset iteration conditions; if the second candidate field cluster meets the preset iteration conditions, determine multiple fields in the second candidate field cluster as the updated M fields, and iteratively execute the step of using density clustering algorithm to cluster the M fields according to the feature information of the M fields to obtain N first field clusters, until the second candidate field cluster meets the preset iteration conditions; if the second candidate field cluster does not meet the preset iteration conditions, determine the unprocessed second field cluster obtained after each iteration operation as the target field cluster, and obtain multiple target field clusters.

[0009] Optionally, determining whether the second candidate field cluster meets the preset iteration conditions includes: obtaining the number of fields contained in the second candidate field cluster to obtain a first number, and determining whether the first number is greater than a preset threshold; if the first number is greater than the preset threshold, determining that the second candidate field cluster meets the preset iteration conditions, and if the first number is less than or equal to the preset threshold, determining that the second candidate field cluster does not meet the preset iteration conditions; or, obtaining the number of iterations of the iteration operation for obtaining the second candidate field cluster, and determining whether the number of iterations is greater than or equal to a preset number of iterations; if the number of iterations is less than the preset number of iterations, determining that the second candidate field cluster meets the preset iteration conditions, and if the number of iterations is greater than or equal to the preset number of iterations, determining that the second candidate field cluster does not meet the preset iteration conditions.

[0010] Optionally, setting the K value of the initial clustering model based on the value of N and the first candidate field cluster includes: obtaining the number of fields in the first candidate field cluster to obtain a second number; dividing the second number by a preset value to obtain the target change, and adding the target change to the value of N to obtain the K value of the initial clustering model.

[0011] Optionally, determining the base field in each target field cluster based on the quantity array of each target field cluster includes: obtaining the maximum quantity value in the quantity array and determining the field corresponding to the maximum quantity value as the base field in the field cluster.

[0012] Optionally, updating the non-reference fields in the target database that are in the same field cluster as the reference fields to obtain the updated target database includes: for any target field cluster, determining the position information of each non-reference field under the target field cluster in the target database; and replacing each non-reference field with the reference field under the target field cluster according to the position information to obtain the updated target database.

[0013] To achieve the above objectives, according to another aspect of this application, a data processing apparatus is provided. The apparatus includes: a first acquisition unit, configured to acquire multiple data tables in a target database and acquire fields located in the multiple data tables to obtain M fields, where M is a positive integer; a clustering unit, configured to acquire feature information of the M fields and cluster the M fields according to the feature information to obtain multiple target field clusters; a second acquisition unit, configured to acquire the number of identical fields in each target field cluster to obtain a count array for each target field cluster, wherein the count array includes the number of identical fields in each group; and an update unit, configured to determine a reference field in each target field cluster based on the count array for each target field cluster and update non-reference fields in the target database that are located in the same field cluster as the reference field, to obtain an updated target database.

[0014] Optionally, the clustering unit includes: a first acquisition module, used to acquire the indicator features required to perform the clustering operation, and determine the feature values ​​under each indicator feature according to the field information of each field; and a first processing module, used to encode the feature values ​​under each field to obtain the feature information of each field.

[0015] Optionally, the clustering unit includes: a second processing module, used to cluster the M fields using a density clustering algorithm based on the feature information of the M fields, to obtain N first field clusters, and to obtain the first candidate field cluster with the largest number of fields among the N first field clusters, where N is a positive integer; a third processing module, used to set the K value of the initial clustering model based on the value of N and the first candidate field clusters, to obtain a target clustering model, and to use the target clustering model to cluster the M fields based on the feature information of the M fields, to obtain H second field clusters, and to obtain the second candidate field cluster with the largest number of fields among the H second field clusters; and a judgment module, used to judge... The first determining module is used to determine whether the second candidate field cluster meets the preset iteration conditions. If the second candidate field cluster meets the preset iteration conditions, it determines multiple fields in the second candidate field cluster as the updated M fields, and iteratively executes the step of using density clustering algorithm to cluster the M fields according to the feature information of the M fields to obtain N first field clusters, until the second candidate field cluster meets the preset iteration conditions. The second determining module is used to determine the unprocessed second field cluster obtained after each iteration operation as the target field cluster if the second candidate field cluster does not meet the preset iteration conditions, thereby obtaining multiple target field clusters.

[0016] Optionally, the judgment module includes: a first judgment submodule, used to obtain the number of fields contained in the second candidate field cluster, obtain a first number, and determine whether the first number is greater than a preset threshold; a first determination submodule, used to determine that the second candidate field cluster meets the preset iteration condition if the first number is greater than the preset threshold, and determine that the second candidate field cluster does not meet the preset iteration condition if the first number is less than or equal to the preset threshold; or, a second judgment submodule, used to obtain the number of iterations of the iteration operation to obtain the second candidate field cluster, and determine whether the number of iterations is greater than or equal to a preset number of iterations; a second determination submodule, used to determine that the second candidate field cluster meets the preset iteration condition if the number of iterations is less than the preset number of iterations, and determine that the second candidate field cluster does not meet the preset iteration condition if the number of iterations is greater than or equal to the preset number of iterations.

[0017] Optionally, the third processing module includes: an acquisition submodule, used to acquire the number of fields in the first candidate field cluster to obtain a second number; and a calculation submodule, used to divide the second number by a preset value to obtain a target change, and add the target change to the value of N to obtain the K value of the initial clustering model.

[0018] Optionally, the update unit includes: a second acquisition module, used to acquire the maximum quantity value in the quantity array and determine the field corresponding to the maximum quantity value as the base field in the field cluster.

[0019] Optionally, the update unit includes: a third determining module, used to determine the position information of each non-baseline field under the target field cluster in the target database for any target field cluster; and a replacement module, used to replace each non-baseline field with a baseline field under the target field cluster according to the position information to obtain the updated target database.

[0020] To achieve the above objectives, according to another aspect of this application, an electronic device is provided, the electronic device including a memory storing an executable program; and a processor for running the program, wherein the program executes the above-described data processing method during runtime.

[0021] To achieve the above objectives, according to another aspect of this application, a computer program product is provided, including computer instructions that, when executed by a processor, implement the steps of the data processing method described above.

[0022] In this embodiment, the method involves obtaining multiple data tables in the target database and acquiring the fields located in these tables, resulting in M ​​fields, where M is a positive integer. Feature information of the M fields is then acquired, and the M fields are clustered based on this feature information to obtain multiple target field clusters. The number of identical fields in each target field cluster is obtained, resulting in a count array for each target field cluster, where the count array includes the number of identical fields in each group. A baseline field is determined for each target field cluster based on the count array, and non-baseline fields in the target database that are located in the same field cluster as the baseline field are updated, resulting in an updated target database. By clustering the fields in each data table of the database, identical fields with the same business content are grouped into a cluster. A baseline field is determined based on the number of each field within the cluster, and then the baseline field is used to replace other non-baseline fields. This achieves the goal of maintaining consistency among fields with the same business content in the database, thereby improving the accuracy of processing business data using the data table content stored in the database. This solves the technical problem in related technologies where inconsistent data fields with the same business content in the database lead to low accuracy in processing business data. Attached Figure Description

[0023] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0024] Figure 1 A hardware structure block diagram of a computer terminal for implementing a data processing method is shown.

[0025] Figure 2This is a flowchart of the data processing method provided according to Embodiment 1 of this application;

[0026] Figure 3 This is a schematic diagram of a data processing apparatus provided according to Embodiment 2 of this application;

[0027] Figure 4 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0028] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0029] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] It should be noted that the data processing methods, apparatuses, and electronic devices defined in this disclosure can be used in the field of big data, or in any field other than big data. The application fields of the data processing methods, apparatuses, and electronic devices defined in this disclosure are not limited.

[0032] It should be noted that all information, user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, and displayed data) used in this application are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with the relevant laws, regulations, and standards of the relevant regions, have taken necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse use. If the user chooses to refuse, the process will proceed to the expert decision-making process. For example, this system has an interface with relevant users or organizations. Before obtaining relevant information, a request to obtain the information needs to be sent to the aforementioned user or organization through the interface. After receiving consent from the aforementioned user or organization, the relevant information is obtained. Users can view the purpose of data use in real time through the authorization interface and have the right to withdraw authorization or delete data at any time. After the authorization is withdrawn, the system will terminate the relevant data processing within 24 hours.

[0033] The embodiments or examples disclosed herein are not exhaustive, but merely illustrative of some embodiments or examples, and are not intended to limit the scope of protection of this disclosure. Unless otherwise specified, each step in a particular embodiment or example can be implemented as an independent embodiment, and the steps can be arbitrarily combined. For example, a solution after removing some steps in a particular embodiment or example can also be implemented as an independent embodiment, and the order of the steps in a particular embodiment or example can be arbitrarily interchanged. Furthermore, optional methods or examples in a particular embodiment or example can be arbitrarily combined; moreover, embodiments or examples can be arbitrarily combined. For example, some or all steps of different embodiments or examples can be arbitrarily combined, and a particular embodiment or example can be arbitrarily combined with optional methods or examples of other embodiments or examples.

[0034] For ease of description, the following explains some of the nouns or terms used in the embodiments of this application:

[0035] DBSCAN algorithm: Density-Based Spatial Clustering of Applications with Noise.

[0036] K-means clustering algorithm: K-means clustering algorithm.

[0037] Example 1

[0038] According to an embodiment of this application, an embodiment of a data processing method is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0039] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware structure block diagram of a computer terminal for implementing a data processing method is shown. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, processing devices such as microprocessors or programmable logic devices), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface, a universal serial bus port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0040] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0041] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the data processing method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the aforementioned data processing method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0042] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0043] The display may be, for example, a touchscreen LCD display that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0044] Under the aforementioned operating environment, this application provides the following: Figure 2 The data processing method shown. Figure 2 This is a flowchart of the data processing method provided in Embodiment 1 of this application, as follows: Figure 2 As shown, the method includes:

[0045] Step S201: Obtain multiple data tables in the target database, and obtain the fields located in the multiple data tables to obtain M fields, where M is a positive integer.

[0046] It should be noted that the execution entity in this embodiment can be a data processing system. This system can identify which fields in the database correspond to the same business content by identifying each field, and automatically unify them in the database, so that the content of fields that should be the same in the database remains consistent, thereby improving the consistency of the data fields stored in the database.

[0047] For example, the target database contains hundreds or thousands of different business data tables, each representing a set of records for one or more specific business operations or entities. The system automatically extracts and records field information from all tables in the database through deep metadata scanning, including but not limited to field names, data types, and lengths. This process covers the entire table structure, ensuring that every field is captured and evaluated, ultimately resulting in M ​​fields.

[0048] For example, retrieve the User_Name field from Table 1 and the username field from Table 2.

[0049] Step S202: Obtain feature information of M fields, and cluster the M fields according to the feature information of M fields to obtain multiple target field clusters.

[0050] It should be noted that the feature information is the information extracted after performing multi-dimensional analysis on each field (such as data type, length, constraints, position, contextual semantics, etc.), and is used as input for the clustering algorithm.

[0051] For example, after obtaining the complete list of fields, the next step is to perform in-depth feature extraction and analysis on each field. This process involves comprehensively considering the structured features (such as data type and constraints), positional / contextual features (such as the field's position in the table and the impact of the table name on the field's meaning), and semantic features (natural language processing analysis of field names and annotation text) of each field to construct a comprehensive feature vector space.

[0052] Furthermore, cluster analysis can be performed on the M fields based on feature information to group fields with high similarity into one cluster, thereby obtaining multiple target field clusters.

[0053] For example, the User_Name field in Table 1 and the username field in Table 2 are identical in terms of field name, position, and semantic features, differing only in structure. Therefore, after clustering, these two fields will be clustered into one cluster, thus obtaining a target field cluster.

[0054] Step S203: Obtain the number of identical fields in each target field cluster to obtain a count array for each target field cluster, wherein the count array includes the number of identical fields in each group.

[0055] For example, after completing the initial field clustering, it can be determined that the fields in each cluster are the same fields. At this point, it is necessary to determine the same fields in the target field cluster, so as to further group the characters in a cluster and determine the number of fields in each group to obtain a number array, thereby determining the field representation method that occupies a large number in the entire database.

[0056] For example, if a table contains three fields: “User_Name”, “username”, and “UserName”, and the number of “User_Name” fields is 1000, the number of “username” fields is 26, and the number of “UserName” fields is 10, then the count array can be determined as [1000, 26, 10].

[0057] Step S204: Determine the base field in each target field cluster based on the number array of each target field cluster, and update the non-base field in the target database that is in the same field cluster as the base field, to obtain the updated target database.

[0058] For example, after obtaining the quantity array, the field with the most quantity in each cluster can be determined based on the quantity array, and this field can be determined as the benchmark field. Then, the non-benchmark fields in each table of the database can be replaced with the benchmark field, thereby completing the field unification operation of the database and improving the field consistency in the database.

[0059] For example, based on the quantity array [1000, 26, 10], we can determine "User_Name" as the base field. At this time, we can modify both "username" and "UserName" in the database to "User_Name" so that the "User_Name" field in the database adopts the same representation.

[0060] The data processing method provided in this application embodiment achieves consistency among fields of the same business content in the database by clustering fields in each data table, thereby improving the accuracy of processing business data using the data table content stored in the database. This solves the technical problem in related technologies where inconsistent data fields of the same business content in the database lead to low accuracy in processing business data.

[0061] To improve the accuracy of clustering operations, optionally, in the data processing method provided in this application embodiment, obtaining the feature information of M fields includes: obtaining the indicator features required to perform clustering operations, and determining the feature values ​​under each indicator feature based on the field information of each field; encoding the feature values ​​under each field to obtain the feature information of each field.

[0062] It should be noted that indicator features refer to the key dimensions used to perform clustering operations, including but not limited to the structural features, semantic features, and contextual features of a field. Feature values ​​represent the specific performance or attribute values ​​of a field under the indicator features. Field information may include field name, data type, length, constraints, table name, location information, etc., which are the basis for determining feature values. Encoding processing involves converting the various feature values ​​of the field into numerical or vector representations that can be processed by a computer.

[0063] For example, when determining feature information, it is first necessary to determine the multidimensional indicator features used for clustering. These indicator features encompass structured features, contextual features, and semantic features, thus providing a comprehensive description of the field. Structured features include hard technical attributes such as data type encoding, field length, and whether it is a primary key / foreign key / not null. Contextual features involve the field's relative position in the table and the characteristics of the table it belongs to. Semantic features require the use of natural language processing techniques to extract business meaning and conceptual associations from field names and comments. For instance, the field "CustomerID" might be represented as an "integer" in terms of data type encoding, "10" in terms of length, and "yes" in terms of whether it is a primary key. Semantically, word embedding techniques might yield a multidimensional vector representing "customer ID number."

[0064] Furthermore, the system calculates the feature value under each defined indicator feature for the field information in the M fields. It needs to parse all relevant metadata and content of each field, including but not limited to field name, data type, length, constraints (such as whether it is a primary key, whether null values ​​are allowed), field position information, and possible comment text, so as to determine the specific indicator value under each indicator feature based on the above content.

[0065] For example, for the field "CustomerID", the system will determine its feature value as "integer code" under the structured feature "data type encoding", "10" under the "length" feature, "yes" under the "primary key" feature, and "no" under the "nullable" feature.

[0066] It should be noted that after obtaining the feature values, they need to be encoded to convert them into numerical or vector representations so that further clustering algorithms can process them. Encoding strategies may include: converting categorical features (such as data types) into integer values ​​or one-hot encoded vectors; standardizing or normalizing continuous features (such as field length); and using word embedding models to generate multidimensional vector representations for semantic features. For example, the data type "integer" in the field "CustomerID" may be encoded as the number 1 (if an integer encoding strategy is used), or as a [1, 0, 0] vector (if one-hot encoding is used to represent "integer", "string", and "date" types).

[0067] After the above feature extraction and encoding process, a multi-dimensional feature vector for each field is obtained. This feature vector integrates the description of the field in terms of structure, context, and semantic features, representing the comprehensive attributes of the field. For example, the feature vector of the field "CustomerID" can be represented as [integer encoding, length=10, primary key=yes, not null=no, positional encoding, semantic vector], where "positional encoding" and "semantic vector" are numerical or vector representations obtained based on a specific encoding strategy.

[0068] This embodiment improves the accuracy of subsequent clustering operations by constructing a multi-dimensional feature vector for each field based on the field information and indicator characteristics.

[0069] Optionally, in the data processing method provided in this application embodiment, clustering the M fields according to their feature information to obtain multiple target field clusters includes: using a density clustering algorithm to cluster the M fields according to their feature information to obtain N first field clusters, and obtaining the first candidate field cluster with the largest number of fields among the N first field clusters, where N is a positive integer; setting the K value of the initial clustering model according to the value of N and the first candidate field clusters to obtain a target clustering model, and using the target clustering model to cluster the M fields according to their feature information to obtain H second field clusters, and obtaining H second field clusters. The second candidate field cluster with the largest number of fields in the segment cluster is identified. The process involves determining whether the second candidate field cluster meets the preset iteration conditions. If the second candidate field cluster meets the preset iteration conditions, multiple fields in the second candidate field cluster are identified as the updated M fields, and the density clustering algorithm is iteratively executed to cluster the M fields based on their feature information to obtain N first field clusters, until the second candidate field cluster meets the preset iteration conditions. If the second candidate field cluster does not meet the preset iteration conditions, the unprocessed second field cluster obtained after each iteration is identified as the target field cluster, resulting in multiple target field clusters.

[0070] It should be noted that density clustering algorithms are spatial clustering algorithms based on the density distribution of data points, such as DBSCAN or its improved versions, used to identify clusters of arbitrary shapes and densities.

[0071] For example, when clustering feature information, to improve the accuracy and detail of the clustering results, the system first uses an improved DBSCAN algorithm to process the feature vectors of the M fields generated in the feature extraction stage. Based on similarity metrics between fields (such as cosine similarity or a custom distance metric), the DBSCAN algorithm can discover clusters of arbitrary shapes and uneven densities, effectively identifying "natural clustering" between fields, thus identifying regions with high data density and forming N first-field clusters. This stage of clustering does not set a fixed cluster number target, but allows the algorithm to automatically determine the number and shape of clusters based on the density of the natural data distribution, which helps to identify naturally formed field groups in complex, non-uniformly distributed datasets.

[0072] After the initial clustering, the system will automatically select the first candidate field cluster with the largest number of fields. The fineness of the field division will be determined based on the number of fields contained in the first candidate field cluster. The more fields contained in the first candidate field cluster, the worse the fineness of the division, and further clustering operations are needed.

[0073] Furthermore, to improve the accuracy of cluster division, the system dynamically adjusts the K value of the K-means clustering algorithm based on the number of N clusters obtained from the initial clustering and the number of fields contained in the first candidate field cluster, making the K value more accurate, thereby obtaining the target clustering model. The target clustering model is then used to divide the fields, so that M fields can be divided according to the specified division precision, resulting in a more accurate clustering result, namely the second field cluster.

[0074] Furthermore, after obtaining the clustering results, it is necessary to obtain the second candidate field cluster with the largest number of fields among the H second field clusters, and determine whether to continue with more refined partitioning operations based on the second candidate field cluster. If the second candidate field cluster meets the preset iteration conditions, it is still necessary to perform iterative operations. At this time, it is necessary to re-execute the above operation with multiple fields in the second candidate field cluster as M fields, or to use the fields in the second field cluster with a field number greater than the preset field number threshold as the above M fields, and execute the above clustering operation in parallel until the number of fields in each field cluster obtained by the final partition is less than the preset field number threshold, or a certain number of iterations is reached, and it is no longer possible to further improve the partitioning accuracy, thereby improving the accuracy of the results of each cluster.

[0075] Finally, after completing the above iterative operations, starting from the first clustering, the field clusters that have not been further processed in each clustering result will be statistically analyzed to obtain multiple target field clusters, thereby completing the target field cluster partitioning operation.

[0076] This embodiment achieves the technical effect of improving the accuracy and precision of field clustering by iteratively performing field clustering operations.

[0077] To improve the efficiency of clustering operations, optionally, in the data processing method provided in this application embodiment, determining whether the second candidate field cluster meets the preset iteration conditions includes: obtaining the number of fields contained in the second candidate field cluster to obtain a first number, and determining whether the first number is greater than a preset threshold; if the first number is greater than the preset threshold, determining that the second candidate field cluster meets the preset iteration conditions, and if the first number is less than or equal to the preset threshold, determining that the second candidate field cluster does not meet the preset iteration conditions; or, obtaining the number of iterations of the iteration operation of the second candidate field cluster, and determining whether the number of iterations is greater than or equal to a preset number of iterations; if the number of iterations is less than the preset number of iterations, determining that the second candidate field cluster meets the preset iteration conditions, and if the number of iterations is greater than or equal to the preset number of iterations, determining that the second candidate field cluster does not meet the preset iteration conditions.

[0078] For example, when determining whether the second candidate field cluster meets the preset iteration conditions, it can be determined in two ways. First, the number of fields contained in the second candidate field cluster can be obtained to get the first number, and it can be determined whether the first number is greater than the preset threshold. If it is greater than the threshold, it indicates that there may be a situation where the clustering result has low precision. At this time, it can be determined that the preset iteration conditions are met, and the data in the second candidate field cluster can be treated as M fields to perform the above clustering operation again, thereby further clustering and dividing the fields in the second candidate field cluster with more fields, thereby improving the precision of the clustering result.

[0079] For example, the number of iterations can be used to determine whether the second candidate field cluster meets the preset iteration conditions. In some cases, there may be a large number of fields in a field cluster, such as a field with a high frequency of use and a wide range of use. In this case, no matter how many iterations of clustering operations are performed, further division cannot be performed. Therefore, the number of iterations can be used to determine whether the clustering operation can be completed.

[0080] If the number of iterations is less than the preset number of iterations, the characterization needs to continue with further iterations. If the number of iterations is greater than or equal to the preset number of iterations, it is determined that no further iterations are needed. This improves the granularity of the clustering results, reduces invalid iterations, and increases the efficiency of clustering operations.

[0081] It should be noted that when iterating over multiple field clusters simultaneously, the iteration operation can only be stopped when the number of fields in each field cluster is less than a preset threshold, or when the required number of iterations is reached.

[0082] This embodiment determines whether the clustering operation is complete by judging whether the field clusters meet the preset iteration conditions, thereby improving the granularity of the clustering results and the execution efficiency of the clustering operation.

[0083] Optionally, in the data processing method provided in this application embodiment, setting the K value of the initial clustering model according to the value of N and the first candidate field cluster includes: obtaining the number of fields in the first candidate field cluster to obtain a second number; dividing the second number by a preset value to obtain a target change, and adding the target change to the value of N to obtain the K value of the initial clustering model.

[0084] For example, when determining the K value, after completing the initial density clustering operation and determining the first candidate field cluster, the total number of fields within the first candidate field cluster, i.e., the second quantity, is counted. This second quantity is then compared with a preset value to calculate the target change. The selection of the preset value reflects the ideal expectation for the final cluster size and is the basis for the system to dynamically adjust the K value.

[0085] Furthermore, the target change is added to the value of N to calculate the K value of the initial clustering model. This K value will be used in the next step of the K-means clustering algorithm. The value of N reflects the total number of clusters after the initial clustering. Adding the target change, the calculation of the K value aims to achieve a more refined and accurate division of the M fields, while taking into account the global distribution of the clusters. For example, if the value of N is 100 (i.e., the initial clustering yields 100 clusters for the first field) and the target change is 4, then the K value of the initial clustering model is 104(100+4). This K value setting implies that when clustering the M fields, the system expects to obtain more than 100 clusters, thereby improving the accuracy of the clustering results.

[0086] This embodiment determines the K value based on the number of fields in the first candidate field cluster and the value of N, thereby improving the accuracy and precision of the clustering results when performing clustering operations based on the K value.

[0087] Optionally, in the data processing method provided in this application embodiment, determining the base field in each target field cluster based on the quantity array of each target field cluster includes: obtaining the maximum quantity value in the quantity array and determining the field corresponding to the maximum quantity value as the base field in the field cluster.

[0088] For example, in order to improve the accuracy of the updated database, the field corresponding to the maximum quantity value in the quantity array can be determined as the base field in the field cluster. This makes the field with a large application range and quantity in the database the base field, thereby reducing the number of fields to be updated in the database. Furthermore, the fewer fields are more likely to be erroneous fields, so the erroneous fields in the database can be updated to the correct fields, thus enabling a more accurate database update operation.

[0089] For example, if the system identifies "Customer_ID" as the field corresponding to the maximum quantity value in a cluster, then "Customer_ID" will be officially determined as the base field for this cluster. When updating database fields, the format and naming rules of "Customer_ID" will be used as a reference.

[0090] This embodiment determines the baseline field based on the number of each field in the cluster, thereby improving the accuracy of updating the database based on the baseline field.

[0091] Optionally, in the data processing method provided in this application embodiment, updating the non-reference fields in the target database that are located in the same field cluster as the reference fields to obtain the updated target database includes: for any target field cluster, determining the position information of each non-reference field under the target field cluster in the target database; replacing each non-reference field with the reference field under the target field cluster according to the position information to obtain the updated target database.

[0092] For example, after determining the baseline field, it is necessary to determine the location information of other non-baseline fields in the same cluster as the baseline field in the database, and replace the non-baseline fields with the baseline field according to the location information, thereby completing the operation of updating erroneous or inconsistent fields in the database, so that the fields of the same objects in the database are consistent and the accuracy of the fields in the database is improved.

[0093] For example, if the database includes "User_Name", "username", and "UserName", and "User_Name" is determined as the base field, then it is necessary to determine the position of each "username" and "UserName" field in the database and change them all to "User_Name" to complete the database modification operation.

[0094] This embodiment achieves the technical effect of accurately updating the database by replacing non-reference fields with reference fields based on location information.

[0095] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0096] Example 2

[0097] This application also provides a data processing apparatus. It should be noted that the data processing apparatus of this application can be used to execute the data processing methods provided in the above embodiments. The data processing apparatus provided in this application will be described below.

[0098] According to an embodiment of this application, an apparatus for implementing the above-described data processing method is also provided. Figure 3 This is a schematic diagram of a data processing apparatus according to Embodiment 2 of this application, as shown below. Figure 3 As shown, the device includes:

[0099] The first acquisition unit 31 is used to acquire multiple data tables in the target database and acquire the fields located in the multiple data tables to obtain M fields, where M is a positive integer.

[0100] Clustering unit 32 is used to obtain feature information of M fields and cluster the M fields according to the feature information of M fields to obtain multiple target field clusters.

[0101] The second acquisition unit 33 is used to acquire the number of identical fields in each target field cluster to obtain a quantity array for each target field cluster, wherein the quantity array includes the number of identical fields in each group.

[0102] The update unit 34 is used to determine the base field in each target field cluster based on the number array of each target field cluster, and update the non-base field in the target database that is in the same field cluster as the base field, so as to obtain the updated target database.

[0103] The data processing apparatus provided in this application embodiment acquires multiple data tables in a target database through a first acquisition unit 31, and acquires fields located in the multiple data tables to obtain M fields, where M is a positive integer; a clustering unit 32 acquires feature information of the M fields, and clusters the M fields according to the feature information of the M fields to obtain multiple target field clusters; a second acquisition unit 33 acquires the number of identical fields in each target field cluster to obtain a count array for each target field cluster, wherein the count array includes the number of identical fields in each group; and an update unit 34 determines the base field in each target field cluster according to the count array for each target field cluster, and updates the non-base field in the target database that is located in the same field cluster as the base field, to obtain an updated target database. By clustering the fields in various data tables in the database, identical fields with the same business content are grouped together. A benchmark field is determined based on the number of each field in the cluster, and other non-benchmark fields are replaced using the benchmark field. This achieves the goal of keeping the fields with the same business content in the database consistent, thereby improving the accuracy of processing business data using the data table content stored in the database. This solves the technical problem in related technologies where inconsistencies in data fields with the same business content in the database lead to low accuracy in processing business data.

[0104] Optionally, in the data processing apparatus provided in this application embodiment, the clustering unit 32 includes: a first acquisition module, used to acquire the index features required to perform clustering operations, and determine the feature values ​​under each index feature according to the field information of each field; and a first processing module, used to encode the feature values ​​under each field to obtain the feature information of each field.

[0105] Optionally, in the data processing apparatus provided in this application embodiment, the clustering unit 32 includes: a second processing module, configured to use a density clustering algorithm to cluster the M fields according to the feature information of the M fields, to obtain N first field clusters, and to obtain the first candidate field cluster with the largest number of fields among the N first field clusters, where N is a positive integer; and a third processing module, configured to set the K value of the initial clustering model according to the value of N and the first candidate field clusters, to obtain a target clustering model, and to use the target clustering model to cluster the M fields according to the feature information of the M fields, to obtain H second field clusters, and to obtain the second candidate field cluster with the largest number of fields among the H second field clusters. The system comprises: a cluster determination module for determining whether the second candidate field cluster meets the preset iteration conditions; a first determination module for determining multiple fields in the second candidate field cluster as updated M fields when the second candidate field cluster meets the preset iteration conditions, and iteratively executing the step of using density clustering algorithm to cluster the M fields according to the feature information of the M fields to obtain N first field clusters, until the second candidate field cluster meets the preset iteration conditions; and a second determination module for determining the unprocessed second field cluster obtained after each iteration operation as the target field cluster when the second candidate field cluster does not meet the preset iteration conditions, thereby obtaining multiple target field clusters.

[0106] Optionally, in the data processing apparatus provided in this application embodiment, the judgment module includes: a first judgment submodule, configured to obtain the number of fields contained in the second candidate field cluster, obtain a first number, and determine whether the first number is greater than a preset threshold; a first determination submodule, configured to determine that the second candidate field cluster meets the preset iteration condition when the first number is greater than the preset threshold, and determine that the second candidate field cluster does not meet the preset iteration condition when the first number is less than or equal to the preset threshold; or, a second judgment submodule, configured to obtain the number of iterations of the iteration operation for obtaining the second candidate field cluster, and determine whether the number of iterations is greater than or equal to a preset number of iterations; a second determination submodule, configured to determine that the second candidate field cluster meets the preset iteration condition when the number of iterations is less than the preset number of iterations, and determine that the second candidate field cluster does not meet the preset iteration condition when the number of iterations is greater than or equal to the preset number of iterations.

[0107] Optionally, in the data processing apparatus provided in the embodiments of this application, the third processing module includes: an acquisition submodule, used to acquire the number of fields in the first candidate field cluster to obtain a second number; and a calculation submodule, used to divide the second number by a preset value to obtain a target change, and add the target change to the value of N to obtain the K value of the initial clustering model.

[0108] Optionally, in the data processing apparatus provided in this application embodiment, the update unit 34 includes: a second acquisition module, used to acquire the maximum quantity value in the quantity array and determine the field corresponding to the maximum quantity value as the base field in the field cluster.

[0109] Optionally, in the data processing apparatus provided in this application embodiment, the update unit 34 includes: a third determining module, used to determine the position information of each non-reference field under the target field cluster in the target database for any target field cluster; and a replacement module, used to replace each non-reference field with a reference field under the target field cluster according to the position information to obtain an updated target database.

[0110] It should be noted that the first acquisition unit 31, clustering unit 32, second acquisition unit 33, and update unit 34 mentioned above correspond to steps S201 to S204 in Embodiment 1. The instances and application scenarios implemented by each of the above units and the corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above modules can also be part of a device and can run in the computer terminal 10 provided in Embodiment 1.

[0111] Example 3

[0112] Embodiments of this application may provide an electronic device. Figure 4 This is a structural block diagram of an electronic device according to an embodiment of this application. Figure 4 As shown, the electronic device may include: one or more ( Figure 4 (Only one is shown) processor 1002, memory 1004, memory controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0113] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the above-described methods. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0114] Those skilled in the art will understand that Figure 4 The structure shown is for illustrative purposes only. Electronic devices can also be smartphones, tablets, handheld computers, mobile internet devices (MIDs), PADs, and other terminal devices. Figure 4 This does not limit the structure of the aforementioned electronic device. For example, electronic devices may also include components that are more... Figure 4 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 4 The different configurations shown.

[0115] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0116] Example 4

[0117] Embodiments of this application also provide a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the data processing method provided in Embodiment 1.

[0118] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0119] Embodiments of this application also provide a computer program product, which, when executed on a data processing device, is a program adapted to perform the steps of a data processing method.

[0120] Embodiments of this application also provide a computer-readable storage medium, which includes a stored executable program, wherein the executable program controls the device where the computer-readable storage medium is located to perform the above-described data processing method when it runs.

[0121] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0122] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0123] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0124] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0125] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0126] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0127] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, include: Obtain multiple data tables from the target database, and extract the fields located in the multiple data tables to obtain M fields, where M is a positive integer; Obtain the feature information of the M fields, and cluster the M fields according to the feature information of the M fields to obtain multiple target field clusters; The number of identical fields in each target field cluster is obtained to obtain a count array for each target field cluster, wherein the count array includes the number of identical fields in each group; The baseline field in each target field cluster is determined based on the number array of each target field cluster, and the non-baseline fields in the target database that are located in the same field cluster as the baseline field are updated to obtain the updated target database.

2. The method according to claim 1, characterized in that, Obtaining the feature information of the M fields includes: Obtain the indicator features required to perform clustering operations, and determine the feature values ​​under each indicator feature based on the field information of each field; The feature values ​​under each field are encoded to obtain the feature information of each field.

3. The method according to claim 1, characterized in that, Based on the feature information of the M fields, clustering is performed on the M fields to obtain multiple target field clusters, including: The density clustering algorithm is used to cluster the M fields according to their feature information to obtain N first field clusters, and the first candidate field cluster with the largest number of fields among the N first field clusters is obtained, where N is a positive integer; Based on the value of N and the first candidate field cluster, the initial clustering model K value is set to obtain the target clustering model. The target clustering model is then used to cluster the M fields according to the feature information of the M fields to obtain H second field clusters. The second candidate field cluster with the largest number of fields among the H second field clusters is then obtained. Determine whether the second candidate field cluster meets the preset iteration conditions; If the second candidate field cluster meets the preset iteration conditions, multiple fields in the second candidate field cluster are determined as the updated M fields, and the step of using density clustering algorithm to cluster the M fields according to the feature information of the M fields is iteratively executed to obtain N first field clusters until the second candidate field cluster meets the preset iteration conditions. If the second candidate field cluster does not meet the preset iteration conditions, the unprocessed second field cluster obtained after each iteration operation is determined as the target field cluster, thus obtaining the multiple target field clusters.

4. The method according to claim 3, characterized in that, Determining whether the second candidate field cluster meets the preset iteration conditions includes: Obtain the number of fields contained in the second candidate field cluster to obtain a first number, and determine whether the first number is greater than a preset threshold; If the first quantity is greater than the preset threshold, the second candidate field cluster is determined to meet the preset iteration condition; if the first quantity is less than or equal to the preset threshold, the second candidate field cluster is determined to not meet the preset iteration condition; or... Obtain the number of iterations for the iteration operation of the second candidate field cluster, and determine whether the number of iterations is greater than or equal to a preset number of iterations; If the number of iterations is less than the preset number of iterations, the second candidate field cluster is determined to meet the preset iteration condition; if the number of iterations is greater than or equal to the preset number of iterations, the second candidate field cluster is determined to not meet the preset iteration condition.

5. The method according to claim 3, characterized in that, The initial clustering model K value is set based on the value of N and the first candidate field cluster, including: Obtain the number of fields in the first candidate field cluster to get the second number; Divide the second quantity by the preset value to obtain the target change, and add the target change to the value of N to obtain the K value of the initial clustering model.

6. The method according to claim 1, characterized in that, The baseline fields in each target field cluster are determined based on the array of numbers for each target field cluster, including: Obtain the maximum quantity value in the quantity array, and determine the field corresponding to the maximum quantity value as the base field in the field cluster.

7. The method according to claim 1, characterized in that, Updating non-reference fields in the target database that are in the same field cluster as the reference field yields the updated target database, including: For any target field cluster, determine the location information of each non-benchmark field under the target field cluster in the target database; Based on the location information, each non-baseline field is replaced by a baseline field under the target field cluster to obtain the updated target database.

8. A data processing apparatus, characterized in that, include: The first acquisition unit is used to acquire multiple data tables in the target database and acquire the fields located in the multiple data tables to obtain M fields, where M is a positive integer; A clustering unit is used to obtain feature information of the M fields and cluster the M fields according to the feature information of the M fields to obtain multiple target field clusters; The second acquisition unit is used to acquire the number of identical fields in each target field cluster to obtain a quantity array for each target field cluster, wherein the quantity array includes the number of identical fields in each group; The update unit is used to determine the base field in each target field cluster based on the number array of each target field cluster, and update the non-base field in the target database that is in the same field cluster as the base field, so as to obtain the updated target database.

9. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the data processing method according to any one of claims 1 to 7.

10. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the data processing method according to any one of claims 1 to 7.

Images