Data field processing method, device, equipment, medium and program product

Abstract

Embodiments of the present invention relate to the technical field of big data intelligent analysis, and disclose a data field processing method, apparatus, device, medium, and program product, wherein the method comprises: analyzing the importance of data fields in a preset data field set to obtain an importance index for each data field; performing cluster analysis on the data fields in the preset data field set based on the importance index to obtain multiple field clustering results; and performing synonymous data field analysis and merging on each of the field clustering results to obtain a target data field processing result. The technical solution of this embodiment solves the problem of a large amount of duplication and redundancy in externally accessed data fields, can reduce the redundancy of data fields, and improve the storage efficiency and usage efficiency of external data.

Description

Technical Field

[0001] Embodiments of the present invention relate to the technical field of big data intelligent analysis, and in particular to a data field processing method, apparatus, device, medium, and program product. Background Art

[0002] When processing business operations, some business entities will obtain large amounts of data from external data sources other than the business entity through external data interfaces. As the business needs of the business entity develop, the number of external data sources and data interfaces connected will usually increase.

[0003] However, since different data sources provide different standards for data services, when the amount of data is large, some fields or services with different names but highly similar in nature cannot be identified, and a large amount of data redundancy and duplication may occur. Summary of the Invention

[0004] The embodiments of the present invention provide a data field processing method, apparatus, device, medium and program product, which can integrate a large number of data fields, reduce the redundancy and duplication of data fields, and improve the storage efficiency and usage efficiency of external data.

[0005] In a first aspect, an embodiment of the present invention provides a data field processing method, the method comprising:

[0006] Analyze the importance of data fields in the preset data field set to obtain the importance index of each data field;

[0007] Performing cluster analysis on the data fields in the preset data field set based on the importance index to obtain multiple field clustering results;

[0008] Synonymous data field analysis and merging are performed on the clustering results of each field to obtain a target data field processing result.

[0009] In a second aspect, an embodiment of the present invention provides a data field processing device, the device comprising:

[0010] A field analysis module is used to analyze the importance of data fields in a preset data field set and obtain the importance index of each data field;

[0011] A field clustering module, configured to perform cluster analysis on the data fields in the preset data field set based on the importance index to obtain a plurality of field clustering results;

[0012] The field merging processing module is used to perform synonymous data field analysis and merging on each of the field clustering results to obtain a target data field processing result.

[0013] In a third aspect, an embodiment of the present invention further provides a computer device, comprising:

[0014] one or more processors;

[0015] a memory for storing one or more programs;

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the data field processing method provided by any embodiment of the present invention.

[0017] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data field processing method provided by any embodiment of the present invention.

[0018] In a fifth aspect, an embodiment of the present invention further provides a computer program product having a computer program stored thereon, which, when executed by a processor, implements the data field processing method provided by any embodiment of the present invention.

[0019] The embodiments of the above invention have the following advantages or beneficial effects:

[0020] In an embodiment of the present invention, the importance of data fields in a preset data field set is analyzed to obtain an importance index for each data field; based on the importance index, the data fields in the preset data field set are clustered and analyzed to obtain multiple field clustering results, that is, data fields with the same business scenario and similar importance are clustered into one category; then, synonymous data field analysis and merging are performed on each field clustering result, and fields with the same semantics in the same category are uniformly represented to obtain target data field processing results, thereby reducing redundancy and repetition of data fields. The technical solution of this embodiment solves the problem of a large amount of repetition and redundancy in externally accessed data fields, can reduce the redundancy of data fields, and improve the storage efficiency and usage efficiency of external data. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a flow chart of a data field processing method provided by an embodiment of the present invention;

[0022] Figure 2 This is a flow chart of a data field processing method provided by an embodiment of the present invention;

[0023] Figure 3 1 is a structural diagram of a data field processing device provided by an embodiment of the present invention;

[0024] Figure 4 It is a structural diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, not all structures.

[0026] It should be noted that similar reference numerals and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined or explained in subsequent figures. Furthermore, in the description of the present invention, the terms "first," "second," etc. are used only to distinguish descriptions and should not be understood to indicate or imply relative importance. The acquisition, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0027] Figure 1 This is a flow chart of a data field processing method provided by an embodiment of the present invention. This embodiment is applicable to scenarios where large amounts of data are received from external data sources, processing the data fields of the received data to optimize data storage and usage. This method can be performed by a data field processing device, which can be implemented in software and / or hardware and integrated into a computer device with application development capabilities.

[0028] like Figure 1 As shown, the data field processing method includes the following steps:

[0029] S110: Analyze the importance of data fields in the preset data field set to obtain an importance index of each data field.

[0030] The preset data field set is a data field set determined by extracting data fields of all data of all external data interfaces of a business entity with external data requirements, and is an object that requires data field integration processing.

[0031] A business entity typically has multiple external data interfaces based on business needs, connecting to multiple different data sources and acquiring data within different business domains. Thus, each data field has a corresponding associated external data interface. Furthermore, the term frequency-inverse document frequency index (TF-IDF) of each data field can be analyzed by analyzing the associated external data interfaces and field call information for the data fields in a preset data field set to obtain an importance index for each data field. The Term Frequency-Inverse Document Frequency (TF-IDF) index is a statistical analysis method for keywords, used to assess the importance of a word to a document set or corpus. The importance of a word is directly proportional to the number of times it appears in an article and inversely proportional to the number of times it appears in a corpus. This calculation method effectively avoids the influence of common words on keywords and improves the relevance between keywords and articles. TF refers to the total number of times a word appears in a document. This metric is typically normalized to define TF = (number of times a word appears in a document / total number of words in the document). This prevents the results from being biased towards overly long documents (the same word typically appears in longer documents than in shorter ones).

[0032] Specifically, the specific process of calculating the importance index of each data field in the preset data field set includes the following steps:

[0033] In the first step, the ratio of the number of times each data field appears in the preset data field set to the number of interfaces containing each data field is used as the term frequency of each data field. The formula can be expressed as: TF = number of times a data field appears / number of data interfaces containing the field.

[0034] In the second step, the decimal logarithm of the ratio of the number of all external data interfaces associated with the preset data field set to the number of external data interfaces containing each data field is used as the first importance factor (IDF1) of each data field. This can be expressed as: IDF1 of each data field = lg (total number of external data interfaces / number of interfaces containing the field).

[0035] In the third step, the product of the word frequency of each data field and the corresponding first importance coefficient is used as the importance index of each data field, which can be expressed as: importance index = TF*IDF1.

[0036] S120 . Perform cluster analysis on the data fields in the preset data field set based on the importance index to obtain multiple field clustering results.

[0037] Clustering of data fields can be done by classifying them according to the data business scenarios corresponding to the data sources, and classifying and integrating digital segments based on the scenarios. For example, the unified social credit code, business registration number, company organization number, etc. can be classified as company codes.

[0038] In this embodiment, K-means clustering is used to perform cluster analysis on data fields. The specific cluster analysis process includes:

[0039] First, a preset number of cluster centers are randomly selected from all data fields, and K-means clustering is performed based on the importance index of each data field to obtain the initial clustering result; then, the mean importance index of the data fields in each category in the initial clustering result is calculated as the updated cluster center, and K-means clustering is performed again based on the updated cluster center, and the clustering process is iterated until the target clustering result is obtained.

[0040] The preset number of cluster centers is determined by business personnel based on analysis of business areas and scenarios. This value can be set based on the actual number of business scenarios and the clustering results. After each clustering run, the importance index mean of the clustering results is recalculated and the cluster centers are adjusted until they no longer change or converge.

[0041] In a preferred embodiment, the user (relevant business personnel) can also intervene in the classification results to adjust some classification results that cannot be explained by business to make them reasonable. That is, in response to the user's clustering result adjustment instruction, the target clustering result is adjusted to obtain the final clustering result. In this process, the degree of user adjustment of the clustering result is also monitored to judge the validity of the K-means clustering result. For example, the user's clustering result adjustment ratio is determined, and the validity of the final clustering result is determined based on the adjustment ratio. If the adjustment range exceeds 5%, it is considered that the K-means clustering has failed and the clustering calculation needs to be re-performed.

[0042] S130 , performing synonymous data field analysis and merging on the field clustering results to obtain target data field processing results.

[0043] In the clustering results, there is a significant amount of redundancy in the data fields within each category. For example, the unified social credit code, credit code, social credit code, unified social code, and company unified social credit code are essentially unified social credit codes. Therefore, for each field in the clustering result, semantic analysis can be used to analyze the semantic relevance between multiple data fields. Fields with identical or similar semantics can then be merged and represented using the same data field, resulting in the processing results for the target data field.

[0044] Accordingly, in the subsequent reading and use of data, the merged and unified data fields can be used for processing, thereby avoiding data duplication and redundancy and improving data use efficiency and storage efficiency.

[0045] The technical solution of this embodiment analyzes the importance of data fields in a preset data field set to obtain an importance index for each data field; performs cluster analysis on the data fields in the preset data field set based on the importance index to obtain multiple field clustering results, that is, clusters data fields with the same business scenario and similar importance into one category; then performs synonymous data field analysis and merging on each field clustering result, uniformly represents fields with the same semantics in the same category, and obtains the target data field processing result, thereby reducing the redundancy and duplication of data fields. The technical solution of this embodiment solves the problem of a large amount of duplication and redundancy in externally accessed data fields, can reduce the redundancy of data fields, and improve the storage efficiency and usage efficiency of external data.

[0046] Figure 2 This is a flowchart of a data field processing method provided in an embodiment of the present invention. This embodiment, which shares the same inventive concept as the data field processing method in the previous embodiment, further describes the process of further processing the data field clustering results. This method can be performed by a data field processing device, which can be implemented in software and / or hardware and integrated into a computer device with application development capabilities.

[0047] like Figure 2 As shown, the data field processing method includes the following steps:

[0048] S210: Analyze the importance of the data fields in the preset data field set to obtain an importance index of each data field.

[0049] S220 . Perform cluster analysis on the data fields in the preset data field set based on the importance index to obtain multiple field clustering results.

[0050] S230: Using a preset correlation analysis algorithm to perform correlation analysis on the data fields in the field clustering results to obtain correlation analysis results.

[0051] In an optional implementation, principal component analysis (PCA) can be used as the default correlation analysis algorithm to analyze the correlation between fields in each clustering result. PCA is a data dimensionality reduction algorithm that maps n dimensions to k dimensions, where k (a positive integer less than n) is the principal component, i.e., a k-dimensional feature reconstructed from the original n-dimensional feature.

[0052] S240: Unify multiple data fields in the correlation analysis result whose correlation values ​​are greater than or equal to a preset correlation threshold into the same data field to obtain a target data field processing result.

[0053] Based on the results of principal component analysis, highly correlated data fields belonging to the same principal component can be merged and represented using a unified data field content. This means selecting a standard field name from multiple data fields within the same principal component and replacing other non-standard data fields to obtain the target data field processing result. For example, "unified social credit code" can be used to replace highly correlated data fields such as credit code, social credit code, unified social code, and company unified social credit code.

[0054] In a preferred embodiment, considering that principal component analysis cannot completely identify data fields that need to be replaced, the distance between each data field in a clustering result can be further calculated, and data fields with closer distances can also be uniformly replaced. Specifically, the decimal logarithm of the ratio of the total number of external data interface calls associated with all data fields to the number of calls to the external data interface containing each data field can be used as the second importance factor (IDF2) of each data field. This can be expressed as: IDF2 of each data field = lg (total number of external data interface calls / number of calls to the external data interface containing the data field). Then, the product of the word frequency of each data field and the corresponding second importance factor is used as the intra-class distance of each data field; this can be expressed as: field intra-class distance = TF * IDF2. Finally, fields in the clustering results whose correlation is less than a preset correlation threshold but whose corresponding intra-class distance difference is less than the preset distance difference are unified as the same data field. This further reduces duplication and redundancy of data fields.

[0055] The technical solution of this embodiment analyzes the importance of data fields in a preset data field set to obtain an importance index for each data field; performs cluster analysis on the data fields in the preset data field set based on the importance index to obtain multiple field clustering results, that is, clusters data fields with the same business scenario and similar importance into one category; then performs synonymous data field analysis and merging on each field clustering result, uniformly represents fields with the same semantics in the same category, and obtains the target data field processing result, thereby reducing the redundancy and duplication of data fields. The technical solution of this embodiment solves the problem of a large amount of duplication and redundancy in externally accessed data fields, can reduce the redundancy of data fields, and improve the storage efficiency and usage efficiency of external data.

[0056] In a specific example, the business entity accessing external data is a bank. The process of processing data fields on the external data source accessed by the bank's business system is as follows:

[0057] 1. Build a corpus.

[0058] Currently, a certain banking entity has thousands of external data interfaces, involving hundreds of thousands of data fields. A comprehensive extraction is performed on all of these data fields. The external data interfaces with the highest call frequency are statistically analyzed, and those with a higher call frequency than a preset frequency are selected as preferred external data interfaces. The call frequency of all data fields in each preferred external data interface is then calculated for field screening, resulting in the 10,000 most frequently used data fields, which are then used as the preset data field set.

[0059] 2. For the data fields in the preset data field set, perform vectorized calculations using TF-IDF.

[0060] In this step, two values ​​need to be calculated: one is the importance value of the data field before classification, and the other is the distance value after the field is classified.

[0061] When calculating TF-IDF, we first count the number of occurrences of the same data field, including the number of interfaces for each field, and the number of calls to external data interfaces related to each data field within a preset historical time period (e.g., within the past year). If a data field appears frequently and is called a high number of times, TF-IDF is used to assess its importance.

[0062] Specifically, the TF of each data field = the number of occurrences of the data field / the number of external data interfaces containing the corresponding data field; the IDF1 of each data field = lg (the total number of external interfaces / the number of external data interfaces containing the corresponding data field); and the IDF2 of each data field = log (the total number of external data interface calls / the number of external data interface calls containing the corresponding data field). Furthermore, the value Y1 obtained by multiplying TF by IDF1 is the importance value of the field before classification. The value Y2 obtained by multiplying TF by IDF2 is the distance value between the field and different fields in the same category after classification.

[0063] 3. Perform cluster analysis on data fields.

[0064] Based on the data application scenario classification, approximately 80 categories are needed (this conclusion is based on accumulated experience with data usage and continuous iteration). Initially, there are 10,000 data fields, divided into 80 categories. The initial clustering value is set to 80, which can be calculated using tools such as Python. To achieve the above classification results, k-mean clustering is performed using the Y1 = TF * IDF1 value calculated in the second step. This requires iteration: using the calculated Y1 minimum center point principle, the Y1 values ​​of the 10,000 fields are distributed among the 80 center points. Each category has several data fields. The mean of the sample points in the 80 categories can be calculated and used as the 80 center points for the second iteration of k-mean clustering analysis. At this point, the classification of some fields may change. This process is repeated until the center points no longer change or convergence is achieved.

[0065] Furthermore, users can fine-tune the clustering results. The purpose of the adjustment is to adjust some classifications that cannot be explained by the business to make them reasonable. The adjustment range should be monitored to not exceed 5%. Otherwise, the clustering is considered to have failed and the clustering calculation needs to be repeated.

[0066] Finally, the 10,000 fields are divided into 80 categories of data, such as the unified social credit code, industrial and commercial registration number, company organization number, etc. are classified as company codes. The company code is a class, and under this class there are fields such as the unified social credit code, industrial and commercial registration number, company organization number, parent company level, subsidiary level, etc.

[0067] The correlation between fields in the same category can be judged and quantified using vectors.

[0068] 4. After classification, similar fields are merged to reduce redundancy.

[0069] Among the 80 types of data fields obtained through classification, a large number of data fields are redundant, such as the unified social credit code, credit code, social credit code, unified social code, company unified social credit code, etc. These are essentially unified social credit codes.

[0070] When processing redundant data fields, you can follow the following two steps:

[0071] 1) Use principal component analysis to conduct correlation analysis on data fields of the same type. Fields with extremely high correlation must be replaced (essentially the same purpose and meaning). That is, for the same field, choose a standard field name as a non-standard field name, such as using "unified social credit code" to replace credit code, social credit code, unified social code, company unified social credit code, and other highly correlated fields.

[0072] 2) Principal component analysis cannot completely identify the fields that need to be replaced. It is necessary to calculate the distance value of each data field within the same clustering result using Y2 = TF * IDF2. Fields with close distances also need to be considered for replacement with standard field names.

[0073] Through the above steps, 10,000 fields can be reduced to 4,268 fields used by non-repetitive business. At the same time, because the interface calls contained in these fields are in the tens of millions per year, the storage pressure can be greatly reduced by about 5.43 times.

[0074] Figure 3 This is a schematic diagram of the structure of a data field processing device provided in an embodiment of the present invention. This embodiment is applicable to scenarios where data fields are processed when large amounts of data are received from external data sources. The device can be implemented using software and / or hardware and integrated into a computer device with application development capabilities.

[0075] like Figure 3 As shown, the data field processing device includes: a field analysis module 310, a field clustering module 320 and a field merging processing module 330.

[0076] Among them, the field analysis module 310 is used to analyze the importance of data fields in the preset data field set to obtain the importance index of each data field; the field clustering module 320 is used to perform cluster analysis on the data fields in the preset data field set based on the importance index to obtain multiple field clustering results; the field merging processing module 330 is used to perform synonymous data field analysis and merging on each of the field clustering results to obtain the target data field processing result.

[0077] The technical solution of this embodiment analyzes the importance of data fields in a preset data field set to obtain an importance index for each data field; performs cluster analysis on the data fields in the preset data field set based on the importance index to obtain multiple field clustering results, that is, clusters data fields with the same business scenario and similar importance into one category; then performs synonymous data field analysis and merging on each field clustering result, uniformly represents fields with the same semantics in the same category, and obtains the target data field processing result, thereby reducing the redundancy and duplication of data fields. The technical solution of this embodiment solves the problem of a large amount of duplication and redundancy in externally accessed data fields, can reduce the redundancy of data fields, and improve the storage efficiency and usage efficiency of external data.

[0078] Optionally, the field analysis module 310 is specifically configured to:

[0079] Based on the external data interface and field call information associated with each data field, the word frequency-inverse text frequency index of each data field is analyzed to obtain the importance index of each data field.

[0080] Optionally, the field analysis module 310 is specifically configured to:

[0081] The ratio of the number of times each data field appears in the preset data field set to the number of external data interfaces containing each data field is used as the word frequency of each data field;

[0082] The decimal logarithm of the ratio of the number of all external data interfaces associated with the preset data field set to the number of external data interfaces containing each data field is used as the first importance coefficient of each data field;

[0083] The product of the word frequency of each data field and the corresponding first importance coefficient is used as the importance index of each data field.

[0084] Optionally, the field clustering module 320 is specifically configured to:

[0085] Randomly selecting a preset number of cluster centers from all the data fields, performing K-means clustering based on the importance index of each data field, and obtaining an initial clustering result;

[0086] The importance index mean of the data fields in each category in the initial clustering result is calculated as the updated cluster center, and K-means clustering is re-performed based on the updated cluster center, and the clustering process is iterated until the target clustering result is obtained.

[0087] Optionally, the data field processing device further includes a classification adjustment module, configured to:

[0088] In response to the user's clustering result adjustment instruction, the target clustering result is adjusted to obtain a final clustering result.

[0089] Optionally, the classification adjustment module is further configured to:

[0090] An adjustment ratio of the clustering result of the user is determined, and the validity of the final clustering result is determined according to the adjustment ratio.

[0091] Optionally, the field merging processing module 330 is specifically configured to:

[0092] Using a preset correlation analysis algorithm to perform correlation analysis on the data fields in the field clustering results to obtain correlation analysis results;

[0093] Multiple data fields in the correlation analysis result whose correlation values ​​are greater than or equal to a preset correlation threshold are unified into the same data field.

[0094] Optionally, the preset correlation analysis algorithm includes principal component analysis.

[0095] Optionally, the field merging processing module 330 is further configured to:

[0096] The decimal logarithm of the ratio of the total call volume of the external data interface associated with all the data fields to the call volume of the external data interface containing each of the data fields is used as the second importance coefficient of each of the data fields;

[0097] The product of the word frequency of each data field and the corresponding second importance coefficient is used as the intra-class distance of each data field;

[0098] The fields whose correlation among the data fields in the field clustering results is less than the preset correlation threshold and whose corresponding intra-class distance difference is less than the preset distance difference are unified as the same data field.

[0099] Optionally, the preset number is set based on the number of business scenarios and is 80.

[0100] Optionally, the data field processing device further includes a data acquisition module for constructing the preset data field set. The specific process includes:

[0101] Among all external data interfaces of the target business entity, select the preferred external data interface with a call frequency higher than the preset frequency;

[0102] Field screening is performed on all data fields in each of the preferred external data interfaces to obtain the preset data field set.

[0103] The data field processing device provided in the embodiment of the present invention can execute the data field processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0104] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the embodiments of the present invention.

[0105] Figure 4 A structural diagram of a computer device is also provided for an embodiment of the present invention. Figure 4 A block diagram of an exemplary computer device 12 suitable for use in implementing embodiments of the present invention is shown. Figure 4The computer device 12 shown is only an example and should not limit the functionality and scope of use of the embodiments of the present invention. The computer device 12 can be any terminal device with computing capabilities, such as an intelligent controller, a server, a mobile phone, or other terminal devices.

[0106] like Figure 4 As shown, computer device 12 is implemented as a general-purpose computing device. Components of computer device 12 may include, but are not limited to, one or more processors or processing units 16, system memory 28, and a bus 18 that connects various system components (including system memory 28 and processing unit 16).

[0107] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. Examples of these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0108] The computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0109] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be configured to read and write non-removable, non-volatile magnetic media ( Figure 4 Not shown, often called a "hard drive"). Although Figure 4 Not shown, a magnetic disk drive for reading and writing to a removable non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. System memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments of the present invention.

[0110] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may include an implementation of a network environment. Program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0111] The computer device 12 may also communicate with one or more external devices 14 (e.g., a keyboard, a pointing device, a display 24, etc.), one or more devices that enable a user to interact with the computer device 12, and / or any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the computer device 12 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with the other modules of the computer device 12 via the bus 18. It should be understood that although Figure 4 Not shown, other hardware and / or software modules may be used in conjunction with computer device 12, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0112] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the data field processing method provided in the embodiment of the present invention, which includes:

[0113] Analyze the importance of data fields in the preset data field set to obtain the importance index of each data field;

[0114] Performing cluster analysis on the data fields in the preset data field set based on the importance index to obtain multiple field clustering results;

[0115] Synonymous data field analysis and merging are performed on the clustering results of each field to obtain a target data field processing result.

[0116] This embodiment further provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the method for processing a data field as provided in any embodiment of the present invention is implemented. The method includes:

[0117] Analyze the importance of data fields in the preset data field set to obtain the importance index of each data field;

[0118] Performing cluster analysis on the data fields in the preset data field set based on the importance index to obtain multiple field clustering results;

[0119] Synonymous data field analysis and merging are performed on the clustering results of each field to obtain a target data field processing result.

[0120] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0121] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0122] Program code embodied on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0123] The computer program code for performing the operations of the present invention can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0124] Those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using a general-purpose computing device. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Alternatively, they can be implemented using program code executable by a computer device, which can then be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.

[0125] An embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the data field processing method provided in any embodiment of the present application.

[0126] The computer program product may be implemented by writing computer program code for performing the operations of the present invention in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0127] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the concept of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. A data field processing method, characterized in that: include: Analyzing the importance of data fields in a preset data field set to obtain an importance index for each data field; wherein the preset data field set is a data field set determined by extracting data fields of all data of all external data interfaces of a business entity with external data requirements; analyzing the importance of data fields in the preset data field set to obtain an importance index for each data field includes analyzing a word frequency-inverse text frequency index of each data field based on external data interfaces and field call information associated with each data field to obtain an importance index for each data field; Performing cluster analysis on the data fields in the preset data field set based on the importance index to obtain multiple field clustering results; Using a preset correlation analysis algorithm to perform correlation analysis on the data fields in the field clustering results to obtain correlation analysis results; Multiple data fields whose correlation values ​​in the correlation analysis results are greater than or equal to a preset correlation threshold are unified into the same data field to obtain a target data field processing result.

2. The method according to claim 1, characterized in that The analysis of the word frequency-inverse text frequency index of each data field based on the external data interface and field call information associated with each data field to obtain the importance index of each data field includes: The ratio of the number of times each data field appears in the preset data field set to the number of external data interfaces containing each data field is used as the word frequency of each data field; The decimal logarithm of the ratio of the number of all external data interfaces associated with the preset data field set to the number of external data interfaces containing each data field is used as the first importance coefficient of each data field; The product of the word frequency of each data field and the corresponding first importance coefficient is used as the importance index of each data field.

3. The method according to claim 1, characterized in that The clustering analysis is performed on the data fields in the preset data field set based on the importance index to obtain multiple field clustering results, including: Randomly selecting a preset number of cluster centers from all the data fields, performing K-means clustering based on the importance index of each data field, and obtaining an initial clustering result; The importance index mean of the data fields in each category in the initial clustering result is calculated as the updated cluster center, and K-means clustering is re-performed based on the updated cluster center, and the clustering process is iterated until the target clustering result is obtained.

4. The method according to claim 3, characterized in that The method further comprises: In response to the user's clustering result adjustment instruction, the target clustering result is adjusted to obtain a final clustering result.

5. The method according to claim 4, characterized in that The method further comprises: An adjustment ratio of the clustering result of the user is determined, and the validity of the final clustering result is determined according to the adjustment ratio.

6. The method according to claim 1, characterized in that The preset correlation analysis algorithm includes principal component analysis.

7. The method according to claim 1, characterized in that The method further comprises: The decimal logarithm of the ratio of the total call volume of the external data interface associated with all the data fields to the call volume of the external data interface containing each of the data fields is used as the second importance coefficient of each of the data fields; The product of the word frequency of each data field and the corresponding second importance coefficient is used as the intra-class distance of each data field; The fields whose correlation among the data fields in the field clustering results is less than the preset correlation threshold and whose corresponding intra-class distance difference is less than the preset distance difference are unified as the same data field.

8. The method according to claim 3, characterized in that The preset number is a value set based on the number of business scenarios, which is 80.

9. The method according to claim 1, characterized in that The process of constructing the preset data field set includes: Among all external data interfaces of the target business entity, select the preferred external data interface with a call frequency higher than the preset frequency; Field screening is performed on all data fields in each of the preferred external data interfaces to obtain the preset data field set.

10. A data field processing device, characterized in that: include: A field analysis module is configured to analyze the importance of data fields in a preset data field set to obtain an importance index for each data field; wherein the preset data field set is a data field set determined by extracting data fields from all data of all external data interfaces of a business entity with external data requirements; analyzing the importance of data fields in the preset data field set to obtain an importance index for each data field includes analyzing a word frequency-inverse text frequency index for each data field based on the external data interface and field call information associated with each data field to obtain an importance index for each data field; A field clustering module, configured to perform cluster analysis on the data fields in the preset data field set based on the importance index to obtain a plurality of field clustering results; The field merging processing module is used to use a preset correlation analysis algorithm to perform correlation analysis on the data fields in each of the field clustering results to obtain a correlation analysis result; multiple data fields in the correlation analysis results whose correlation values ​​are greater than or equal to a preset correlation threshold are unified into the same data field to obtain a target data field processing result.

11. A computer device, characterized in that: The computer device comprises: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the data field processing method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data field processing method according to any one of claims 1 to 9 is implemented.

13. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the data field processing method according to any one of claims 1 to 9.

Images