Multi-resolution data element generation method and corresponding device

Abstract

The invention discloses a multi-resolution data element generation method and a corresponding device. The multi-resolution data element generation method comprises the following steps: carrying out correlation analysis on customer data so as to divide the customer data into multiple groups of related data according to correlation relationships among data in the customer data; for each group of related data, calculating a covariance matrix of the related data, performing eigenvalue decomposition on the covariance matrix to obtain a plurality of eigenvectors and an eigenvalue corresponding to each eigenvector, each eigenvector representing a principal component, calculating a ratio of the eigenvalue corresponding to each eigenvector to the total value of the eigenvalues, and calculating the eigenvalue corresponding to each eigenvector to the total value of the eigenvalues; determining the contribution rate of each principal component; and based on an application scene, according to the contribution rate of each principal component, screening the principal components of each group of related data, and based on the screened principal components of all the related data, generating a data element. According to the invention, the data controllability can be improved and the difficulty can be reduced.

Description

Technical Field

[0001] The present invention relates to the field of data application technology, and in particular to a multi-resolution data element generation method and a corresponding device. Background Art

[0002] At present, the direct application of raw data transactions is prone to leakage or abuse of raw data, and it is difficult to supervise the raw data after it is circulated. The current mainstream technology adopts privacy computing methods such as homomorphic encryption and multi-party computing, and provides them in the form of services. The calculation complexity is higher than the traditional method, and the data is not truly circulated. Summary of the invention

[0003] The embodiment of the present invention provides a multi-resolution data element generation method and a corresponding device, so as to reduce the difficulty of data element generation.

[0004] The method for generating a multi-resolution data element according to an embodiment of the present invention comprises:

[0005] Performing correlation analysis on the customer data to divide the customer data into multiple groups of related data according to correlation relationships between the data in the customer data;

[0006] For each group of the related data: calculating the covariance matrix of the related data, and performing eigenvalue decomposition on the covariance matrix to obtain a plurality of eigenvectors and eigenvalues ​​corresponding to each of the eigenvectors, each of the eigenvectors representing a principal component, and determining the contribution rate of each of the principal components by calculating the ratio of the eigenvalue corresponding to each of the eigenvectors to the total eigenvalues;

[0007] Based on the application scenario, the principal components of each group of the related data are screened according to the contribution rate of each principal component, and data elements are generated based on all the screened principal components of the related data.

[0008] According to some embodiments of the present invention, based on the application scenario, according to the contribution rate of each principal component, the principal component of each group of the related data is screened, and a data element is generated based on the screened principal components of all the related data, including:

[0009] Arbitrarily combining the principal components of each group of the related data, and generating data elements of different resolutions based on the principal components of all the related data of different combinations;

[0010] According to the application scenario requirements, one of the data elements with different resolutions is selected for application.

[0011] According to some embodiments of the present invention, the performing correlation analysis on the customer data to divide the customer data into multiple groups of related data according to the correlation relationship between each data in the customer data includes:

[0012] According to the attribute similarity of each data in the customer data, the customer data is divided into multiple groups of related data.

[0013] According to some embodiments of the present invention, dividing the customer data into multiple groups of related data according to the association relationship between each data in the customer data includes:

[0014] Clean customer data;

[0015] Convert the cleaned customer data into feature vectors;

[0016] Calculating the similarity between different feature vectors;

[0017] The feature vectors whose similarity values ​​exceed the threshold are divided into a group.

[0018] A multi-resolution data element generating device according to an embodiment of the present invention includes:

[0019] The customer data processing module is used to: perform correlation analysis on the customer data to divide the customer data into multiple groups of related data according to the correlation relationship between each data in the customer data; for each group of the related data: calculate the covariance matrix of the related data, and perform eigenvalue decomposition on the covariance matrix to obtain multiple eigenvectors and eigenvalues ​​corresponding to each eigenvector, each eigenvector represents a principal component, and calculate the ratio of the eigenvalue corresponding to each eigenvector to the total eigenvalue to determine the contribution rate of each principal component;

[0020] The data element construction module is used to screen the principal components of each group of the related data based on the application scenario and the contribution rate of each principal component, and generate data elements based on the screened principal components of all the related data.

[0021] According to some embodiments of the present invention, the data element construction module is used to:

[0022] Arbitrarily combining the principal components of each group of the related data, and generating data elements of different resolutions based on the principal components of all the related data of different combinations;

[0023] According to the application scenario requirements, one of the data elements with different resolutions is selected for application.

[0024] According to some embodiments of the present invention, the customer data processing module is used to:

[0025] According to the attribute similarity of each data in the customer data, the customer data is divided into multiple groups of related data.

[0026] According to some embodiments of the present invention, the customer data processing module is used to:

[0027] Clean customer data;

[0028] Convert the cleaned customer data into feature vectors;

[0029] Calculating the similarity between different feature vectors;

[0030] The feature vectors whose similarity values ​​exceed the threshold are divided into a group.

[0031] An embodiment of the present invention further provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the multi-resolution data element generation method as described above when executed by the processor.

[0032] The embodiment of the present invention further provides a computer-readable storage medium, on which a program for implementing information transmission is stored. When the program is executed by a processor, the steps of the multi-resolution data element generation method as described above are implemented.

[0033] By adopting the embodiments of the present invention, specific business requirements can be met without leaking original customer data information, thereby improving data controllability. Furthermore, by generating data elements based on principal components, the amount of calculation for generating data elements can be reduced, thereby lowering the difficulty.

[0034] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Various other advantages and benefits will become apparent to those skilled in the art by reading the detailed description of the embodiments below. The accompanying drawings are only used to illustrate the preferred embodiments and are not to be considered as limiting the present invention. In the accompanying drawings:

[0036] Figure 1 is a flow chart of a method for generating a multi-resolution data element in an embodiment of the present invention;

[0037] Figure 2 It is a schematic diagram of the effect of the method for generating a multi-resolution data element in an embodiment of the present invention. DETAILED DESCRIPTION

[0038] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art. In addition, in some examples, known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this specification.

[0039] Reference Figure 1 As shown, the method for generating a multi-resolution data element according to an embodiment of the present invention includes:

[0040] The original customer data is subjected to correlation analysis, so as to divide the customer data into multiple groups of related data according to the correlation relationship between each data in the customer data; for example, the customer data may be bank customer data, including the core and chain of corporate customer ID, customer type, industrial chain position, settlement frequency, daily average deposit balance, daily average RMB deposit balance, credit balance, loan balance, loan amount, bank bill balance, national standard industry sub-category, etc. Through correlation analysis of these data, they are divided into three groups of related data: customer identification data, customer loan data, and customer asset data.

[0041] There are many methods for correlation analysis, and the purpose is to group fields with the same characteristics into the same group. For example, the kmeans method can be used to divide the original customer data into three categories, and three groups of related data can be obtained, which are recorded as customer identification data, customer loan data, and customer asset data. The specific method is as follows: select k = 3, randomly select k data points from the customer data set as the initial centroid (cluster center), calculate the distance from each field to the centroid, and assign it to the class of the nearest centroid, recalculate the centroids of each class that have been obtained multiple times until the new centroid is equal to the original centroid or less than the specified threshold, and the algorithm ends. Attribute similarity can be obtained by calculating the distance from each field to the centroid. The smaller the distance, the higher the attribute similarity.

[0042] For each group of the related data: calculating the covariance matrix of the related data, and performing eigenvalue decomposition on the covariance matrix to obtain a plurality of eigenvectors and eigenvalues ​​corresponding to each of the eigenvectors, each of the eigenvectors representing a principal component, and determining the contribution rate of each of the principal components by calculating the ratio of the eigenvalue corresponding to each of the eigenvectors to the total eigenvalues;

[0043] The calculation process of the covariance matrix of the relevant data involves the following steps:

[0044] The first is standardization: First, standardize the relevant data, that is, remove the mean of each variable so that the mean of each variable is 0. The purpose of this step is to standardize the range of continuous initial variables so that each variable contributes equally to the analysis. It should be noted that before standardization, the text data in the relevant data can be converted into feature codes, such as customer type can be represented by 1, 2, 3, 4.

[0045] The second is to calculate the covariance: calculate the covariance between the standardized variables. Covariance is a way to measure the strength and direction of the linear relationship between two variables. If two variables increase or decrease at the same time, their covariance is positive; if one increases and the other decreases, the covariance is negative.

[0046] The third is to construct a covariance matrix: the covariance values ​​of all variable pairs calculated are summarized into a matrix to form a covariance matrix. This matrix is ​​a square matrix, the elements on the diagonal are the variances of each variable itself, and the elements on the off-diagonal are the covariances between the variables.

[0047] For three-dimensional data, the covariance matrix can be expressed as:

[0048]

[0049] Among them, Cov(X,Y) represents the covariance between variables X and Y. Through the calculated covariance matrix, its eigenvalues ​​and eigenvectors can be further calculated to determine the principal component directions of the relevant data, which can explain the variance in the data set to the greatest extent, that is, the direction of data change.

[0050] Based on the application scenario, the principal components of each group of the related data are screened according to the contribution rate of each principal component, and data elements are generated based on the screened principal components of all the related data. For example, for bank customer data, its application scenarios include credit rating and potential customer mining.

[0051] For example, customer data - customer asset data is first standardized and converted, and then six components are obtained through principal component analysis. Assuming that the contribution rate of the first two principal components reaches 90%, the first two principal components are directly used to obtain a high-resolution data element, which is used to replace the original customer data for potential customer credit rating analysis.

[0052] The data component includes component model, principal component and component result. The principal component is input into the component model to generate the component result. The component model can be obtained by the component developer through modeling according to the specific business. For example, in this example, a CNN neural network is used to create the corresponding component model.

[0053] By adopting the embodiment of the present invention, specific business needs can be met without leaking original and relatively sensitive customer data information, data controllability can be improved, and direct circulation and trading of original data can be avoided. Moreover, generating data elements based on principal components can reduce the amount of calculation for generating data elements and reduce the difficulty.

[0054] Reference Figure 2 As shown in the figure, multi-resolution data components are constructed using bank customer data as an example, including component 1, component 3, component 1.1, component 3.1, etc., and the prediction accuracy and time complexity of the application experiment of directly applying the original customer data to generate data components and using the principal component to generate data components are compared when conducting application experiments based on the neural network of data components. Among them, the O() function is used to represent the relationship between the time or space consumption of the algorithm and the amount of data growth, K represents the side length of the convolution kernel of the neural network, and C represents the time complexity of the neural network. in Represents the number of input channels of the neural network convolution kernel, C out It indicates the number of output channels of the convolution kernel of the neural network. It can be concluded that the method of the present invention can not only meet the application requirements, but also reduce the time complexity.

[0055] On the basis of the above-mentioned embodiments, various variant embodiments are further proposed. It should be noted that, in order to make the description concise, only the differences from the above-mentioned embodiments are described in each variant embodiment.

[0056] According to some embodiments of the present invention, based on the application scenario, according to the contribution rate of each principal component, the principal component of each group of the related data is screened, and a data element is generated based on the screened principal components of all the related data, including:

[0057] Arbitrarily combining the principal components of each group of the related data, and generating data elements of different resolutions based on the principal components of all the related data of different combinations;

[0058] According to the application scenario requirements, one of the data elements with different resolutions is selected for application.

[0059] Therefore, data elements with corresponding resolutions can be selected for application according to the application requirements of different scenarios, which are sufficient to support the application scenarios and can replace the original data to meet the accuracy requirements of the application model.

[0060] In this embodiment, after obtaining data elements of different resolutions, the data elements of different resolutions can be compared with the original customer data for application experiments, so that the information loss of customer data is minimized under a certain model accuracy loss r (r≤2%). In this way, data elements of different resolutions can retain customer data information to the greatest extent while reducing information leakage, thereby avoiding the direct circulation and transaction of original customer data.

[0061] According to some embodiments of the present invention, the performing correlation analysis on the customer data to divide the customer data into multiple groups of related data according to the correlation relationship between each data in the customer data includes:

[0062] According to the attribute similarity of each data in the customer data, the customer data is divided into multiple groups of related data.

[0063] For example, the data can be grouped directly according to the name field of the data. Another example is to group the data according to the application scenario of the data.

[0064] According to some embodiments of the present invention, dividing the customer data into multiple groups of related data according to the association relationship between each data in the customer data includes:

[0065] Clean customer data;

[0066] Convert the cleaned customer data into feature vectors;

[0067] Calculating the similarity between different feature vectors;

[0068] The feature vectors whose similarity values ​​exceed the threshold are divided into a group.

[0069] A multi-resolution data element generating device according to an embodiment of the present invention includes:

[0070] The customer data processing module is used to: perform correlation analysis on the customer data to divide the customer data into multiple groups of related data according to the correlation relationship between each data in the customer data; for each group of the related data: calculate the covariance matrix of the related data, and perform eigenvalue decomposition on the covariance matrix to obtain multiple eigenvectors and eigenvalues ​​corresponding to each eigenvector, each eigenvector represents a principal component, and calculate the ratio of the eigenvalue corresponding to each eigenvector to the total eigenvalue to determine the contribution rate of each principal component;

[0071] The data element construction module is used to screen the principal components of each group of the related data based on the application scenario and the contribution rate of each principal component, and generate data elements based on the screened principal components of all the related data.

[0072] By adopting the embodiments of the present invention, specific business requirements can be met without leaking original customer data information, thereby improving data controllability. Furthermore, by generating data elements based on principal components, the amount of calculation for generating data elements can be reduced, thereby lowering the difficulty.

[0073] On the basis of the above-mentioned embodiments, various variant embodiments are further proposed. It should be noted that, in order to make the description concise, only the differences from the above-mentioned embodiments are described in each variant embodiment.

[0074] According to some embodiments of the present invention, the data element construction module is used to:

[0075] Arbitrarily combining the principal components of each group of the related data, and generating data elements of different resolutions based on the principal components of all the related data of different combinations;

[0076] According to the application scenario requirements, one of the data elements with different resolutions is selected for application.

[0077] According to some embodiments of the present invention, the customer data processing module is used to:

[0078] According to the attribute similarity of each data in the customer data, the customer data is divided into multiple groups of related data.

[0079] According to some embodiments of the present invention, the customer data processing module is used to:

[0080] Clean customer data;

[0081] Convert the cleaned customer data into feature vectors;

[0082] Calculating the similarity between different feature vectors;

[0083] The feature vectors whose similarity values ​​exceed the threshold are divided into a group.

[0084] An embodiment of the present invention further provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the multi-resolution data element generation method as described above when executed by the processor.

[0085] The embodiment of the present invention further provides a computer-readable storage medium, on which a program for implementing information transmission is stored. When the program is executed by a processor, the steps of the multi-resolution data element generation method as described above are implemented.

[0086] A computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device.

[0087] Computer readable storage media may be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or components, or any combination thereof. More specific examples of computer readable storage media may include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0088] It should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0089] The contents not described in detail in the specification of the present invention belong to the common knowledge of the professionals in this field.

[0090] The terms "comprises", "includes" or any other variation thereof are intended to cover non-exclusive inclusion, such that a process, method, article or apparatus that includes a list of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

Claims

1. A method for generating a multi-resolution data element, characterized in that: include: Performing correlation analysis on the customer data to divide the customer data into multiple groups of related data according to correlation relationships between the data in the customer data; For each group of the related data: calculating the covariance matrix of the related data, and performing eigenvalue decomposition on the covariance matrix to obtain a plurality of eigenvectors and eigenvalues ​​corresponding to each of the eigenvectors, each of the eigenvectors representing a principal component, and determining the contribution rate of each of the principal components by calculating the ratio of the eigenvalue corresponding to each of the eigenvectors to the total eigenvalues; Based on the application scenario, the principal components of each group of the related data are screened according to the contribution rate of each principal component, and data elements are generated based on all the screened principal components of the related data.

2. The method according to claim 1, characterized in that The method of screening the principal components of each group of the related data based on the application scenario according to the contribution rate of each principal component, and generating data elements based on the screened principal components of all the related data includes: Arbitrarily combining the principal components of each group of the related data, and generating data elements of different resolutions based on the principal components of all the related data of different combinations; According to the application scenario requirements, one of the data elements with different resolutions is selected for application.

3. The method according to claim 1, characterized in that The performing correlation analysis on the customer data to divide the customer data into multiple groups of related data according to the correlation relationship between each data in the customer data includes: According to the attribute similarity of each data in the customer data, the customer data is divided into multiple groups of related data.

4. The method according to claim 1, characterized in that The step of dividing the customer data into a plurality of groups of related data according to the association relationship between the data in the customer data comprises: Clean customer data; Convert the cleaned customer data into feature vectors; Calculating the similarity between different feature vectors; The feature vectors whose similarity values ​​exceed the threshold are divided into a group.

5. A multi-resolution data element generating device, characterized in that: The multi-resolution data element generation device is used to implement the multi-resolution data element generation method according to any one of claims 1 to 4, and the multi-resolution data element generation device includes: a client data processing module and a data element construction module.

6. A computer device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the steps of the multi-resolution data element generation method according to any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an information transmission implementation program, and when the program is executed by a processor, the steps of the multi-resolution data element generation method according to any one of claims 1 to 4 are implemented.

Images