A Data Collection and Statistics Method Based on a Loan Order Transaction Cloud Data Center

Abstract

This invention discloses a data collection and statistical method based on a loan order transaction cloud data center. First, multiple raw data sets generated by data terminal devices are initially collected. Then, the raw data undergoes preprocessing to reduce noise and redundant information, resulting in first data with important information. Simultaneously, clustering is performed on the first data, grouping multiple semantically similar first data sets into a single cluster to reduce data processing pressure. Each cluster is then processed and analyzed separately; that is, each first data set within a cluster is detected to obtain second data that better represents the semantics of the original data. This second data is then statistically analyzed to obtain statistical results. By preprocessing to remove useless information from the original data, the collected data becomes more refined. Furthermore, through data clustering and the output of data detection models, the data becomes more ordered, and the visualization of the statistical results is further improved.

Description

Technical Field

[0001] This invention relates to the field of data acquisition, and in particular to a data acquisition and statistical method based on a loan order transaction cloud data center. Background Technology

[0002] Data collection and statistics are crucial for server operation and management. For example, server log data needs to be collected and analyzed regularly to understand the server's operational status; another example is the collection of log data for a bank loan order transaction system.

[0003] Existing technologies often employ random sampling for data collection and statistics, and directly perform statistical analysis on the collected data manually or on identical data to obtain statistical results. This often leads to errors in the statistical results due to human subjectivity. Furthermore, directly analyzing the collected data can be inconvenient due to the complexity and disorder of the collected data, as well as the large amount of data.

[0004] This will further increase the computational pressure on the server, causing great inconvenience to data collection and statistics. Summary of the Invention

[0005] The purpose of this invention is to provide a data collection and statistics method based on a cloud data center for loan order transactions, which solves the aforementioned technical problems pointed out in the prior art.

[0006] This invention proposes a data collection and statistics method based on a loan order transaction cloud data center, comprising the following steps:

[0007] Obtain multiple raw data sets;

[0008] The original data is preprocessed to obtain the first data;

[0009] All the first data are divided into multiple clusters based on the semantic similarity between every two first data points;

[0010] Traverse all the clusters and use a pre-trained data detection model to detect and output the second data for each first data in the cluster.

[0011] The second data was statistically analyzed to obtain the statistical results.

[0012] Preferably, the second data includes label data and the label weights corresponding to the label data.

[0013] Preferably, the data detection model includes at least one convolutional layer and at least one pooling layer; the pooling layer is a global max pooling layer.

[0014] Preferably, the step of detecting and outputting the second data by passing each first data in the cluster through a pre-trained data detection model includes the following steps:

[0015] Determine n convolution kernels of a fixed size, and perform convolution on the first data in the cluster to extract an n-dimensional feature vector;

[0016] Pooling operations are performed on all the aforementioned feature vectors to obtain p largest feature vectors;

[0017] The second data is obtained by performing principal component analysis on all the largest eigenvectors.

[0018] Preferably, the step of performing principal component analysis on all the largest eigenvectors to obtain the second data includes the following steps:

[0019] Construct a first feature vector matrix R based on all the maximum feature vectors corresponding to all the clusters;

[0020] For each cluster, perform zero-mean operation on the p largest eigenvectors to obtain the second eigenvector matrix R'.

[0021] The covariance matrix C is calculated based on the second eigenvector matrix R'.

[0022] The third eigenvector is obtained by performing eigenvalue decomposition on the covariance matrix C.

[0023] The third feature vector is decoded to obtain the second data.

[0024] Preferably, the first eigenvector matrix R is represented as:

[0025] ;

[0026] Where Jqn'p is the p-th largest eigenvector in the q-th cluster;

[0027] The second eigenvector matrix R' is represented as:

[0028] .

[0029] Preferably, the step of calculating the covariance matrix C based on the second eigenvector matrix R' includes the following steps:

[0030] For each cluster, calculate the weight w corresponding to the largest eigenvector.

[0031] Obtain the weight inverse matrix W based on all the aforementioned weights w;

[0032] The covariance matrix C is calculated based on the second eigenvector matrix R' and the weight inverse matrix W.

[0033] Preferably, the covariance matrix C is calculated as follows:

[0034] C = R' × W;

[0035] Where C is the covariance matrix; W is the weight inverse matrix; and R' is the second eigenvector matrix.

[0036] Preferably, the step of calculating the weight w for each largest eigenvector in each cluster includes the following steps:

[0037] Calculate the word frequency distribution f for each largest eigenvector;

[0038] The term frequency distribution refers to the number of times each maximum feature vector appears;

[0039] The weight w of the largest feature vector is calculated based on the word frequency distribution f;

[0040] The weight w is calculated as follows:

[0041] w = ln(f), where f is the word frequency distribution.

[0042] Preferably, the step of statistically analyzing the word frequency distribution of each maximum feature vector includes the following steps:

[0043] The initial word frequency distribution is obtained by statistically analyzing each of the maximum feature vectors; and the total performance feature vector corresponding to the initial word frequency distribution and the initial word frequency distribution value corresponding to the total performance feature vector are obtained.

[0044] Clustering is performed on all the total performance feature vectors to obtain multiple performance feature vector clusters;

[0045] Traverse all the aforementioned performance feature vector clusters, sort each of the total performance feature vectors in the clusters from high to low according to their corresponding initial word frequency distribution values, select the highest-ranked total performance feature vector as the comprehensive performance feature vector, and add all the initial word frequency distribution values ​​in the current clusters to obtain the comprehensive word frequency distribution value.

[0046] Compared with the prior art, the embodiments of the present invention have at least the following technical advantages:

[0047] Analysis of the data collection and statistics method based on a loan order transaction cloud data center provided by this invention reveals that, in practical applications, multiple raw data generated by data terminal devices are first collected initially. Then, preprocessing is performed on the raw data to reduce noise and redundant information, resulting in first data with important information. Simultaneously, clustering is performed on the first data, grouping multiple semantically similar first data into a single cluster to reduce data processing pressure. Each cluster is then processed and analyzed separately; that is, each first data within a cluster is detected to obtain keywords that better represent the semantics of the original data (i.e., second data). Statistical analysis is then performed on the second data to obtain statistical results. By preprocessing data to remove useless information from the original data, the collected data becomes more refined. Furthermore, through data clustering and the output of the data detection model, the data becomes more ordered, and the visualization of the statistical results is further improved. Attached Figure Description

[0048] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0049] Figure 1 This is a schematic diagram illustrating the overall operation steps of a data collection and statistics method based on a loan order transaction cloud data center, as provided in an embodiment of the present invention.

[0050] Figure 2 This is a schematic diagram illustrating the steps for obtaining the second data in a data collection and statistics method based on a loan order transaction cloud data center, as provided in an embodiment of the present invention.

[0051] Figure 3 This is a schematic diagram illustrating the further explanation of the operation steps for obtaining the second data in a data collection and statistics method based on a cloud data center for loan order transactions, provided in an embodiment of the present invention.

[0052] Figure 4 This is a schematic diagram illustrating the steps for obtaining the covariance matrix C in a data collection and statistics method based on a cloud data center for loan order transactions, as provided in an embodiment of the present invention.

[0053] Figure 5 This is a schematic diagram illustrating the steps involved in calculating the weight w in a data collection and statistics method based on a cloud data center for loan order transactions, as provided in an embodiment of the present invention.

[0054] Figure 6This is a schematic diagram illustrating the operational steps of calculating the word frequency distribution of each maximum feature vector in a data collection and statistics method based on a cloud data center for loan order transactions, provided in an embodiment of the present invention.

[0055] Figure 7 This is a schematic diagram illustrating the steps of decoding the third feature vector to obtain the second data in a data collection and statistics method based on a loan order transaction cloud data center provided in an embodiment of the present invention. Detailed Implementation

[0056] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings.

[0058] like Figure 1 As shown, this invention also proposes a data collection and statistics method based on a loan order transaction cloud data center, including the following steps:

[0059] Step S10: Acquire multiple raw data, wherein the raw data is collected from the current data terminal device by the data acquisition module within a preset time period;

[0060] The aforementioned data terminal equipment includes data ports, web pages, etc.; the aforementioned raw data is generally text data or binary encoded data.

[0061] The above-described embodiments of this application collect raw data from data terminals such as data ports or web pages within a preset time period (e.g., collecting raw log data within the time period). This raw data is text data with complex content and unclear categories.

[0062] Step S20: Preprocess the raw data according to a preset data preprocessing strategy to obtain the first data;

[0063] The above data preprocessing strategies include text cleaning (stop word removal, etc.), text vectorization, and text normalization. These operations extract useful information from the raw data (i.e., perform data preprocessing on the raw log data to obtain only the time information, summary information, and important information from the log details), reduce noise and redundancy, and prepare for subsequent data collection operations.

[0064] Step S30: Traverse all of the first data and calculate the semantic similarity between every two first data.

[0065] The first data is divided into multiple clusters based on the semantic similarity.

[0066] The embodiments described above calculate the semantic similarity between every two pieces of first data using word vectors, sentence vectors, and word frequency features. K-means clustering is then used to cluster all the first data into multiple clusters. This allows original data with the same semantics to be grouped into one category, enabling further analysis and processing of the first data within each cluster, thereby reducing the processing pressure caused by processing large amounts of original data simultaneously. For example, clustering the first data means grouping logs with similar or identical data content into one cluster.

[0067] Step S40: Traverse all the clusters and detect each first data in the cluster using a pre-trained data detection model to output the second data;

[0068] The second data includes tag data and the tag weights corresponding to the tag data;

[0069] The data detection model includes at least one convolutional layer and at least one pooling layer; the pooling layer is a global max pooling layer.

[0070] The data detection model described above can extract keywords from the first set of data and represent repeated keywords numerically, thereby making the data more organized.

[0071] For example, we can analyze each cluster and examine the keyword tags and their weights within each cluster. The weight is determined by the frequency of keyword repetition.

[0072] Step S50: Perform statistical analysis on the second data to obtain statistical results.

[0073] It should be noted that the statistical method in the above embodiments of this application is as follows: for each cluster, calculate the word frequency distribution of each second data, that is, determine the number of times the keyword appears in each second data, and calculate the weight corresponding to each keyword; for each cluster, calculate the word vector of each second data, that is, treat each keyword in the second data as a word, and average the weight values ​​corresponding to all keywords to obtain the word vector corresponding to each keyword; further, for each cluster, calculate the similarity matrix of each second data, that is, perform a dot product of the word vectors of the second data in all clusters, and normalize the dot product result to obtain the similarity matrix for further analysis; through the above statistical analysis method, the statistical results of each cluster can be obtained, and the keywords, weight values, word vectors and other information in each cluster can also be obtained, so as to better visualize each cluster and facilitate further analysis and processing.

[0074] The technical solution adopted in the above-described embodiments of this application firstly collects multiple raw data generated by the data terminal device, then preprocesses the raw data to reduce noise and redundant information, thereby obtaining first data with important information; then, clustering is performed on the first data to group multiple semantically similar first data into one cluster, thereby reducing data processing pressure; then, each cluster is processed and analyzed separately, that is, each first data in the cluster is detected to obtain keywords (i.e., second data) that can better represent the semantics of the original data, and then the second data is statistically analyzed to obtain statistical results; by removing useless information from the original information through data preprocessing, the collected data is made more refined, and then the data clustering and data detection model output operations make the data more orderly and further improve the visualization of statistical results.

[0075] Specifically, such as Figure 2 As shown, in step S40, the first data in the cluster is detected and output as second data through a pre-trained data detection model, including the following steps:

[0076] Step S41: Determine n convolution kernels of fixed size, and perform convolution on the first data in the cluster to extract an n-dimensional feature vector;

[0077] Step S42: Perform pooling operation on all the feature vectors to obtain p largest feature vectors;

[0078] Step S43: Perform principal component analysis on all the largest eigenvectors to obtain the second data;

[0079] It should be noted that the second data obtained by the above maximum eigenvector analysis is obtained through principal component analysis, which can transform high-dimensional data into a low-dimensional space while retaining as much information as possible. The second data, i.e., the data after dimensionality reduction, is obtained by analyzing the maximum eigenvector through PCA.

[0080] The convolution kernels mentioned above can be determined by random initialization, thereby extracting features from the first data in different clusters. The pooling operation can be performed using global max pooling, thereby obtaining the maximum feature in different feature vectors. Then, the features are output through linear transformation to obtain the second data, making the second data more visually representable.

[0081] Through the above steps, the technical solution adopted in this application embodiment can refine multiple raw data generated by data terminal devices to obtain first data with important information. Then, clustering operations are performed on the first data, and the clusters are analyzed and processed to obtain keywords (i.e., second data) that better represent the semantics of the original data. Specifically, firstly, convolution operations are used to process the data and extract features related to the input data. These features reflect the dimension of the feature vectors. Then, pooling operations are performed on all feature vectors to retain the most significant features. Finally, principal component analysis (PCA) is used to reduce the dimensionality of the data, transforming high-dimensional data into a low-dimensional space while retaining as much information as possible. The largest feature vector is analyzed by PCA to obtain the second data, i.e., the data after dimensionality reduction. By extracting representative features from the original data through operations such as convolution, pooling, and PCA, the data is reduced to a form that is easier to process and understand. This can reduce the complexity of the data and improve the effectiveness and efficiency of subsequent processing tasks.

[0082] Furthermore, during the specific implementation of the present application, the technicians found that in the process of analyzing the largest eigenvector to obtain the second data, the complexity and high dimensionality of the data often lead to inconvenience in calculation, and it is also difficult to ensure that the second data obtained from the analysis can represent the key features in the first data. Therefore, how to analyze the largest eigenvector to obtain the second data is an urgent problem to be solved, as detailed in the subsequent specific implementation methods.

[0083] Specifically, such as Figure 3 As shown, in step S43, principal component analysis is performed on all the largest eigenvectors to obtain the second data, including the following steps:

[0084] Step S431: Construct a first eigenvector matrix R based on all the largest eigenvectors corresponding to all the clusters;

[0085] The first eigenvector matrix R is represented as:

[0086] ;

[0087] Where Jqn'p is the p-th largest eigenvector in the q-th cluster;

[0088] Step S432: Perform zero-mean operation on the p largest eigenvectors in each cluster to obtain the second eigenvector matrix R';

[0089] The second eigenvector matrix R' is represented as:

[0090] ;

[0091] Step S433: Calculate and obtain the covariance matrix C based on the second eigenvector matrix R';

[0092] Step S434: Perform eigenvalue decomposition on the covariance matrix C to obtain the third eigenvector;

[0093] It should be noted that the above embodiments of this application can perform eigenvalue decomposition using functions or methods provided in numerical computing software (such as NumPy, SciPy, etc.); typically, these functions return the eigenvalues ​​and eigenvectors (i.e., the third eigenvectors) of the covariance matrix.

[0094] Step S435: Decode the third feature vector to obtain the second data.

[0095] It should be noted that the second data obtained by decoding the third feature vector in the above embodiments of this application is based on the encoding and decoding operation of the autoencoder in the neural network, which can achieve fast and accurate output of the second data;

[0096] The embodiments described above reduce high-dimensional data to low-dimensional data using a matrix approach, thereby reducing data complexity and redundancy. Furthermore, zero-mean reduction eliminates the influence of mean values ​​between features, standardizes the data, and sorts and selects feature vectors to achieve data standardization, reduce data dimensionality, highlight key features, and improve the accuracy and efficiency of data analysis. Further, feature dimensionality reduction is performed by calculating the covariance matrix, which reduces data redundancy, improves computational efficiency, and retains as much of the main data information as possible. Finally, eigenvalue decomposition selects and retains the most important feature information (i.e., the aforementioned third feature vector), further reducing data dimensionality, computational complexity, and algorithm efficiency. After retaining the main feature with the largest variance in the data, the third feature vector is decoded to obtain the second data. The entire computational process of the embodiments described above is concise and reduces the complexity of data processing.

[0097] Specifically, such as Figure 4 As shown, in step S433, the covariance matrix C is calculated based on the second eigenvector matrix R', including the following steps:

[0098] Step S4331: For each cluster, calculate the weight w corresponding to all the largest eigenvectors.

[0099] Step S4332: Obtain the weight inverse matrix W based on all the weights w;

[0100] Step S4333: Calculate the covariance matrix C based on the second eigenvector matrix R' and the weight inverse matrix W;

[0101] The covariance matrix C is calculated as follows:

[0102] C = R' × W;

[0103] Where C is the covariance matrix; W is the weight inverse matrix; and R' is the second eigenvector matrix;

[0104] It should be noted that, in the specific technical solution of this application embodiment, the calculation of the covariance matrix C requires matrix multiplication of the second eigenvector matrix R' and the inverse matrix W of the weights to obtain the covariance matrix C.

[0105] Specifically, such as Figure 5 As shown, in step S4331, for each cluster, the weight w corresponding to all the largest eigenvectors is calculated, including the following steps:

[0106] Step S43311: Calculate the word frequency distribution f for each maximum feature vector;

[0107] The term frequency distribution refers to the number of times each maximum feature vector appears;

[0108] Step S43312: Calculate the weight w of the largest feature vector based on the word frequency distribution f;

[0109] The weight w is calculated as follows:

[0110] w = ln(f), where f is the word frequency distribution;

[0111] It should be noted that ln(f) above is a logarithmic function; the weight w is calculated using a logarithmic function to obtain the weight w of each eigenvector for subsequent calculations.

[0112] In the above embodiments of the present application, the occurrence times of each maximum eigenvector are first recorded in detail and sorted according to the occurrence frequency. This word frequency distribution not only reflects the importance of each maximum eigenvector in the clustering cluster, but also provides an important data basis for subsequent calculations;

[0113] Furthermore, using the word frequency distribution information, the weight w of each maximum eigenvector is obtained through a specific calculation method (logarithmic function), and the word frequency distribution is transformed into the corresponding weight w;

[0114] Calculating using the logarithmic function can simplify the calculation process and obtain the weight w of each maximum eigenvector. The logarithmic function has the property of transforming multiplication into addition, which enables direct addition operations on all weights w in subsequent calculations, thus simplifying the calculation process and improving the calculation efficiency; at the same time, this calculation method can also ensure that the value of the weight w will not be too large or too small due to the difference in word frequency, thereby ensuring the stability and comparability of the weight w. Such a processing method not only ensures the simplicity of the calculation process but also ensures the effectiveness and comparability of the weight w, providing an important basis for subsequent calculations.

[0115] In the specific implementation process of the above embodiments of the present application, the technical personnel also found that there are extremely many data of the maximum eigenvectors, resulting in a large number of vocabulary in the statistically obtained word frequency distribution, and thus a large amount of calculation. Therefore, when statistically calculating the word frequency distribution of the maximum eigenvectors, some words similar to the maximum eigenvectors need to be grouped as the same set of words for statistics, so as to obtain a word frequency distribution statistical result with a smaller amount of data. For example, when the maximum eigenvectors are respectively represented by words such as "our side", "my side", "the side of us", "this side", etc., there will be corresponding word frequency distributions, but their meanings are the same. A large number of data representations will affect the calculation speed. Therefore, it is necessary to analyze and process them to obtain the total word frequency distribution representing "our side", "my side", "the side of us", "this side", "this enterprise" by comprehensively statistically using "our side" or "my side" or "the side of us" or "this side" or "this enterprise", thereby reducing the amount of data and improving the data calculation and processing speed.

[0116] Specifically, as Figure 6 shown, in step S43311, statistically calculating the word frequency distribution of each maximum eigenvector (that is, statistically calculating the word frequency distribution f of each maximum eigenvector) includes the following operation steps:

[0117] Step S433111: Statistically calculate the initial word frequency distribution for each of the maximum eigenvectors; and obtain the total performance eigenvector corresponding to the initial word frequency distribution and the initial word frequency distribution value corresponding to the total performance eigenvector;

[0118] The aforementioned total performance feature vector refers to the largest feature vector corresponding to the word frequency distribution;

[0119] It should be noted that the above embodiments of this application perform initial statistics on the occurrence frequency of each maximum feature vector to obtain the initial word frequency distribution (that is, the occurrence frequency of each maximum feature vector, such as the occurrence frequency of "our side", the occurrence frequency of "our side", the occurrence frequency of "our side", etc.).

[0120] Step S433112: Cluster all the total performance feature vectors to obtain multiple performance feature vector clusters;

[0121] It should be noted that the above embodiments of this application can perform clustering operations through clustering methods such as K-means clustering and hierarchical clustering, thereby obtaining multiple total performance feature vectors that are semantically and performancely similar to each other to form a performance feature vector cluster.

[0122] Step S433113: Traverse all the said performance feature vector clusters, sort each of the total performance feature vectors in the performance feature vector clusters from high to low according to their corresponding initial word frequency distribution values, select the total performance feature vector with the highest ranking as the comprehensive performance feature vector; and add up all the initial word frequency distribution values ​​in the current performance feature vector clusters to obtain the comprehensive word frequency distribution value;

[0123] Thus, the comprehensive performance feature vector and the comprehensive word frequency distribution value obtained are the word frequency distribution mentioned above;

[0124] It should be noted that the technical solutions adopted in the above embodiments of this application can be illustrated by the following examples:

[0125] The current maximum feature vectors are represented as: "our side", "our side", "our side", "our side", "our company", "our side", "our side", "our side", "our company", "our side", "our side", "our side", "our side", "our side"; In step S433111 above, the initial word frequency distribution is first obtained by statistically analyzing each maximum feature vector, and the total performance feature vector and the initial word frequency distribution value corresponding to the total performance feature vector are obtained at the same time, namely "our side"-4, "our side"-3, "our side"-3, "our side"-2. "Our company"-2; then, the various total performance feature vectors are clustered to obtain performance feature vector clusters. In the examples of this application, "our side", "our side", "our side", "our company" and "our company" have the same meaning, so they are clustered into one cluster. At the same time, according to the description in step S433113 above, the total performance feature vector with the highest initial word frequency distribution value is taken as the comprehensive performance feature vector, and all the initial word frequency distribution values ​​are summed to obtain the comprehensive word frequency distribution value, which is the word frequency distribution of each of the above maximum feature vectors, represented as "our side"-14.

[0126] For example, through the specific implementation of the above embodiments of this application, the word frequency distributions of the largest feature vectors with the same actual meaning can be comprehensively statistically analyzed (thus reducing the computational pressure by reducing the dimension of the same or similar word frequency analysis), thereby avoiding the data computation pressure caused by separately statistically analyzing a large number of largest feature vectors, thereby improving computational efficiency.

[0127] Specifically, such as Figure 7 As shown, in step S435, the third feature vector is decoded to obtain the second data, including the following steps:

[0128] Step S4351: The third feature vector is used as input and an encoder is used to map the third feature vector into a hidden layer representation vector;

[0129] Step S4352: The hidden layer representation vector is used as input, and a decoder maps the hidden layer representation vector to an output vector to obtain the second data;

[0130] It should be noted that, in the embodiments of this application described above, the encoding and decoding operations are performed by an autoencoder, which can map the third feature vector to the second data. The second data can more intuitively represent the semantic information of the original data.

[0131] In summary, the data collection and statistical method proposed in this invention based on a loan order transaction cloud data center processes and analyzes each cluster separately. Specifically, it detects each first data point in the cluster to obtain keywords (i.e., second data) that better represent the semantics of the original data, and then performs statistics on the second data to obtain statistical results. By preprocessing the data to remove useless information from the original information, the collected data is made more refined. Furthermore, through data clustering and the output operation of the data detection model, the data is made more orderly, and the visualization of the statistical results is further improved.

[0132] Meanwhile, in further technical solutions, the process of analyzing the maximum eigenvector to obtain the second data often leads to inconvenience in calculation due to the complexity and high dimensionality of the data, and it is also difficult to ensure that the second data obtained from the analysis can represent the key features in the first data. Therefore, this embodiment uses the maximum eigenvector to analyze and obtain the second data, which simplifies the entire calculation process, reduces the complexity of data processing, and further improves the computing power of the cloud server in collecting and sorting log data.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; those skilled in the art can modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data collection and statistics method based on a loan order transaction cloud data center, characterized in that, The following steps are included: Obtain multiple raw data sets; The original data is preprocessed to obtain the first data; All the first data are divided into multiple clusters based on the semantic similarity between every two first data points; Traversing all the clusters, each first data point in the cluster is detected and output as second data using a pre-trained data detection model, including the following steps: Determine n convolution kernels of a fixed size, and perform convolution on the first data in the cluster to extract an n-dimensional feature vector; Pooling operations are performed on all the aforementioned feature vectors to obtain p largest feature vectors; The second data is obtained by performing principal component analysis on all the largest eigenvectors. The step of obtaining the second data by performing principal component analysis on all the largest eigenvectors includes the following steps: Construct a first feature vector matrix R based on all the maximum feature vectors corresponding to all the clusters; For each cluster, perform zero-mean operation on the p largest eigenvectors to obtain the second eigenvector matrix R'. The covariance matrix C is calculated based on the second eigenvector matrix R'. The third eigenvector is obtained by performing eigenvalue decomposition on the covariance matrix C. The third feature vector is decoded to obtain the second data; The second data was statistically analyzed to obtain the statistical results.

2. The data collection and statistics method based on a loan order transaction cloud data center according to claim 1, characterized in that, The second data includes label data and the label weights corresponding to the label data.

3. The data collection and statistics method based on a loan order transaction cloud data center according to claim 1, characterized in that, The data detection model includes at least one convolutional layer and at least one pooling layer; the pooling layer is a global max pooling layer.

4. The data collection and statistics method based on a loan order transaction cloud data center according to claim 1, characterized in that, The first eigenvector matrix R is represented as: ； Where Jqn'p is the p-th largest eigenvector in the q-th cluster; The second eigenvector matrix R' is represented as: 。 5. The data collection and statistics method based on a loan order transaction cloud data center according to claim 4, characterized in that, The calculation of the covariance matrix C based on the second eigenvector matrix R' includes the following steps: For each cluster, calculate the weight w corresponding to the largest eigenvector. Obtain the weight inverse matrix W based on all the aforementioned weights w; The covariance matrix C is calculated based on the second eigenvector matrix R' and the weight inverse matrix W.

6. The data collection and statistics method based on a loan order transaction cloud data center according to claim 5, characterized in that, The covariance matrix C is calculated as follows: C = R' × W; Where C is the covariance matrix; W is the weight inverse matrix; and R' is the second eigenvector matrix.

7. The data collection and statistics method based on a loan order transaction cloud data center according to claim 6, characterized in that, The step of calculating the weight w for the largest eigenvector in each cluster includes the following steps: Calculate the word frequency distribution f for each largest eigenvector; The term frequency distribution refers to the number of times each maximum feature vector appears; The weight w of the largest feature vector is calculated based on the word frequency distribution f; The weight w is calculated as follows: w = ln(f), where f is the word frequency distribution.

8. The data collection and statistics method based on a loan order transaction cloud data center according to claim 7, characterized in that, The statistical analysis of the word frequency distribution for each maximum feature vector includes the following steps: The initial word frequency distribution is obtained by statistically analyzing each of the maximum feature vectors; and the total performance feature vector corresponding to the initial word frequency distribution and the initial word frequency distribution value corresponding to the total performance feature vector are obtained. Clustering is performed on all the total performance feature vectors to obtain multiple performance feature vector clusters; Traverse all the aforementioned performance feature vector clusters, sort each of the total performance feature vectors in the clusters from high to low according to their corresponding initial word frequency distribution values, select the highest-ranked total performance feature vector as the comprehensive performance feature vector, and add all the initial word frequency distribution values ​​in the current cluster to obtain the comprehensive word frequency distribution value.

Images