Statistics multi-dimensional data processing system and method combined with clustering analysis

Abstract

The invention discloses a clustering analysis-combined statistical multi-dimensional data processing system and method, and aims to solve the problems of low quality, poor precision, insufficient value mining and the like of a traditional data processing technology in a multi-dimensional data scene. The system comprises a data acquisition module, a preprocessing module, a clustering analysis module, a statistical processing module, a result output module and a storage module. The method sequentially executes the steps of data acquisition, preprocessing, clustering analysis, statistical processing, result output and storage. In the acquisition stage, data are integrated through a multi-source interface and formats are unified, in the preprocessing stage, abnormal values are removed through the 3 sigma principle, missing values are supplemented through multiple methods, in the clustering stage, the optimal clustering number is determined by depending on a contour coefficient or a Clinski-Harabasz index, in the statistical stage, deep association is mined for all clusters, in the output stage, diversified forms are provided, and in the storage stage, data safety is guaranteed. According to the scheme, data quality and analysis precision are improved, and reliable support is provided for data-driven decision-making of each industry.

Description

Technical Field

[0001] This invention relates to statistics, specifically to a statistical multidimensional data processing system and method that combines cluster analysis. Background Technology

[0002] With the development of the digital age, data from various industries is becoming increasingly multi-dimensional and multi-sourced. For example, e-commerce data includes metrics such as user age, spending amount, and purchase frequency, while the financial sector provides data on customer assets, transaction frequency, and risk ratings. This places higher demands on data processing technologies. However, traditional data processing technologies have many shortcomings and struggle to meet these practical needs.

[0003] In terms of data acquisition and preprocessing, traditional techniques often rely on a single data source and lack an effective mechanism for unifying formats. Data from different sources tend to have chaotic formats and large differences in dimensions. Furthermore, the methods for handling outliers and missing values ​​are simple, which can easily lead to "garbage data in, garbage results out," affecting the accuracy of subsequent analysis.

[0004] In the cluster analysis process, traditional techniques often involve manually pre-setting the number of clusters, lacking a scientific optimization mechanism. The clustering results are either coarse or fragmented, failing to accurately reflect the inherent relationships in the data and making it difficult to support refined analysis.

[0005] In statistical processing, traditional methods analyze the overall data from a single dimension, neglecting in-depth mining of grouped data. This fails to discover unique patterns and dimensional relationships among different groups, resulting in weak guidance for the analysis results.

[0006] The output and security measures also have shortcomings. The output format is limited and has poor adaptability. Furthermore, the data storage and backup are insufficient and the encryption mechanism is imperfect, which makes it easy for data to be lost or leaked, thus hindering the realization of data value. New data processing technologies are urgently needed to overcome these bottlenecks. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to overcome the defects of the above-mentioned technologies and provide a statistical multidimensional data processing system and method that combines cluster analysis.

[0008] To address the aforementioned technical problems, the present invention provides a statistical multidimensional data processing system and method combining cluster analysis: The statistical multidimensional data processing system combining cluster analysis includes:

[0009] The data acquisition module is used to acquire multi-dimensional raw data. The multi-dimensional raw data has two or more independent descriptive dimensions. Each descriptive dimension corresponds to an attribute or feature variable of the data. The data object is defined and characterized by multiple descriptive dimensions.

[0010] The data preprocessing module is communicatively connected to the data acquisition module and is used to clean, standardize, and fill in missing values ​​for the multi-dimensional raw data to obtain standardized multi-dimensional data.

[0011] The clustering analysis module is communicatively connected to the data preprocessing module and is used to perform similarity grouping processing on the standardized multi-dimensional data based on a preset clustering algorithm to obtain multiple data clusters. The similarity grouping processing is based on the feature differences of data objects in each descriptive dimension.

[0012] The statistical processing module is communicatively connected to the cluster analysis module and is used to perform statistical analysis on each of the data clusters. The statistical analysis includes at least two of the following: mean calculation, analysis of variance, and correlation analysis, and outputs the statistical analysis results corresponding to each data cluster.

[0013] The results output module is communicatively connected to the statistical processing module and is used to output the data clustering information and corresponding statistical analysis results in a visual or structured data format.

[0014] As an improvement, the data acquisition module includes a multi-source data interface, which supports at least two data acquisition methods: database import, API interface call, and file upload. It can also perform format unification processing on multi-dimensional raw data from different sources. During the format unification process, for numerical descriptive dimension data, a standardized formula is used for preliminary dimensional unification. The standardized formula is:

[0015] Where, x i Let x represent the i-th original data value in a descriptive dimension of the multi-dimensional original data, μ represent the mean of all original data values ​​in that descriptive dimension, σ represent the standard deviation of all original data values ​​in that descriptive dimension, and x represent the mean of all original data values ​​in that descriptive dimension. i′ This represents the i-th data value after standardization. The purpose of this formula is to convert numerical descriptive dimensional data of different magnitudes to the same magnitude range, so as to avoid the accuracy of subsequent data format unification due to differences in units.

[0016] As an improvement, the preset clustering algorithm in the clustering analysis module includes at least one of K-means clustering, hierarchical clustering, and density clustering. Furthermore, the clustering analysis module also includes a cluster number optimization unit, which adaptively adjusts the number of clusters based on the silhouette coefficient or the Calinski-Harabasz index to determine the optimal number of data clusters. The silhouette coefficient is calculated using the following formula:

[0017]

[0018] In the formula, s(i) represents the silhouette coefficient of the i-th data object, a(i) represents the average distance between the i-th data object and all other data objects in the same cluster, and b(i) represents the average distance between the i-th data object and all data objects in the nearest different clusters. The purpose of this formula is to quantitatively evaluate the compactness of the data object within its own cluster and its separation from other clusters. The closer s(i) is to 1, the better the clustering effect of the data object. The clustering number optimization unit determines the optimal number of clusters by calculating the average silhouette coefficient of all data objects.

[0019] As an improvement, a data storage module is also included. This data storage module is communicatively connected to the data acquisition module, data preprocessing module, cluster analysis module, and statistical processing module, respectively. It stores the multi-dimensional raw data, standardized multi-dimensional data, data cluster information, and statistical analysis results. The data storage module adopts a distributed storage architecture, supporting real-time data read / write and backup. During data backup, a data integrity verification mechanism is employed, verifying the consistency between the backup data and the source data through hash value calculation. The hash value calculation uses the formula: H = SHA-256(D).

[0020] Where H represents the SHA-256 hash value of data D, D represents any type of data to be backed up (multi-dimensional raw data, standardized multi-dimensional data, etc.), and SHA-256 represents the cryptographic hash function algorithm; the function of this formula is to convert data D of arbitrary length to be backed up into a hash value H of fixed length (256 bits). By comparing the hash values ​​H of the source data and the backup data, if the two are consistent, it means that the backup data is complete and undamaged, thus ensuring the reliability of the data backup.

[0021] The statistical multidimensional data processing method combining cluster analysis includes the following steps:

[0022] S1. Data acquisition steps: Acquire multi-dimensional raw data, wherein the multi-dimensional raw data has two or more independent descriptive dimensions, each descriptive dimension corresponds to an attribute or feature variable of the data, and the data object is defined and characterized by multiple descriptive dimensions.

[0023] S2. Data preprocessing steps: Clean, standardize and fill missing values ​​in the multi-dimensional raw data, remove abnormal data and convert the data into a uniform dimension to obtain standardized multi-dimensional data;

[0024] S3. Clustering analysis steps: Based on the preset clustering algorithm, the standardized multi-dimensional data is grouped by similarity. Based on the feature differences of data objects in each descriptive dimension, multiple data clusters are obtained.

[0025] S4. Statistical processing steps: Perform statistical analysis on each of the data clusters, including at least two of the following: mean calculation, analysis of variance, and correlation analysis, to obtain the statistical analysis results corresponding to each data cluster;

[0026] S5. Result Output Step: Output the data cluster information and corresponding statistical analysis results in a visualization or structured data format.

[0027] As an improvement, in step S1, when acquiring multi-dimensional raw data, data acquisition is achieved through a multi-source data interface. This multi-source data interface supports at least two data acquisition methods: database import, API interface call, and file upload. Furthermore, after acquiring the data, the multi-dimensional raw data from different sources undergoes format unification processing to ensure consistent data structure. In the format unification processing, for percentage-type descriptive dimension data, a normalization formula is used to convert it to decimal form. The normalization formula is:

[0028] Among them, y j y represents the j-th raw data value (in %) of a certain percentage-type descriptive dimension. j′ This represents the j-th data value (in decimal form) after normalization. The purpose of this formula is to convert percentage data into a uniform decimal format, which facilitates collaborative processing with other numerical descriptive dimension data and avoids data fusion errors caused by format differences.

[0029] As an improvement, in step S3, the preset clustering algorithm includes at least one of K-means clustering, hierarchical clustering, and density clustering, and before performing similarity grouping processing, it further includes a cluster number optimization step: adaptively adjusting the number of clusters based on the silhouette coefficient or the Calinski-Harabasz index to determine the optimal number of data clusters before performing similarity grouping processing; wherein, the Calinski-Harabasz index is calculated using the formula: In the formula,

[0030] CH represents the Calinski-Harabasz index, SSB represents the sum of squared deviations from the mean among different clusters, SSW represents the sum of squared deviations from the mean within the same cluster, k represents the number of clusters, and n represents the total number of data objects. This formula evaluates the clustering effect by measuring the ratio of inter-cluster differences to intra-cluster differences. The larger the CH value, the greater the differences between clusters and the more concentrated the data within the clusters, indicating a better clustering effect. The cluster number optimization step calculates the CH value corresponding to different k values ​​and selects the k with the largest CH value as the optimal number of clusters.

[0031] As an improvement, in step S2, when imputing missing values ​​in the multi-dimensional raw data, the mean imputation method, median imputation method, or K-nearest neighbor imputation method are used. Furthermore, during data cleaning, outlier data is identified and removed using the 3σ principle or box plot method. The 3σ principle for judging outlier data is based on the following: if the data value z satisfies |z-μ... z |>3σ z ,

[0032] Then z is determined to be abnormal data, where μ z σ represents the mean of all data values ​​under a certain descriptive dimension. z This represents the standard deviation of all data values ​​under this descriptive dimension. The purpose of this formula is to identify data that deviates from the mean by more than 3 times the standard deviation based on the normal distribution characteristics of the data. It can quickly filter out extreme outliers, ensure the accuracy and reliability of the data after data cleaning, and provide a high-quality data foundation for subsequent standardization processing.

[0033] As an improvement, a data storage step is also included: during the execution of steps S1 to S4, the multi-dimensional raw data, standardized multi-dimensional data, data clustering information, and statistical analysis results are stored respectively. A distributed storage architecture is used for data storage, supporting real-time data reading, writing, and backup. Furthermore, the data is encrypted before storage to ensure data security. The data encryption process uses a symmetric encryption algorithm, with the encryption formula: C = E(K, M).

[0034] Where C represents the encrypted ciphertext data, E represents the symmetric encryption algorithm (such as the AES-256 algorithm), K represents the encryption key, and M represents the plaintext data to be encrypted (multi-dimensional original data, standardized multi-dimensional data, etc.). The function of this formula is to convert plaintext data M into ciphertext data C that cannot be directly read using the encryption key K and the preset encryption algorithm E. Even if the data is illegally obtained during storage or transmission, the original data cannot be decrypted without the key K, thus effectively ensuring data security.

[0035] As an improvement, in step S5, the visualization output uses line charts, bar charts, heatmaps, or cluster scatter plots. Structured data formats include Excel spreadsheets, CSV files, or JSON formats, and the user can select the appropriate output format based on their needs. Exporting and sharing the output results are also supported. When outputting the cluster scatter plot, a two-dimensional projection method is used to display the multi-dimensional data clustering results. The projection calculation uses the first and second principal components in Principal Component Analysis (PCA). The basic formula for calculating the first principal component PC1 is: PC1 = w1x1 + w2x2 + ... + w m x m ,

[0036] Where PC1 represents the projection values ​​of the multi-dimensional data onto the first principal component, w1, w2, ..., w m The values ​​represent the weight coefficients (calculated by the PCA algorithm) corresponding to each descriptive dimension, x1, x2, ..., x m This formula represents the standardized data values ​​of a data object across m descriptive dimensions. Its function is to project the standardized data of m dimensions (m≥2) onto the first principal component of a one-dimensional plane, retaining the main information of the multi-dimensional data. Combined with the second principal component, it can display the clustering distribution of multi-dimensional data on a two-dimensional plane, making it easier for users to intuitively observe the data clustering effect.

[0037] The advantages of this invention compared to existing technologies are as follows: In terms of data quality, it integrates data from different channels through multi-source data interfaces, unifies the format and eliminates dimensional differences by using standardization and normalization formulas, removes outliers by combining the 3σ principle and fills in missing values ​​by using various imputation methods, thus completely solving the problem of low data quality in traditional methods and laying a solid foundation for subsequent analysis.

[0038] Cluster analysis accuracy has been significantly improved. The optimal number of clusters is adaptively determined using the silhouette coefficient and Calinski-Harabasz index, avoiding subjective bias from manual settings and achieving precise data grouping, allowing analysis to focus on relevant data. Statistical analysis is more in-depth, conducting multi-dimensional statistics on each cluster to uncover unique group characteristics and dimensional correlations, providing precise guidance for decision-making.

[0039] The output results are adaptable to diverse needs, with visual formats for intuitive observation, structured formats supporting secondary analysis, and customizable content to enhance usability and flexibility. Meanwhile, distributed storage, AES-256 encryption, and SHA-256 verification provide comprehensive security, while modular and automated design lowers the technical barrier, making it easy to promote and providing efficient and reliable support for data-driven decision-making across various industries. Attached Figure Description

[0040] Figure 1 This is a system framework diagram of the statistical multidimensional data processing system and method that combines cluster analysis according to the present invention.

[0041] Figure 2 This is a flowchart of the statistical multidimensional data processing system and method that combines cluster analysis according to the present invention. Detailed Implementation

[0042] To facilitate understanding of this application, a more complete description will be provided below with reference to the accompanying drawings, which illustrate embodiments of the present application. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that the disclosure of this application will be thorough and complete.

[0043] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0044] It is understood that spatial relation terms such as "below," "under," "below," "below," "above," "over," etc., can be used here to describe the relationship between one element or feature shown in the figure and other elements or features. It should be understood that, in addition to the orientation shown in the figure, spatial relation terms also include different orientations of the device in use and operation. For example, if the device in the figure is flipped, the element or feature described as "below" or "under" or "below" of the other element or feature will be oriented "over" the other element or feature. Therefore, the exemplary terms "below" and "under" can include both upper and lower orientations. Furthermore, the device may also include other orientations, such as being rotated 90 degrees or other orientations, and the spatial descriptive terms used herein will be interpreted accordingly.

[0045] It should be noted that when one element is considered to be "connected" to another element, it can be directly connected to the other element or connected to the other element through an intermediary element. In the following embodiments, "connection" should be understood as "electrical connection," "communication connection," etc., if the connected circuits, modules, units, etc., have the transmission of electrical signals or data between them.

[0046] When used herein, the singular forms of “a,” “an,” and “the” may also include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “comprising,” “including,” or “having,” etc., specify the presence of the stated feature, whole, step, operation, component, part, or combination thereof, but do not preclude the possibility of the presence or addition of one or more other features, wholes, steps, operations, components, parts, or combinations thereof.

[0047] Referring to the attached diagram, the statistical multidimensional data processing system and methods based on cluster analysis include:

[0048] The data acquisition module is used to acquire multi-dimensional raw data. The multi-dimensional raw data has two or more independent descriptive dimensions. Each descriptive dimension corresponds to an attribute or feature variable of the data. The data object is defined and characterized by multiple descriptive dimensions.

[0049] The data preprocessing module is communicatively connected to the data acquisition module and is used to clean, standardize, and fill in missing values ​​for the multi-dimensional raw data to obtain standardized multi-dimensional data.

[0050] The clustering analysis module is communicatively connected to the data preprocessing module and is used to perform similarity grouping processing on the standardized multi-dimensional data based on a preset clustering algorithm to obtain multiple data clusters. The similarity grouping processing is based on the feature differences of data objects in each descriptive dimension.

[0051] The statistical processing module is communicatively connected to the cluster analysis module and is used to perform statistical analysis on each of the data clusters. The statistical analysis includes at least two of the following: mean calculation, analysis of variance, and correlation analysis, and outputs the statistical analysis results corresponding to each data cluster.

[0052] The results output module is communicatively connected to the statistical processing module and is used to output the data clustering information and corresponding statistical analysis results in a visual or structured data format.

[0053] The data acquisition module includes a multi-source data interface, which supports at least two data acquisition methods: database import, API interface call, and file upload. It can also perform format unification processing on multi-dimensional raw data from different sources. During the format unification process, for numerical descriptive dimension data, a standardized formula is used for preliminary dimensional unification. The standardized formula is:

[0054] Where, x i Let x represent the i-th original data value in a descriptive dimension of the multi-dimensional original data, μ represent the mean of all original data values ​​in that descriptive dimension, σ represent the standard deviation of all original data values ​​in that descriptive dimension, and x represent the mean of all original data values ​​in that descriptive dimension. i′ This represents the i-th data value after standardization. The purpose of this formula is to convert numerical descriptive dimensional data of different magnitudes to the same magnitude range, so as to avoid the accuracy of subsequent data format unification due to differences in units.

[0055] The clustering analysis module includes at least one of the following preset clustering algorithms: K-means clustering, hierarchical clustering, and density clustering. The module also includes a cluster number optimization unit, which adaptively adjusts the number of clusters based on the silhouette coefficient or the Calinski-Harabasz index to determine the optimal number of data clusters. The silhouette coefficient is calculated using the following formula:

[0056]

[0057] In the formula, s(i) represents the silhouette coefficient of the i-th data object, a(i) represents the average distance between the i-th data object and all other data objects in the same cluster, and b(i) represents the average distance between the i-th data object and all data objects in the nearest different clusters. The purpose of this formula is to quantitatively evaluate the compactness of the data object within its own cluster and its separation from other clusters. The closer s(i) is to 1, the better the clustering effect of the data object. The clustering number optimization unit determines the optimal number of clusters by calculating the average silhouette coefficient of all data objects.

[0058] It also includes a data storage module, which is communicatively connected to the data acquisition module, data preprocessing module, cluster analysis module, and statistical processing module. This module stores the multi-dimensional raw data, standardized multi-dimensional data, data cluster information, and statistical analysis results. The data storage module employs a distributed storage architecture, supporting real-time data read / write and backup. During data backup, a data integrity verification mechanism is used, verifying the consistency between the backup data and the source data through hash value calculation. The hash value calculation uses the formula: H = SHA-256(D).

[0059] Where H represents the SHA-256 hash value of data D, D represents any type of data to be backed up (multi-dimensional raw data, standardized multi-dimensional data, etc.), and SHA-256 represents the cryptographic hash function algorithm; the function of this formula is to convert data D of arbitrary length to be backed up into a hash value H of fixed length (256 bits). By comparing the hash values ​​H of the source data and the backup data, if the two are consistent, it means that the backup data is complete and undamaged, thus ensuring the reliability of the data backup.

[0060] The statistical multidimensional data processing method combining cluster analysis includes the following steps:

[0061] S1. Data acquisition steps: Acquire multi-dimensional raw data, wherein the multi-dimensional raw data has two or more independent descriptive dimensions, each descriptive dimension corresponds to an attribute or feature variable of the data, and the data object is defined and characterized by multiple descriptive dimensions.

[0062] S2. Data preprocessing steps: Clean, standardize and fill missing values ​​in the multi-dimensional raw data, remove abnormal data and convert the data into a uniform dimension to obtain standardized multi-dimensional data;

[0063] S3. Clustering analysis steps: Based on the preset clustering algorithm, the standardized multi-dimensional data is grouped by similarity. Based on the feature differences of data objects in each descriptive dimension, multiple data clusters are obtained.

[0064] S4. Statistical processing steps: Perform statistical analysis on each of the data clusters, including at least two of the following: mean calculation, analysis of variance, and correlation analysis, to obtain the statistical analysis results corresponding to each data cluster;

[0065] S5. Result Output Step: Output the data cluster information and corresponding statistical analysis results in a visualization or structured data format.

[0066] In step S1, when acquiring multi-dimensional raw data, data acquisition is achieved through a multi-source data interface. This multi-source data interface supports at least two data acquisition methods: database import, API interface call, and file upload. After acquiring the data, the multi-dimensional raw data from different sources undergoes format unification processing to ensure consistent data structure. In the format unification processing, for percentage-type descriptive dimension data, a normalization formula is used to convert it to decimal form. The normalization formula is:

[0067] Among them, y j y represents the j-th raw data value (in %) of a certain percentage-type descriptive dimension. j′ This represents the j-th data value (in decimal form) after normalization. The purpose of this formula is to convert percentage data into a uniform decimal format, which facilitates collaborative processing with other numerical descriptive dimension data and avoids data fusion errors caused by format differences.

[0068] In step S3, the preset clustering algorithm includes at least one of K-means clustering, hierarchical clustering, and density clustering. Before performing similarity grouping, a cluster number optimization step is also included: adaptively adjusting the number of clusters based on the silhouette coefficient or the Calinski-Harabasz index to determine the optimal number of data clusters before performing similarity grouping. The Calinski-Harabasz index is calculated using the following formula: In the formula,

[0069] CH represents the Calinski-Harabasz index, SSB represents the sum of squared deviations from the mean among different clusters, SSW represents the sum of squared deviations from the mean within the same cluster, k represents the number of clusters, and n represents the total number of data objects. This formula evaluates the clustering effect by measuring the ratio of inter-cluster differences to intra-cluster differences. The larger the CH value, the greater the differences between clusters and the more concentrated the data within the clusters, indicating a better clustering effect. The cluster number optimization step calculates the CH value corresponding to different k values ​​and selects the k with the largest CH value as the optimal number of clusters.

[0070] In step S2, when imputing missing values ​​in the multi-dimensional raw data, the mean imputation method, median imputation method, or K-nearest neighbor imputation method are used. Furthermore, during data cleaning, outlier data is identified and removed using the 3σ principle or box plot method. The 3σ principle for judging outlier data is based on the following: if the data value z satisfies |z-μ... z |>3σ z ,

[0071] Then z is determined to be abnormal data, where μ z σ represents the mean of all data values ​​under a certain descriptive dimension. z This represents the standard deviation of all data values ​​under this descriptive dimension. The purpose of this formula is to identify data that deviates from the mean by more than 3 times the standard deviation based on the normal distribution characteristics of the data. It can quickly filter out extreme outliers, ensure the accuracy and reliability of the data after data cleaning, and provide a high-quality data foundation for subsequent standardization processing.

[0072] The process also includes a data storage step: during the execution of steps S1 to S4, the multi-dimensional raw data, standardized multi-dimensional data, data clustering information, and statistical analysis results are stored respectively. A distributed storage architecture is used for data storage, supporting real-time data reading, writing, and backup. Furthermore, the data is encrypted before storage to ensure data security. The data encryption process uses a symmetric encryption algorithm, with the encryption formula: C = E(K, M).

[0073] Where C represents the encrypted ciphertext data, E represents the symmetric encryption algorithm (such as the AES-256 algorithm), K represents the encryption key, and M represents the plaintext data to be encrypted (multi-dimensional original data, standardized multi-dimensional data, etc.). The function of this formula is to convert plaintext data M into ciphertext data C that cannot be directly read using the encryption key K and the preset encryption algorithm E. Even if the data is illegally obtained during storage or transmission, the original data cannot be decrypted without the key K, thus effectively ensuring data security.

[0074] In step S5, the visualization output uses line charts, bar charts, heatmaps, or cluster scatter plots. Structured data formats include Excel spreadsheets, CSV files, or JSON formats, and the user can select the appropriate output format based on their needs. Exporting and sharing the output results are also supported. When outputting the cluster scatter plot, a two-dimensional projection method is used to display the multi-dimensional data clustering results. The projection calculation uses the first and second principal components in Principal Component Analysis (PCA). The basic formula for calculating the first principal component PC1 is: PC1 = w1x1 + w2x2 + ... + w m x m ,

[0075] Where PC1 represents the projection values ​​of the multi-dimensional data onto the first principal component, w1, w2, ..., w m The values ​​represent the weight coefficients (calculated by the PCA algorithm) corresponding to each descriptive dimension, x1, x2, ..., x m This formula represents the standardized data values ​​of a data object across m descriptive dimensions. Its function is to project the standardized data of m dimensions (m≥2) onto the first principal component of a one-dimensional plane, retaining the main information of the multi-dimensional data. Combined with the second principal component, it can display the clustering distribution of multi-dimensional data on a two-dimensional plane, making it easier for users to intuitively observe the data clustering effect.

[0076] This invention discloses a statistical multi-dimensional data processing system and method that combines cluster analysis. It aims to achieve in-depth analysis of complex data through the collaborative work of multi-dimensional data acquisition, preprocessing, cluster analysis, and statistical processing, providing accurate data support for decision-making. The specific implementation of this invention will be described in detail below from two aspects: system structure and methodological steps.

[0077] I. Specific Implementation of System Structure:

[0078] The statistical multidimensional data processing system combining cluster analysis described in this invention has a specific structure including a data acquisition module, a data preprocessing module, a cluster analysis module, a statistical processing module, a result output module, and a data storage module. These modules work together to achieve full-process processing of multidimensional data.

[0079] (I) Data Acquisition Module:

[0080] The data acquisition module is the entry point for the system to obtain raw data. Its core function is to collect multi-dimensional raw data with two or more independent descriptive dimensions. These descriptive dimensions may include, but are not limited to, time, space, numerical attributes, and category attributes. For example, in e-commerce user data analysis scenarios, descriptive dimensions may include user age, spending amount, purchase frequency, and location, which together characterize the user's consumption features.

[0081] To accommodate different data sources, the data acquisition module is configured with multi-source data interfaces, specifically supporting the following data acquisition methods:

[0082] Database import: Connect to relational databases such as MySQL, Oracle, and SQL Server, or non-relational databases such as MongoDB and Redis, through interfaces such as JDBC and ODBC, to read multi-dimensional data in batches;

[0083] API Interface Calls: Connect to RESTful APIs or SOAP APIs provided by third-party platforms, such as user behavior data APIs of e-commerce platforms and multi-dimensional meteorological data APIs of meteorological platforms, to obtain structured data in real time;

[0084] File upload: Supports users to upload local files in formats such as Excel, CSV, and JSON through the system interface, and the system automatically parses the multi-dimensional data in the file.

[0085] After acquiring multi-dimensional raw data, the data acquisition module needs to standardize the format of data from different sources. For numerical descriptive data, a standardized formula is used for initial unit unification:

[0086]

[0087] Where, x i Let x represent the i-th original data value in a descriptive dimension of the multi-dimensional original data, μ represent the mean of all original data values ​​in that descriptive dimension, σ represent the standard deviation of all original data values ​​in that descriptive dimension, and x represent the mean of all original data values ​​in that descriptive dimension. i′ This represents the i-th data value after standardization.

[0088] The purpose of this standardization process is to transform numerical data of different magnitudes (such as user age in "years" and spending in "yuan") to a similar magnitude range (usually near a normal distribution with a mean of 0 and a standard deviation of 1). This avoids the excessive influence of one dimension on the overall result during subsequent data fusion due to differences in units of measurement. For example, if "age (18-60 years old)" and "spending amount (100-10000 yuan)" are processed directly, the numerical range of spending amount is much larger than that of age, and it will dominate in subsequent analysis. However, after standardization using the formula above, the magnitudes of the two become more consistent, which can more objectively reflect their respective contributions to the data characteristics.

[0089] For percentage-based descriptive data (such as user repurchase rate, product discount rate, etc.), a normalization formula is used to convert it to decimal form. The formula is as follows:

[0090]

[0091] Among them, y j y represents the j-th raw data value (in %) of a certain percentage-type descriptive dimension. j′ This represents the j-th data value (in decimal form) after normalization.

[0092] For example, if the "user repurchase rate" in the original data is 30%, it will be converted to 0.3 using this formula, so that it is consistent with other numerical data (such as standardized age and consumption amount) in terms of format. This makes it easier for subsequent modules to perform unified calculation and analysis on multi-dimensional data and avoid data fusion errors caused by format differences.

[0093] (II) Data Preprocessing Module:

[0094] The data preprocessing module communicates with the data acquisition module, receiving multi-dimensional raw data that has undergone standardized formatting. This data is then cleaned, standardized, and imputed for missing values, ultimately outputting standardized multi-dimensional data. This step is crucial for ensuring the accuracy of subsequent analysis, as raw data often contains outliers, missing values, and inconsistencies in units.

[0095] Data cleaning: The main purpose is to identify and remove outlier data. This embodiment uses the 3σ principle for outlier detection, based on the following criteria: if the data value z satisfies...

[0096] |z-μ z |>3σ z

[0097] Then z is determined to be abnormal data.

[0098] Where, μ z σ represents the mean of all data values ​​under a certain descriptive dimension. z This represents the standard deviation of all data values ​​under this descriptive dimension.

[0099] The theoretical basis of the 3σ principle is the property of normal distribution. That is, if data follows a normal distribution, approximately 99.7% of the data will fall within the range of the mean plus or minus three standard deviations. Data outside this range can be considered extreme outliers. For example, in the dimension of "user spending amount," if the mean μ... z It is 500 yuan, and the standard deviation σ is 500 yuan. z If the threshold is 100 yuan, then data with a consumption amount greater than 800 yuan (500 + 3 × 100) or less than 200 yuan (500 - 3 × 100) will be identified as outliers. By removing these outliers, we can avoid their interference with subsequent cluster analysis and statistical processing, thus ensuring the reliability of the data.

[0100] Standardization Processing: Building upon the initial standardization in the data acquisition module, further standardization is performed on all numerical descriptive dimensions to ensure that each dimension has the same units of measurement and distribution range. The standardization formula is consistent with the formula used in the data acquisition module, i.e.

[0101]

[0102] By standardizing again, errors that may be introduced during the collection process from different batches of data can be eliminated, making the data more in line with the input requirements of subsequent clustering algorithms.

[0103] Missing value imputation: For missing values ​​in the data (such as an empty "age" field in user information, or a missing "purchase time" in consumption records), this embodiment uses the following method to imput them:

[0104] Mean imputation: For numerical data that follows a normal distribution, the mean of the data in that dimension is used to impute missing values;

[0105] Median filling method: For skewed numerical data, median filling is used to avoid the deviation caused by extreme values ​​affecting the mean;

[0106] K-Nearest Neighbors Imputation: For data that is strongly correlated with other dimensions, the mean of the corresponding dimension of the K most similar data objects is selected to fill the missing data object by calculating the similarity between the missing data object and other data objects.

[0107] By performing the missing value imputation process described above, the integrity of the data can be guaranteed, and the interruption of cluster analysis or statistical processing due to missing values ​​can be avoided.

[0108] (III) Cluster Analysis Module:

[0109] The clustering analysis module communicates with the data preprocessing module, receives standardized multi-dimensional data, and performs similarity grouping processing based on a preset clustering algorithm to obtain multiple data clusters. In this embodiment, the preset clustering algorithms include K-means clustering, hierarchical clustering, and density clustering. Users can select a suitable algorithm based on data characteristics, or the system can automatically recommend the optimal algorithm.

[0110] Taking the K-means clustering algorithm as an example, its core steps include:

[0111] K data objects are randomly selected as the initial cluster centers;

[0112] Calculate the distance between each data object and each cluster center (such as Euclidean distance, Manhattan distance, etc.), and assign the data object to the cluster containing the nearest cluster center;

[0113] The mean of each cluster is recalculated as the new cluster center;

[0114] Repeat steps 2 and 3 until the cluster centers no longer change significantly or the preset number of iterations is reached.

[0115] To determine the optimal number of clusters K, the cluster analysis module includes a cluster number optimization unit, which adaptively adjusts the number of clusters based on the silhouette coefficient or the Calinski-Harabasz index.

[0116] The contour coefficient is calculated using the following formula:

[0117]

[0118] In the formula, s(i) represents the silhouette coefficient of the i-th data object, a(i) represents the average distance between the i-th data object and all other data objects in the same cluster (intra-cluster similarity), and b(i) represents the average distance between the i-th data object and all data objects in the nearest different clusters (inter-cluster similarity).

[0119] The silhouette coefficient s(i) ranges from -1 to 1. A value closer to 1 indicates a higher degree of matching between the data object and its cluster, and a more obvious distinction from other clusters; a value closer to -1 indicates that the data object may have been assigned to the wrong cluster. The clustering number optimization unit calculates the average silhouette coefficient of all data objects and selects the K value with the highest average silhouette coefficient as the optimal number of clusters. For example, when K=3, the average silhouette coefficient is 0.85, and when K=4, it is 0.72. Therefore, the system determines that K=3 is the better number of clusters.

[0120] The Calinski-Harabasz index is another commonly used indicator for evaluating clustering effectiveness. It is calculated using the following formula:

[0121]

[0122] In the formula, CH represents the Calinski-Harabasz index, SSB represents the sum of squared deviations from the mean among different clusters (reflecting inter-cluster differences), SSW represents the sum of squared deviations from the mean among data objects within the same cluster (reflecting intra-cluster differences), k represents the number of clusters, and n represents the total number of data objects.

[0123] The Calinski-Harabasz index assesses clustering effectiveness by measuring the ratio of inter-cluster variance to intra-cluster variance. A higher CH value indicates more significant inter-cluster variance and more concentrated intra-cluster data, resulting in better clustering. The clustering number optimization unit calculates the CH value corresponding to different K values ​​and selects the K with the highest CH value as the optimal number of clusters. For example, when clustering user consumption data, the CH value reaches its maximum when K=5, so the system selects 5 clusters as the optimal grouping result.

[0124] (iv) Statistical Processing Module:

[0125] The statistical processing module communicates with the cluster analysis module, receives data cluster information, and performs statistical analysis on each cluster, including mean calculation, variance analysis, correlation analysis, etc., and outputs the statistical analysis results corresponding to each cluster.

[0126] Mean calculation: Calculate the mean of each cluster across all descriptive dimensions to reflect the central tendency of the cluster's data in that dimension. For example, in the "high-spending users" cluster, calculating indicators such as "average spending amount" and "average purchase frequency" can intuitively reflect the consumption characteristics of this group.

[0127] Analysis of variance (ANOVA): This analyzes the dispersion of data by calculating the variance (or standard deviation) of each cluster across all descriptive dimensions. A larger variance indicates a more dispersed distribution of data along that dimension; a smaller variance indicates a more concentrated distribution of data.

[0128] Correlation analysis: Calculate the correlation coefficient (such as Pearson correlation coefficient) between different descriptive dimensions within the same cluster to analyze the degree of linear association between dimensions. For example, in the "young users" cluster, analyzing the correlation between "spending amount" and "app usage time" can uncover the intrinsic connection between the two.

[0129] The statistical analysis described above can reveal the data characteristics of each cluster, providing refined data analysis support for subsequent decision-making.

[0130] (V) Results Output Module:

[0131] The results output module communicates with the statistical processing module to output data cluster information and corresponding statistical analysis results in a visual or structured data format to meet the needs of different users.

[0132] Visualization output: Results are displayed using line charts, bar charts, heatmaps, or cluster scatter plots. Among these, the cluster scatter plot projects multi-dimensional data onto a two-dimensional plane using Principal Component Analysis (PCA), facilitating intuitive observation of cluster distribution. The basic formula for calculating the first principal component, PC1, is:

[0133] PC1 = w1x1 + w2x2 + ... + w m x m

[0134] Where PC1 represents the projection values ​​of the multi-dimensional data onto the first principal component, w1, w2, ..., w m The values ​​represent the weight coefficients (calculated by the PCA algorithm) corresponding to each descriptive dimension, x1, x2, ..., x m This represents the standardized data values ​​of a data object across m descriptive dimensions.

[0135] The first principal component is the dimension containing the most information in multi-dimensional data. Combined with the second principal component (which is calculated in a similar way to the first principal component but with different weight coefficients), the original features of multi-dimensional data can be preserved to the greatest extent on a two-dimensional plane, clearly showing the distribution and boundaries of different clusters.

[0136] Structured data output formats: Supports Excel spreadsheets, CSV files, or JSON formats, facilitating the import of results into other data analysis tools (such as SPSS, Python data analysis libraries, etc.) for further processing. The system also supports exporting and sharing output results, allowing users to share analysis results via email, cloud storage, and other methods.

[0137] (vi) Data storage module:

[0138] The data storage module communicates with the data acquisition module, data preprocessing module, cluster analysis module, and statistical processing module, respectively, and is used to store multi-dimensional raw data, standardized multi-dimensional data, data cluster information, and statistical analysis results. This module adopts a distributed storage architecture (such as Hadoop HDFS), supports real-time reading, writing, and backup of massive amounts of data, and ensures high availability and reliability of the data.

[0139] Before data storage, the system encrypts the data using a symmetric encryption algorithm (such as AES-256). The encryption formula is as follows:

[0140] C = E(K,M)

[0141] Where C represents the encrypted ciphertext data, E represents the symmetric encryption algorithm, K represents the encryption key, and M represents the plaintext data to be encrypted (such as multi-dimensional raw data, standardized multi-dimensional data, etc.).

[0142] Encryption can convert plaintext data into ciphertext data that cannot be directly read. Even if the data is illegally obtained during storage or transmission, the original data cannot be decrypted without the key K, thus effectively ensuring data security.

[0143] During data backup, the system employs a data integrity verification mechanism, which verifies the consistency between the backup data and the source data through hash value calculation. The hash value calculation uses the following formula:

[0144] H = SHA-256(D)

[0145] Where H represents the SHA-256 hash value of data D, D represents any type of data to be backed up, and SHA-256 represents the cryptographic hash function algorithm.

[0146] The SHA-256 algorithm can convert data D of any length to be backed up into a fixed-length (256-bit) hash value H, and the probability of different data generating the same hash value is extremely low. By comparing the hash values ​​H of the source data and the backup data, if they match, it means that the backup data is complete and undamaged, ensuring the reliability of the data backup.

[0147] II. Specific Implementation of the Processing Method:

[0148] The statistical multidimensional data processing method combining cluster analysis described in this invention includes the following steps: data acquisition, data preprocessing, cluster analysis, statistical processing, result output, and data storage. These steps are executed sequentially to form a complete data processing flow.

[0149] (I) Data Acquisition Steps (S1):

[0150] In the data collection step, the system obtains multi-dimensional raw data through multi-source data interfaces. Taking user data analysis of an e-commerce platform as an example, the collected multi-dimensional raw data includes descriptive dimensions such as user ID, age, gender, monthly spending amount, monthly purchase frequency, last purchase time, and city. Among these, age, monthly spending amount, and monthly purchase frequency are numerical dimensions, gender and city are categorical dimensions, and last purchase time is a time-based dimension.

[0151] The system first standardizes the format of data from different sources: for numerical dimensions such as monthly spending and monthly purchase frequency, standardized formulas are used.

[0152]

[0153] Standardize the units of measurement; for "user activity" recorded as a percentage (e.g., 30%, 50%), use a normalization formula.

[0154]

[0155] Convert to decimal form (0.3, 0.5); for categorical dimensions (such as gender), convert to dummy variables (such as "male" = 1, "female" = 0); for time-related dimensions (such as the time of the most recent purchase), convert to the number of days since the current time (such as 30 days, 60 days).

[0156] Through the above processing, multi-dimensional raw data of different types and formats are transformed into unified structured data, laying the foundation for subsequent steps.

[0157] (II) Data preprocessing step (S2):

[0158] The data preprocessing step cleans, standardizes, and fills in missing values ​​for the multi-dimensional raw data that has been formatted in a unified manner.

[0159] Data cleaning: Outliers are identified using the 3σ principle. For example, in the "monthly spending" dimension, the mean μ of the monthly spending of all users is calculated. z = 500 yuan, standard deviation σ z = 200 yuan, according to the formula

[0160] |z-μ z |>3σ z

[0161] The threshold for outliers can be calculated as 500±3×200, which is monthly spending amounts less than -100 yuan (unreasonable, actually taken as 0) or greater than 1100 yuan. The system marks and removes these outliers to prevent them from affecting subsequent analysis.

[0162] Standardization: All numerical data dimensions (such as age, monthly spending, monthly purchase frequency, and days since the last purchase) are standardized again using a formula.

[0163]

[0164] The mean of each dimension of the data is set to 0 and the standard deviation to 1, ensuring the comparability of the data.

[0165] Missing value imputation: For missing values ​​in the "Age" dimension, if the data in this dimension follows an approximately normal distribution, the mean is used for imputation; for missing values ​​in the "City" dimension, the mode (i.e., the city with the highest frequency) is used for imputation; for missing values ​​in "Monthly Purchase Count" that are highly correlated with "Monthly Spending Amount", the K-nearest neighbor method is used to select the average of the monthly purchase counts of the 5 users whose monthly spending amount is closest to that of the missing data object for imputation.

[0166] Through data preprocessing steps, high-quality, standardized, multi-dimensional data is ultimately obtained.

[0167] (III) Cluster analysis steps (S3):

[0168] The clustering analysis step performs similarity grouping on standardized multi-dimensional data based on a preset clustering algorithm. In this embodiment, the K-means clustering algorithm is selected, and the optimal number of clusters K is determined through a cluster number optimization unit.

[0169] Cluster number optimization: The silhouette coefficient and Calinski-Harabasz index were calculated for K=2 to K=10 respectively. For example, when K=4, the average silhouette coefficient is 0.82 (close to 1) and the Calinski-Harabasz index is 350 (relatively high), both of which are better than other K values. Therefore, the system determined the optimal number of clusters to be 4.

[0170] Similarity grouping: The K-means clustering algorithm is used to divide user data into four clusters: "high-consumption, high-frequency users", "high-consumption, low-frequency users", "low-consumption, high-frequency users", and "low-consumption, low-frequency users". During the clustering process, users are assigned to the most similar clusters by calculating the distance between users on each standardized dimension (such as Euclidean distance).

[0171] (iv) Statistical processing steps (S4):

[0172] The statistical processing steps involve in-depth statistical analysis of each cluster:

[0173] For the "high-consumption, high-frequency users" cluster, calculate indicators such as average age (e.g., 35 years old), average monthly consumption amount (e.g., 1500 yuan), and average number of monthly purchases (e.g., 8 times) to reflect the basic characteristics of this group;

[0174] Calculate the variance of each dimension. For example, if the variance of age is 25 (standard deviation 5), it indicates that the age distribution of users in this cluster is relatively concentrated.

[0175] Analyzing the correlation between "monthly spending amount" and "monthly purchase frequency" yielded a correlation coefficient of 0.8 (strong positive correlation), indicating a significant positive correlation between the group's spending amount and purchase frequency.

[0176] The above analysis clearly reveals the unique characteristics and inherent patterns of each cluster.

[0177] (V) Result Output Steps (S5):

[0178] The results output step outputs cluster information and statistical analysis results in multiple formats:

[0179] Visualization output: A cluster scatter plot is used to show the distribution of the four clusters. The PCA algorithm projects the multi-dimensional data onto a two-dimensional plane. The first principal component, PC1, mainly reflects the consumption capacity (the monthly consumption amount accounts for the highest proportion among the weight coefficients), and the second principal component mainly reflects the purchase frequency (the number of monthly purchases accounts for the highest proportion among the weight coefficients). Different colors in the scatter plot represent different clusters, intuitively showing the distribution and boundaries of each cluster.

[0180] Structured output: Export the statistical analysis results of each cluster to an Excel spreadsheet, including the sample size, mean, variance, correlation coefficient and other indicators for each cluster, which facilitates targeted analysis by operations personnel.

[0181] (vi) Data storage steps:

[0182] Throughout the data processing process, the system stores data at each stage in real time through the data storage module:

[0183] Store multi-dimensional raw data for easy subsequent tracing and reprocessing;

[0184] Store standardized, multi-dimensional data as a backup of the input for cluster analysis;

[0185] It stores cluster information and statistical analysis results, and supports secondary queries and analysis of the results.

[0186] All data is encrypted using the AES-256 algorithm before storage. The encryption formula is as follows:

[0187] C = E(K,M)

[0188] To ensure data security, the system regularly backs up data and verifies the integrity of the backup data using SHA-256 hash values. The formula is...

[0189] H = SHA-256(D)

[0190] Ensuring data reliability and usability. The statistical multidimensional data processing system and method combining cluster analysis described in this invention, through innovative design of the entire process of multidimensional data acquisition, preprocessing, cluster analysis, statistical processing, result output, and data storage, effectively solves the problems of low data quality, insufficient analytical accuracy, poor result usability, and weak security in multidimensional data scenarios of traditional data processing technologies. It provides efficient and reliable technical support for data-driven decision-making in various industries. Its specific beneficial effects can be elaborated in detail from the following dimensions:

[0191] Significantly improving the quality of multi-dimensional data lays a solid foundation for subsequent analysis: In traditional data processing techniques, the collection of multi-dimensional data is often limited to a single data source and lacks effective format standardization and preprocessing mechanisms. This results in raw data often having problems such as chaotic format, large differences in units, and a large number of outliers and missing values. For example, in e-commerce user data analysis scenarios, "user spending amount" imported from the database is in "yuan" units, "user points" obtained from the API interface are in "points" units, and "user activity" uploaded from local files is presented as a percentage. Data with different formats and units are mixed together, which directly affects the accuracy of subsequent analysis. At the same time, outliers such as "negative spending amount" and "age over 150 years old" and missing values ​​such as "blank user occupation field" and "missing purchase time" may exist in the raw data, further reducing the usability of the data.

[0192] This invention addresses the quality issues of multi-dimensional data at their source through multi-source data interfaces and a refined preprocessing mechanism. On one hand, the data acquisition module supports various data acquisition methods such as database import, API calls, and file uploads, enabling the integration of multi-dimensional data from different channels and utilizing standardized formulas.

[0193]

[0194] With normalization formula

[0195]

[0196] Numerical and percentage-based data are converted to a uniform scale and format, eliminating interference from differences in units. For example, after standardizing "consumption amount (yuan)" and "user points (points)," both fall within the range of a mean of 0 and a standard deviation of 1, objectively reflecting their respective contributions to user characteristics. Furthermore, the data preprocessing module utilizes the 3σ principle...

[0197] |z-μ z |>3σ z

[0198] This invention accurately identifies and removes outliers, combining various missing value handling methods such as mean imputation, median imputation, and K-nearest neighbor imputation to fill data gaps and ensure data integrity and reliability. After processing by this invention, the quality of multi-dimensional data is significantly improved, eliminating analysis interruptions caused by formatting issues, outlier interference, and missing values. This provides a high-quality, highly available data foundation for subsequent cluster analysis and statistical processing, fundamentally avoiding the problem of "garbage data in, garbage results out."

[0199] Optimize the accuracy of cluster analysis to achieve precise grouping of multi-dimensional data: Traditional cluster analysis techniques often rely on manually preset cluster numbers, lacking a scientific optimization mechanism. This results in clustering results that are either too coarse (e.g., grouping significantly different user groups into one category) or too fragmented (e.g., splitting similar users into multiple unrelated clusters), failing to accurately reflect the inherent relationships in the data. For example, when clustering customer groups, manually setting K=3 may confuse "high-consumption, high-frequency users" with "high-consumption, low-frequency users," while setting K=8 may split "low-consumption, low-frequency users" into multiple meaningless small clusters, making it difficult to meet the needs of refined analysis.

[0200] This invention completely solves the accuracy problem of traditional cluster analysis by optimizing the number of clustering units. This unit is based on the silhouette coefficient.

[0201]

[0202] Compared with the Calinski-Harabasz index

[0203]

[0204] Two scientific indicators adaptively adjust the number of clusters: the silhouette coefficient, which quantifies the clustering effect by calculating the compactness of data objects within a cluster (a(i)) and the separation between clusters (b(i)), with a value closer to 1 indicating more reasonable clustering; and the Calinski-Harabasz index, which reflects the discriminative power of clusters by comparing the ratio of inter-cluster variance (SSB) to intra-cluster variance (SSW), with a higher value indicating higher clustering accuracy. Through the synergistic effect of these two indicators, the system can automatically select the optimal number of clusters, avoiding the subjectivity and blindness of manual setting. For example, in e-commerce user clustering, the system found that when K=4, the average silhouette coefficient reached 0.82 and the Calinski-Harabasz index reached 350, both significantly better than other K values. This allowed the system to divide users into four clusters: "high-frequency consumption," "high-frequency consumption," "low-frequency consumption," and "low-frequency consumption," with highly similar user characteristics within each cluster and clear and significant differences between clusters. This precise clustering not only accurately reflects the inherent structure of multi-dimensional data, but also provides a clear analytical object for subsequent targeted statistical analysis, allowing data analysis to move from "general and extensive" to "precise and focused".

[0205] Deepening the value of statistical analysis and uncovering the profound correlations between multi-dimensional data: Traditional statistical processing techniques often perform single-dimensional analysis on overall data, lacking refined mining of grouped data. This makes it impossible to discover unique patterns within different groups and profound correlations between dimensions. For example, calculating the overall average of "spending amount" for all users only yields the average spending level of all users, but fails to reveal the correlation between "spending amount" and "purchase frequency" within the "high-spending user" group, nor can it uncover the age distribution characteristics of the "low-spending user" group. The practicality and guidance of the analysis results are weak.

[0206] This invention utilizes a collaborative approach of "cluster analysis + statistical processing," focusing statistical analysis on each independent cluster to unlock the deep value of multi-dimensional data. After obtaining precise clusters, the statistical processing module performs multi-dimensional statistical analysis on each cluster, including mean calculation, variance analysis, and correlation analysis. For the "high-spending, high-frequency user" cluster, the "average spending amount" and "average purchase frequency" are calculated, and the correlation between "spending amount" and "APP usage time" is analyzed, revealing a strong positive correlation (correlation coefficient 0.85), which can provide a basis for the platform to formulate "time-based incentive consumption" strategies. For the "low-spending, low-frequency user" cluster, the "age variance" is calculated (e.g., variance = 64, standard deviation = 8), revealing that this group's age distribution is concentrated between 20-28 years old, allowing for targeted promotional activities targeting younger users. For the "high-spending, low-frequency user" cluster, the relationship between "last purchase time" and "spending amount" is analyzed, revealing that this group mainly engages in "large-amount, low-frequency" consumption, allowing for the design of "member-exclusive large-amount discount" activities to increase purchase frequency. This refined statistical analysis targeting clusters can not only reveal the unique characteristics of each group (such as the mean reflecting central tendency and variance reflecting dispersion), but also uncover potential correlations between different dimensions within the group, upgrading the analysis results from "single shallow layer" to "multi-dimensional deep layer", providing more specific and accurate guidance for decision-making.

[0207] Adapt to diverse application scenarios and enhance the practicality and flexibility of output results:

[0208] Traditional data processing systems often have a single output format, mainly in the form of tables or simple charts, and lack adaptability to different user needs. For example, outputting only an Excel spreadsheet containing all statistical data cannot meet the needs of business personnel to "intuitively observe cluster distribution" or the needs of technical personnel to "import into other tools for further analysis". This results in a limited scope of application of the analysis results and makes it difficult to fully realize their value.

[0209] The output module of this invention adapts to the application needs of different industries and roles through diverse output formats and flexible customization functions. In terms of visualization output, it uses line charts to display the "monthly consumption trend" of each cluster, bar charts to compare the "average consumption amount" of each cluster, and heatmaps to present the "dimensional correlation matrix" of each cluster. In particular, it uses clustering scatter plots (based on the first principal component of the PCA algorithm) to...

[0210] PC1 = w1x1 + w2x2 + ... + w m x m

[0211] By using a two-dimensional projection method (which integrates with the second principal component to achieve multi-dimensional data), business personnel can intuitively observe the distribution boundaries and group size of each cluster, quickly grasping the overall characteristics of the data. In terms of structured output, it supports multiple formats such as Excel, CSV, and JSON. Technical personnel can import the results into tools such as SPSS and Python data analysis libraries for secondary modeling to meet in-depth analysis needs. Simultaneously, the system supports customized output content based on user needs—for example, if the marketing department needs "city distribution of users in each cluster," the system can output the statistical results for that dimension separately; if the operations department needs "discount sensitivity analysis for each cluster," the system can supplement relevant statistical indicators accordingly. This diversified and customized output allows the analysis results to be adapted to application scenarios in multiple industries such as e-commerce, finance, healthcare, and education, meeting the needs of different roles such as business personnel, technical personnel, and decision-makers, significantly improving the practicality and flexibility of the results, and ensuring that the analytical value can be fully implemented in actual business operations.

[0212] Building comprehensive security safeguards to ensure the reliability and security of multi-dimensional data: Traditional data storage and management technologies often suffer from problems such as untimely data backups and imperfect encryption mechanisms, leading to the risk of loss, damage, or leakage of multi-dimensional data during storage and transmission. For example, using only local storage without distributed backups means all data will be lost in the event of equipment failure; failing to encrypt sensitive data (such as user ID numbers and consumption records) poses a security risk of data leakage; and the lack of integrity verification for backup data makes it impossible to confirm whether the backup data is complete and usable. All of these problems affect the reliability and security of data processing.

[0213] This invention constructs a comprehensive data security system through a distributed storage architecture, a high-strength encryption mechanism, and an integrity verification mechanism. For data storage, it adopts the Hadoop HDFS distributed storage architecture, distributing data across multiple nodes and supporting real-time read / write and multi-replica backup (3 replicas by default). Even if one node fails, data can be quickly recovered from other replicas, preventing data loss. For data encryption, it employs the AES-256 symmetric encryption algorithm.

[0214] C = E(K,M) encrypts all stored data (including raw data, standardized data, and statistical results), converting plaintext data (M) into ciphertext data (C) that cannot be directly read. Only authorized personnel with the key (K) can decrypt it, effectively preventing data leakage. For data backup, SHA-256 hash value verification is used.

[0215] H = SHA-256(D) converts the data to be backed up (D) into a fixed 256-bit hash value (H). After backup, the hash values ​​of the source data and the backup data are compared. If they match, it means the backup is complete and undamaged, ensuring the reliability of the backup data. This comprehensive security mechanism ensures that multi-dimensional data is protected throughout the entire process from collection, storage, backup to transmission, solving the problem of insufficient data security and reliability in traditional technologies and providing a solid guarantee for the stable operation of data processing.

[0216] Lowering technical barriers and operating costs, and improving the ease of use and scalability of the system: Traditional data processing technologies often require professional technicians to perform complex parameter configurations and operations, such as manually writing clustering algorithm code and manually adjusting statistical analysis dimensions. This is not only difficult to operate, but also requires a lot of manpower and time costs, resulting in high technical barriers and making it difficult to promote and apply in small and medium-sized enterprises or non-technical departments.

[0217] This invention significantly reduces technical barriers and operating costs through modular design and automated processing, enhancing the system's usability and scalability. In terms of system architecture, it adopts modular decomposition (data acquisition, preprocessing, cluster analysis, statistical processing, etc., operate independently yet collaboratively). Users do not need to understand the internal technical details of each module; they only need to perform simple operations through a visual interface. For example, in the data acquisition stage, users select the data source (database / API / file) via a drop-down menu, and the system automatically completes the format standardization. In the cluster analysis stage, selecting "automatically optimize the number of clusters" automatically calculates the silhouette coefficient and Calinski-Harabasz index to determine the optimal K value. In the results output stage, users select the desired output format (scatter plot / Excel / JSON), and the system automatically generates and exports the results. The entire process requires no user coding or manual adjustment of complex parameters; non-technical personnel can operate it proficiently after simple training. Simultaneously, the system supports batch processing and scheduled tasks, such as setting "automatically collect and analyze user data from the previous day every morning at midnight," significantly reducing the time cost of manual operation. This low-barrier, highly automated design allows the system to be used not only in the technical departments of large enterprises, but also in the non-technical departments of small and medium-sized enterprises, government departments, research institutions, and other entities, thus expanding the scope of application and enhancing its practical promotion value.

[0218] In summary, the statistical multidimensional data processing system and method combining cluster analysis of this invention effectively solves many pain points of traditional data processing technologies through innovations in improving data quality, optimizing clustering accuracy, deepening statistical mining, adapting to diverse scenarios, ensuring data security, and lowering the barrier to entry. It achieves a leap from "extensive and inefficient" to "precise and efficient" multidimensional data processing, providing strong technical support for data analysis and decision-making in various industries, and has significant technical value and practical application significance.

Claims

1. A statistical multi-dimensional data processing system combined with clustering analysis, comprising: a data acquisition module configured to acquire multi-dimensional raw data, the multi-dimensional raw data having two or more independent description dimensions, each description dimension corresponding to an attribute or characteristic variable of the data, and the data object being defined and characterized by the multiple description dimensions; a data preprocessing module communicatively connected to the data acquisition module and configured to clean, standardize, and fill in missing values of the multi-dimensional raw data to obtain standardized multi-dimensional data; a clustering analysis module communicatively connected to the data preprocessing module and configured to perform similarity grouping processing on the standardized multi-dimensional data based on a preset clustering algorithm to obtain a plurality of data clustering clusters, the similarity grouping processing being based on the feature difference of the data object in each description dimension; a statistical processing module communicatively connected to the clustering analysis module and configured to perform statistical analysis on each data clustering cluster, the statistical analysis including at least two of mean calculation, variance analysis, and correlation analysis, and outputting statistical analysis results corresponding to each data clustering cluster; and a result output module communicatively connected to the statistical processing module and configured to output the data clustering cluster information and the corresponding statistical analysis results in a visual or structured data format. 2.The statistical multi-dimensional data processing system combined with clustering analysis of claim 1, wherein the data acquisition module is further configured to acquire multi-dimensional raw data of a plurality of data objects, and the data preprocessing module is further configured to clean, standardize, and fill in missing values of the multi-dimensional raw data of the plurality of data objects to obtain standardized multi-dimensional data of the plurality of data objects. 3.The statistical multi-dimensional data processing system combined with clustering analysis of claim 1, wherein the preset clustering algorithm in the clustering analysis module includes at least one of a K-means clustering algorithm, a hierarchical clustering algorithm, and a density clustering algorithm, and the clustering analysis module is further provided with a clustering number optimization unit, the clustering number optimization unit being configured to adaptively adjust the clustering number based on a silhouette coefficient or a Calinski-Harabasz index to determine an optimal data clustering cluster number, and the silhouette coefficient calculation formula being: s (i) = b (i) - a (i) / max [a (i), b (i) ] wherein s (i) represents the silhouette coefficient of the i th data object, a (i) represents the average distance of the i th data object from all other data objects in the same clustering cluster, and b (i) represents the average distance of the i th data object from all data objects in the nearest different clustering cluster; the formula is used to quantitatively evaluate the compactness of the data object in the clustering cluster and the separation degree from other clustering clusters, and the closer s (i) is to 1, the better the clustering effect of the data object, and the clustering number optimization unit determines the optimal clustering number by calculating the average silhouette coefficient of all data objects. 4.The statistical multi-dimensional data processing system combined with clustering analysis of claim 1, wherein the data acquisition module is further configured to acquire multi-dimensional raw data of a plurality of data objects, and the data preprocessing module is further configured to clean, standardize, and fill in missing values of the multi-dimensional raw data of the plurality of data objects to obtain standardized multi-dimensional data of the plurality of data objects. ​ ​ ​ ​ The data collection module comprises a multi-source data interface, which supports at least two data acquisition modes of database import, API interface calling and file uploading, and can perform format uniform processing on multi-dimensional original data of different sources; in the format uniform processing process, for numerical value type description dimension data, a standardization formula is used for preliminary dimension uniform, and the standardization formula is: wherein x i represents the i-th raw data value of a certain description dimension in the multi-dimensional raw data, μ represents the mean of all raw data values under the description dimension, σ represents the standard deviation of all raw data values under the description dimension, x i′ represents the i-th data value after standardization; the function of the formula is to convert the data of the numerical description dimension of different magnitudes to the same magnitude range, avoiding the influence of the dimensional difference on the accuracy of subsequent data format unification. ​ ​ ​ ​ Also include a data storage module, the data storage module is respectively connected with the data acquisition module, data preprocessing module, clustering analysis module, statistical processing module communication, for storing the multi-dimensional raw data, standardized multi-dimensional data, data clustering cluster information and statistical analysis results, and the data storage module adopts distributed storage architecture, support data real-time read and write and backup; In the data backup process, the data integrity check mechanism is adopted, the consistency of the backup data and the source data is verified by hash value calculation, the hash value calculation adopts the formula: H = SHA-256 (D), Wherein, H represents the SHA-256 hash value of data D, D represents any type of data to be backed up (multi-dimensional raw data, standardized multi-dimensional data, etc.), SHA-256 represents the password hash function algorithm; The function of this formula is to convert any length of backup data D into a fixed length (256 bits) hash value H, by comparing the hash value H of the source data and the backup data, if they are consistent, it means that the backup data is complete and undamaged, which guarantees the reliability of data backup.

5. The statistical multi-dimensional data processing method combined with clustering analysis, characterized in that: It comprises the following steps: S1. Data acquisition step: obtaining multi-dimensional raw data, the multi-dimensional raw data has two or more independent description dimensions, each description dimension corresponds to a data attribute or characteristic variable, and the data object is defined and described by multiple description dimensions; S2. Data preprocessing step: cleaning, standardizing and missing value filling of the multi-dimensional raw data, removing abnormal data and converting the data to a unified dimension to obtain standardized multi-dimensional data; S3. Clustering analysis step: similarity grouping processing of the standardized multi-dimensional data based on a preset clustering algorithm, and dividing a plurality of data clustering clusters based on the feature difference of the data object in each description dimension; S4. Statistical processing step: statistical analysis is carried out on each data clustering cluster respectively, the statistical analysis at least includes two of mean calculation, variance analysis and correlation analysis, and the statistical analysis result corresponding to each data clustering cluster is obtained; S5. Result output step: outputting the data clustering cluster information and the corresponding statistical analysis result in visual or structured data format.

6. The statistical multi-dimensional data processing method combined with clustering analysis according to claim 5, characterized in that: In the step S1, when the multi-dimensional original data is acquired, the data acquisition is realized through a multi-source data interface, the multi-source data interface supports at least two data acquisition modes in database import, API interface calling and file uploading, and after the data is acquired, the multi-dimensional original data of different sources is subjected to format uniform processing to ensure that the data structures are consistent; in the format uniform processing, for the percentage type description dimension data, a normalization formula is adopted to convert into a decimal form, and the normalization formula is: wherein, y j represents the jth raw data value of a certain percentage type descriptive dimension (unit: %), y j′ represents the jth normalized data value (decimal form); the function of this formula is to convert percentage data into a unified decimal format, facilitating collaborative processing with other numerical descriptive dimension data, and avoiding data fusion errors caused by format differences.

7. The statistical multi-dimensional data processing method combined with clustering analysis according to claim 5, characterized in that: In the step S3, the preset clustering algorithm includes at least one of K-means clustering algorithm, hierarchical clustering algorithm and density clustering algorithm, and further includes a clustering number optimization step before performing the similarity grouping processing: based on the silhouette coefficient or Calinski-Harabasz index, the clustering number is adaptively adjusted, and after the optimal data clustering cluster number is determined, the similarity grouping processing is performed; wherein the Calinski-Harabasz index calculation adopts the formula: In the formula, CH represents the Calinski-Harabasz index, SSB represents the sum of squares of deviations from the mean between different clustering clusters, SSW represents the sum of squares of deviations from the mean of data objects within the same clustering cluster, k represents the number of clustering clusters, and n represents the total number of data objects; the formula is used to evaluate the clustering effect by measuring the ratio of the difference between the clustering clusters and the difference within the cluster, the larger the CH value, the larger the difference between the clustering clusters, the more concentrated the data within the cluster, and the better the clustering effect, and the clustering number optimization step selects the k corresponding to the maximum CH value as the optimal clustering cluster number by calculating the CH value corresponding to different k values.

8. The statistical multi-dimensional data processing method combined with clustering analysis according to claim 5, characterized in that: In the step S2, when the missing value filling processing is performed on the multi-dimensional original data, the mean filling method, the median filling method or the K-nearest neighbor filling method is adopted, and in the data cleaning process, the abnormal data is identified and removed through the 3σ principle or the box plot method; wherein the abnormal data judgment basis of the 3σ principle is: if the data value z satisfies |z-μ z |>3σ z , If z is greater than 3, then the data is determined to be abnormal, where μ z represents the mean of all data values in a certain description dimension, and σ z represents the standard deviation of all data values in the description dimension; the formula is used to determine the data that deviates from the mean by more than 3 times the standard deviation as abnormal data, quickly filter out extreme outliers, ensure the accuracy and reliability of the data after cleaning, and provide a high-quality data basis for subsequent standardization processing.

9. The statistical multi-dimensional data processing method combined with clustering analysis according to claim 5, characterized in that: The data storage step is further included: during the execution of the steps S1 to S4, the multi-dimensional original data, the standardized multi-dimensional data, the data clustering cluster information and the statistical analysis results are respectively stored, distributed storage architecture is used for data storage, real-time reading and writing and backup of data are supported, and data encryption processing is performed before data storage to ensure data security; the data encryption processing uses a symmetric encryption algorithm, and the encryption formula is: C = E(K, M), wherein C represents encrypted ciphertext data, E represents a symmetric encryption algorithm (such as AES-256 algorithm), K represents an encryption key, and M represents plaintext data (multi-dimensional original data, standardized multi-dimensional data, etc.) to be encrypted; the formula is used to convert the plaintext data M into ciphertext data C that cannot be directly read by using the encryption key K and the preset encryption algorithm E, so that even if the data is illegally obtained during storage or transmission, the original data cannot be decrypted and obtained without the key K, and the data security is effectively ensured.

10. The statistical multi-dimensional data processing method combined with clustering analysis according to claim 5, characterized in that: In the step S5, the line chart, the column chart, the heat map or the clustering scatter plot is used in the form of visual output, the structured data format includes the Excel table, the CSV file or the JSON format, and the corresponding output form can be selected according to the user demand, and the output result can be exported and shared. In outputting the cluster scatter diagram, a two-dimensional projection mode is adopted to display the multi-dimensional data clustering result, and the first principal component and the second principal component in principal component analysis (PCA) are adopted for projection calculation, and a calculation basis formula of the first principal component PC1 is: PC1=w1x1+w2x2+…+w m x m , wherein, PC1 represents the projection value of the multi-dimensional data on the first principal component, w1, w2, …, w m represent the weight coefficients corresponding to each description dimension (calculated by the PCA algorithm), x1, x2, …, x m represent the standardized data values of the data object on the m description dimensions; the function of this formula is to project the m-dimensional (m≥2) standardized data onto the first principal component of one dimension, retain the main information of the multi-dimensional data, and combine the second principal component to show the clustering distribution of the multi-dimensional data on the two-dimensional plane, facilitating the user to intuitively observe the data clustering effect.

Images