Data analysis method, data analysis system, storage medium and program product

Abstract

A data analysis method, a data analysis system, a storage medium and a program product, relating to the field of data analysis. In implementing the present application, the data analysis system calculates the resource occupancy ratio and dynamically selects a single-process or multi-process processing mode based on a preset ratio threshold judgment, effectively balancing resource utilization and processing efficiency. When the data set to be analyzed is small, a single-process mode is adopted, and the data set to be analyzed is directly converted into an R language data frame for processing, avoiding the additional overhead caused by multiple processes; when the data set to be analyzed is large, multiple R processes are created based on the current number of CPU cores and a TCP / IP connection pool is established, and multi-core computing power is fully utilized through data sharding and parallel computing. This adaptive processing mode not only ensures the processing efficiency of small data sets, but also effectively responds to the computing needs of large data sets, thereby improving the performance and resource utilization of data analysis as a whole.

Description

Technical Field

[0001] The present application relates to the field of data analysis, and in particular to a data analysis method, a data analysis system, a storage medium, and a program product. Background Art

[0002] With the advent of the big data era, the application of data analysis and processing is becoming increasingly widespread. Rapid and efficient analysis and processing of massive amounts of data can obtain valuable information and support decision-making. This is especially true in fields such as finance, healthcare, and industry, where the requirements for real-time and accurate data analysis and processing are becoming increasingly stringent.

[0003] Currently, data analysis and processing methods typically use a single-machine, single-process approach. Upon receiving a data analysis request, the system directly loads all data into memory and then calls statistical analysis functions for calculations. This approach is relatively simple to implement, can meet basic analysis and processing needs when the data volume is small, and offers low development and maintenance costs.

[0004] However, as data scale continues to expand, single-process processing on a single machine can lead to severe performance degradation. When the amount of data to be analyzed and processed exceeds the system's available memory, frequent memory swapping significantly slows down computation, resulting in low system resource utilization and poor processing efficiency. Summary of the Invention

[0005] The present application provides a data analysis method, a data analysis system, a storage medium, and a program product for improving the performance and resource utilization of data analysis.

[0006] In a first aspect, the present application provides a data analysis method, which is applied to a data analysis system, and the method includes: obtaining the current number of CPU cores and a task analysis request input by a user, the task analysis request including a data set to be analyzed, dividing the physical memory size occupied by the data set to be analyzed by the current available physical memory size to obtain a resource occupancy ratio; if the resource occupancy ratio is less than a preset ratio threshold, converting the data set to be analyzed into an R language data frame, calling an R function to process the R language data frame, and obtaining a first analysis result; if the resource occupancy ratio is greater than or equal to the preset ratio threshold, determining the number of sub-processes based on the current number of CPU cores, creating R processes equal to the number of sub-processes, and establishing a TCP / IP connection pool for the R processes; calculating the maximum allowable size of a single data slice based on the current available physical memory size, and dividing the data set to be analyzed into multiple data blocks according to the maximum allowable size; sending the data blocks to the corresponding R processes through the TCP / IP connection pool for parallel calculation to obtain multiple partial calculation results, merging the multiple partial calculation results to obtain a second analysis result; and visually displaying the first analysis result or the second analysis result.

[0007] By adopting the above technical solution, the data analysis system calculates the resource utilization ratio and dynamically selects single-process or multi-process processing mode based on a preset ratio threshold, effectively balancing resource utilization and processing efficiency. When the data set to be analyzed is small, the single-process mode is adopted, and the data set to be analyzed is directly converted into an R language data frame for processing, avoiding the additional overhead brought by multiple processes. When the data set to be analyzed is large, multiple R processes are created based on the current number of CPU cores and a TCP / IP connection pool is established. Through data sharding and parallel computing, the multi-core computing power is fully utilized. This adaptive processing mode not only ensures the processing efficiency of small data sets, but also effectively meets the computing needs of large data sets, improving the performance of data analysis and resource utilization as a whole.

[0008] In combination with some embodiments of the first aspect, in some embodiments, after the step of obtaining the current number of CPU cores and the task analysis request input by the user, the method also includes: obtaining the inherent delay of calling the R function based on rpy2 and the time overhead of memory data sharing, and calculating the total delay of the single-process call; obtaining the network delay of data transmission based on Rserve and the time overhead of actually processing data in the R process, and calculating the total delay of multiple process calls; comparing the total delay of the single-process call and the total delay of the multiple process calls; if the total delay of the single-process call is less than the total delay of the multiple process calls, the single-process mode is adopted; if the total delay of the single-process call is greater than or equal to the total delay of the multiple process calls, the multi-process mode is adopted.

[0009] By employing this technical solution, the data analysis system calculates the inherent latency of calling R functions using rpy2, the time overhead of memory data sharing, the network latency of data transmission using Rserve, and the time overhead of actual data processing in the R process. By comparing the actual latency overhead of single-process and multi-process modes, the data analysis system can accurately assess the performance of the current task under different processing modes, thereby selecting the mode with the lowest total latency. This adaptive selection mechanism based on actual latency measurement avoids the performance loss that may result from using a fixed mode. It can select the optimal processing strategy based on the specific task characteristics and system status, effectively improving data processing efficiency.

[0010] In combination with some embodiments of the first aspect, in some embodiments, the maximum allowable size of a single data shard is calculated based on the currently available physical memory size, specifically including: inputting the preset shard size, the empirical coefficient and the currently available physical memory size into the shard size calculation formula to obtain the maximum allowable size of a single data shard; the shard size calculation formula is: B=min(Bmax, k×M); wherein B is used to represent the maximum allowable size of a single data shard, Bmax is used to represent the preset shard size, k is used to represent the empirical coefficient, and M is used to represent the currently available physical memory size.

[0011] By adopting the above technical solution, the data analysis system determines the maximum allowable size of a single data shard using the shard size calculation formula B = min(Bmax, k × M). This shard size calculation formula comprehensively considers three key factors: the preset shard size, the current available physical memory size, and an empirical coefficient. This balances security and efficiency, avoiding the risk of system crashes caused by overly large shards while also allowing for flexible adjustments based on system resource availability, fully utilizing available memory and improving the robustness and efficiency of data processing.

[0012] In combination with some embodiments of the first aspect, in some embodiments, after the step of sending data blocks to corresponding R processes for parallel computing through a TCP / IP connection pool, the method also includes: calculating the network transmission time of a single data block; based on the total size of the data set to be analyzed and the network transmission time, calculating the total data transmission delay; when the current number of CPU cores is greater than a preset value, dividing the total data transmission delay by the current number of CPU cores to obtain the average delay of the R process.

[0013] By employing this technical solution, the data analysis system calculates the network transmission time of a single data block, combines this with the total size of the dataset to be analyzed to calculate the total data transmission latency, and then calculates the average latency of the R process when a large number of CPU cores are used. This multi-level performance monitoring mechanism accurately assesses data transmission efficiency and identifies potential performance bottlenecks. In multi-core systems, calculating average latency can be used to assess the effectiveness of parallel processing and determine whether the advantages of multiple cores are being fully utilized.

[0014] In combination with some embodiments of the first aspect, in some embodiments, when the current number of CPU cores is greater than a preset value, after the step of dividing the total data transmission delay by the current number of CPU cores to obtain the average delay of the R process, the method also includes: obtaining the processing success rate, memory utilization rate and calculation time of the R process to complete data analysis for each data block; dividing the R process into high-performance process groups and low-performance process groups according to the processing success rate, memory utilization rate and calculation time; reclaiming the memory space of the R process in the low-performance process group, initializing the operating environment of the R process in the low-performance process group, and reloading the function library of the R process in the low-performance process group.

[0015] By employing this technical solution, the data analysis system monitors multiple metrics, including the R process's processing success rate, memory usage, and computation time. This allows for a comprehensive assessment of each R process's performance and categorizes each into high-performance and low-performance groups. The data analysis system then performs specialized optimizations for the low-performance process group, including memory recycling, environment initialization, and library reloading. This differentiated processing strategy, based on performance grouping, ensures the continued stable operation of high-performance processes while enabling the timely identification and optimization of low-performance processes. This prevents individual processes from dragging down overall performance, thereby improving overall operational efficiency and stability.

[0016] In combination with some embodiments of the first aspect, in some embodiments, after the step of obtaining the current number of CPU cores and the task analysis request input by the user, the method also includes: obtaining the variable type of the data set to be analyzed; determining whether the data set to be analyzed includes categorical variables; if categorical variables are included, automatically selecting the chi-square test path, performing the chi-square independence test, and obtaining a first verification result; if categorical variables are not included, automatically selecting the variance analysis path, performing one-way variance analysis, and obtaining a second verification result; based on the first verification result or the second verification result, calculating the effect value, and generating a statistical test report.

[0017] By employing this technical solution, the data analysis system automatically identifies the variable type of the dataset to be analyzed and selects the appropriate statistical test method based on whether categorical variables are included: if categorical variables are included, the data analysis system automatically performs a chi-square independence test; if not, a one-way analysis of variance is performed. This automated analysis path selection not only reduces the user's operational difficulty but also ensures the scientific nature and accuracy of the statistical analysis method. Simultaneously, the data analysis system calculates effect values and generates a complete statistical test report, providing users with comprehensive data analysis results and significantly enhancing the professionalism and usability of data analysis.

[0018] In combination with some embodiments of the first aspect, in some embodiments, after the step of sending data blocks to corresponding R processes for parallel computing through the TCP / IP connection pool, the method also includes: periodically detecting the running status of the R process; when the R process exits abnormally, obtaining the data processing progress of the R process, and saving the completed data analysis results to the intermediate result cache area, and recording the error type of the abnormal exit.

[0019] By employing this technical solution, the data analysis system periodically monitors the running status of the R process, enabling timely detection of process anomalies. When an R process exits abnormally, the data analysis system automatically retrieves the data processing progress, saves the completed data analysis results to the intermediate result buffer, and records the error type. This meticulous exception handling mechanism not only ensures that completed work results are not lost, but also provides a valuable reference for subsequent fault analysis by recording error types, significantly improving fault tolerance and reliability. Even in the event of anomalies, it maximizes the protection of data and analysis results, ensuring the continuity and recoverability of data analysis tasks.

[0020] In a second aspect, an embodiment of the present application provides a data analysis system, which includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the data analysis system to execute the method described in the first aspect and any possible implementation method of the first aspect.

[0021] In a third aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when the computer program product is run on a data analysis system, enables the data analysis system to execute the method described in the first aspect and any possible implementation of the first aspect.

[0022] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium comprising instructions. When the instructions are executed on a data analysis system, the data analysis system executes the method described in the first aspect and any possible implementation of the first aspect.

[0023] It is understandable that the data analysis system provided in the second aspect, the computer program product provided in the third aspect, and the computer storage medium provided in the fourth aspect are all used to execute the methods provided in the embodiments of the present application. Therefore, the beneficial effects that can be achieved can be referenced to the beneficial effects of the corresponding methods and will not be repeated here.

[0024] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0025] 1. By adopting the above technical solution, the data analysis system calculates the resource utilization ratio and dynamically selects single-process or multi-process processing mode based on a preset ratio threshold, effectively balancing resource utilization and processing efficiency. When the data set to be analyzed is small, the single-process mode is adopted, and the data set to be analyzed is directly converted into an R language data frame for processing, avoiding the additional overhead brought by multiple processes. When the data set to be analyzed is large, multiple R processes are created based on the current number of CPU cores and a TCP / IP connection pool is established. Through data sharding and parallel computing, the multi-core computing power is fully utilized. This adaptive processing mode not only ensures the processing efficiency of small data sets, but also effectively meets the computing needs of large data sets, improving the performance and resource utilization of data analysis as a whole.

[0026] 2. By employing the above technical solution, the data analysis system determines the maximum allowable size of a single data shard using the shard size calculation formula B = min(Bmax, k × M). This shard size calculation formula comprehensively considers three key factors: the preset shard size, the current available physical memory size, and an empirical coefficient. This balances security and efficiency, mitigating the risk of system crashes caused by overly large shards while also enabling flexible adjustments based on system resource availability, fully utilizing available memory and improving the robustness and efficiency of data processing.

[0027] 3. By employing the aforementioned technical solution, the data analysis system monitors multiple metrics, including the R process's processing success rate, memory usage, and computation time. This allows for a comprehensive assessment of each R process's performance and categorizes each into high-performance and low-performance groups. The data analysis system then performs specialized optimizations for the low-performance process group, including memory recycling, environment initialization, and library reloading. This differentiated processing strategy, based on performance grouping, ensures the continued stable operation of high-performance processes while enabling the timely identification and optimization of low-performance processes. This prevents individual processes from dragging down overall performance, thereby improving overall operational efficiency and stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a flow chart of the data analysis method in the embodiment of the present application;

[0029] Figure 2 This is another flow chart of the data analysis method in the embodiment of the present application;

[0030] Figure 3 It is a schematic diagram of the structure of a physical device of the data analysis system in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The terms used in the following examples of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification of the present application, the singular expressions "a", "an", "above", "the", and "this" are intended to include plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in the present application refers to any or all possible combinations of one or more of the listed items.

[0032] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of the technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0033] The following is a description of the process of the method provided by this implementation. Figure 1 , which is a flow chart of the data analysis method in an embodiment of the present application.

[0034] S101. Obtain the current number of CPU cores and a task analysis request input by a user, where the task analysis request includes a dataset to be analyzed, and divide the physical memory size occupied by the dataset to be analyzed by the current available physical memory size to obtain a resource occupancy ratio.

[0035] Among them, the current number of CPU cores is used to indicate the number of processor cores in the computer that can execute instructions in parallel, including the number of physical cores and logical cores; the task analysis request refers to the task application submitted by the user that requires data analysis and processing, including the data set to be processed and the data analysis and processing requirements; the data set to be analyzed refers to the original data set that needs to be analyzed and processed; the physical memory size refers to the random access memory (RAM) capacity actually installed in the computer; the current available physical memory size refers to the free memory capacity that can be allocated to the system at this moment; the resource utilization ratio refers to the ratio between the memory required for the data set to be analyzed and the available memory of the system.

[0036] When the data analysis system receives a task analysis request submitted by a user, it first needs to assess the system resource status to determine the subsequent processing strategy. Specifically, the data analysis system first obtains the current number of CPU cores through the operating system interface. It also receives and parses the task analysis request input by the user, extracting the dataset to be analyzed. The data analysis system then calculates the memory space required to load the dataset to be analyzed into memory and obtains the available free memory capacity in the system. Finally, the data analysis system divides the physical memory occupied by the dataset to be analyzed by the currently available physical memory to obtain a resource utilization indicator, which is used to determine whether optimization measures such as data sharding are necessary.

[0037] S102: If the resource occupancy ratio is less than a preset ratio threshold, converting the data set to be analyzed into an R language data frame, calling an R function to process the R language data frame, and obtaining a first analysis result;

[0038] Among them, the preset ratio threshold is a pre-set resource usage critical value, which is used to determine whether the data sharding mechanism needs to be enabled; the R language data frame refers to the data structure used in the R language to store two-dimensional tabular data, which supports mixed storage of different types of data; the R function is used to represent various predefined data processing and statistical analysis functions in the R language environment; the first analysis result refers to the data analysis result obtained by direct processing in single-process mode.

[0039] When the data analysis system determines that the scale of the data set to be analyzed is relatively small and will not cause great pressure on system resources, a simple and direct single-process processing mode is adopted. Specifically, first, the data analysis system compares the calculated resource occupancy ratio with the preset ratio threshold. If the resource occupancy ratio is less than the preset ratio threshold, it indicates that the current system resources are sufficient. Next, the data analysis system converts the original data set to be analyzed into a data frame format dedicated to the R language. This process includes data type conversion, missing value processing, etc. Then, the data analysis system calls the corresponding R function to process the R language data frame according to the data analysis and processing requirements, which may include descriptive statistics, hypothesis testing, regression analysis and other operations. Finally, the data analysis system marks the obtained data analysis results as the first analysis results.

[0040] S103: If the resource occupancy ratio is greater than or equal to the preset ratio threshold, the number of child processes is determined based on the current number of CPU cores, a number of R processes equal to the number of child processes is created, and a TCP / IP connection pool is established for the R processes;

[0041] Among them, the number of subprocesses refers to the number of independent R language processes that the data analysis system will create, which is usually determined based on the current number of CPU cores; R process refers to an independently running R language runtime environment instance that can execute R language code and functions; TCP / IP connection pool refers to a resource pool used to manage multiple network connections, supporting data transmission between different processes.

[0042] When the data analysis system finds that the data set to be analyzed is large and may cause great pressure on system resources, it is necessary to enable a parallel processing mechanism. Specifically, first, the data analysis system determines whether the resource occupancy ratio reaches or exceeds the preset ratio threshold. If so, a multi-process processing strategy needs to be adopted. Then, the data analysis system calculates the optimal number of sub-processes based on the current number of CPU cores, and usually sets the number of sub-processes to a multiple or approximate number of the current number of CPU cores. Next, the data analysis system creates a corresponding number of R processes, each of which is an independent computing unit. Finally, the data analysis system establishes a TCP / IP connection pool to manage network communications between these R processes, including establishing connections, maintaining connection status, allocating connection resources, etc., in preparation for subsequent data distribution and result collection.

[0043] S104: Calculate the maximum allowable size of a single data slice based on the current available physical memory size, and divide the data set to be analyzed into multiple data blocks according to the maximum allowable size;

[0044] Among them, a single data shard refers to a smaller data block obtained by splitting a large data set to be analyzed; the maximum allowable size refers to the maximum memory space that a single data shard can occupy; and a data block refers to a data subset obtained by dividing according to the maximum allowable size.

[0045] When the data analysis system needs to process a large-scale data set to be analyzed in parallel, the data must be sharded first. Specifically, first, the data analysis system obtains the current available physical memory size, combines the preset shard size and the empirical coefficient, and calculates the maximum allowable size of a single data shard through the shard size calculation formula. The maximum allowable size must ensure that the sharded data block can be effectively processed by a single R process while avoiding memory pressure. Then, the data analysis system divides the original data set to be analyzed into multiple data blocks of similar size based on the calculated maximum allowable size of a single data shard. The division process will take into account the integrity of the data to avoid dispersing related data into different blocks. The data analysis system will also assign a unique identifier to each data block for subsequent data distribution and result association.

[0046] Optionally, in general, the maximum allowable size of a single data shard can be calculated based on the current available physical memory size in the following way, which is not limited here: input the preset shard size, the empirical coefficient and the current available physical memory size into the shard size calculation formula to obtain the maximum allowable size of a single data shard; the shard size calculation formula is: B=min(Bmax, k×M); where B is used to represent the maximum allowable size of a single data shard, Bmax is used to represent the preset shard size, k is used to represent the empirical coefficient, and M is used to represent the current available physical memory size.

[0047] The following example illustrates the calculation process of this data sharding:

[0048] Assume the following parameters:

[0049] Current available physical memory size (M) = 16GB = 16384MB;

[0050] Default shard size (Bmax) = 4GB = 4096MB;

[0051] Empirical coefficient (k) = 0.25;

[0052] Use the fragment size calculation formula B=min(Bmax,k×M) to calculate: B==4096MB;

[0053] Therefore, the maximum allowed size of a single data shard is 4096MB (4GB).

[0054] Assume that there is a 32GB data set to be analyzed and processed. It can be divided into:

[0055] Data block 1: 0-4GB;

[0056] Data block 2: 4-8 GB;

[0057] Data block 3: 8-12 GB;

[0058] Data block 4: 12-16 GB;

[0059] Data block 5: 16-20 GB;

[0060] Data block 6: 20-24GB;

[0061] Data block 7: 24-28 GB;

[0062] Data block 8: 28-32GB.

[0063] S105, sending the data blocks to the corresponding R processes through the TCP / IP connection pool for parallel computing, obtaining multiple partial computing results, and merging the multiple partial computing results to obtain a second analysis result;

[0064] Among them, the TCP / IP connection pool refers to a resource pool used to manage multiple network connections, supporting reliable data transmission between different processes; parallel computing refers to the process of multiple processing units executing computing tasks simultaneously; some calculation results are used to represent the intermediate results obtained after each R process completes the processing of its own data block; the second analysis result refers to the final analysis result obtained by merging multiple processes after parallel processing.

[0065] After the data sharding is completed, the data analysis system begins to perform parallel data processing. Specifically, first, the data analysis system allocates a dedicated network connection to each R process through the TCP / IP connection pool to ensure the reliability of data transmission. Then, the data analysis system sends each data block to different R processes through the corresponding network connection. The sending process includes steps such as data serialization, network transmission, and data reception confirmation. After receiving the data, each R process independently executes the same analysis and processing logic. During the processing, the data analysis system monitors the execution status and progress of each R process. When all R processes have completed processing, the data analysis system collects the partial calculation results generated by each R process, and combines these partial calculation results into a complete second analysis result according to a predetermined merging strategy. The merging process may include data deduplication, sorting, aggregation and other operations.

[0066] S106: Visually display the first analysis result or the second analysis result.

[0067] Among them, the first analysis result refers to the data analysis result obtained by direct processing through single-process mode; the second analysis result refers to the data analysis result obtained after multi-process parallel processing; visualization display refers to converting the data analysis results into intuitive display forms such as charts and graphs.

[0068] After the data analysis system completes data analysis, it needs to present the results to the user in an intuitive manner. Specifically, first, the data analysis system selects the corresponding data analysis result (first analysis result or second analysis result) based on the processing mode (single-process mode or multi-process mode). The data analysis system then determines the characteristics of the data analysis result, including the data type, data distribution, and relationships between the data, and automatically selects the most appropriate visualization type. Visualization types are used to represent different data display methods, such as line charts, bar charts, and scatter plots. The data analysis system then converts the data analysis results into the selected visualization format, performing operations such as data normalization, axis setting, and legend generation during the conversion process. Finally, the data analysis system generates an interactive visualization interface that supports users in performing operations such as data filtering, detail viewing, and view adjustment, ensuring that users can easily understand and explore the data analysis results.

[0069] By adopting the above technical solution, the data analysis system calculates the resource utilization ratio and dynamically selects single-process or multi-process processing mode based on a preset ratio threshold, effectively balancing resource utilization and processing efficiency. When the data set to be analyzed is small, the single-process mode is adopted, and the data set to be analyzed is directly converted into an R language data frame for processing, avoiding the additional overhead brought by multiple processes. When the data set to be analyzed is large, multiple R processes are created based on the current number of CPU cores and a TCP / IP connection pool is established. Through data sharding and parallel computing, the multi-core computing power is fully utilized. This adaptive processing mode not only ensures the processing efficiency of small data sets, but also effectively meets the computing needs of large data sets, improving the performance of data analysis and resource utilization as a whole.

[0070] The following is a more detailed description of the process of the method provided by this implementation. Figure 2 , is another flow chart of the data analysis method in an embodiment of the present application.

[0071] S201: Obtain the current number of CPU cores and a task analysis request input by a user, where the task analysis request includes a data set to be analyzed.

[0072] For details, please refer to step S101, which will not be described again here.

[0073] S202: Obtain the variable type of the data set to be analyzed, and determine whether the data set to be analyzed includes categorical variables.

[0074] Among them, the variable type is used to represent the data attributes of each field in the data set to be analyzed, including continuous variables, discrete variables, categorical variables, etc.; categorical variables refer to variables with values in a finite number of categories, such as gender and education level; continuous variables refer to variables that can take any value, such as height and weight; discrete variables refer to variables that can only take specific values, such as age and number.

[0075] Before a data analysis system begins statistical analysis, it needs to determine an appropriate analysis method. Specifically, the system first reads the metadata of the dataset to be analyzed and identifies the data type and value characteristics of each variable. The system then performs a detailed type determination on each variable, including checking for the presence of predefined category labels, analyzing the degree of numerical dispersion, and assessing the continuity of values. For potentially ambiguous variables, the system further analyzes their data distribution characteristics and statistical properties to accurately determine their type. Finally, the system determines whether the dataset to be analyzed contains categorical variables. This determination determines the statistical test method to be used subsequently.

[0076] S203: If categorical variables are included, the chi-square test path is automatically selected, and the chi-square independence test is performed to obtain a first verification result.

[0077] The chi-square test path refers to the processing flow for data analysis using the chi-square statistical method; the chi-square independence test refers to a statistical method used to test whether there is a correlation between two categorical variables; and the first verification result refers to the statistical analysis result obtained through the chi-square test, including the chi-square value, degrees of freedom, and P value.

[0078] When the data analysis system detects that the data set to be analyzed contains categorical variables, it uses statistical methods suitable for categorical data analysis. Specifically, first, the data analysis system creates a contingency table for the categorical variables and counts the observed frequencies of each category combination. Then, the data analysis system calculates the expected frequency for each cell. The expected frequency is used to represent the theoretically expected observation value and compares the expected frequency with the actual frequency (the actual frequency refers to the actual observed data frequency). Next, the data analysis system calculates the chi-square statistic. This process includes calculating the difference between the observed value and the expected value, standardization, and summation. The data analysis system also calculates the degrees of freedom and finds the corresponding critical value. Finally, the data analysis system determines the P value based on the calculation results and organizes information such as the chi-square value, degrees of freedom, and P value into the first verification result.

[0079] S204: If no categorical variables are included, the variance analysis path is automatically selected, and a one-way variance analysis is performed to obtain a second verification result.

[0080] Among them, the variance analysis path refers to the processing flow of using the variance analysis method for data analysis; one-way variance analysis refers to the statistical method of studying the impact of one factor on the observed variable; the second verification result refers to the statistical result obtained through variance analysis, including F value, between-group variance, and within-group variance.

[0081] When the data analysis system confirms that the data set to be analyzed does not contain categorical variables, a statistical method suitable for continuous data analysis is used. Specifically, first, the data analysis system calculates the descriptive statistics of each group, including the mean, standard deviation, etc. Then, the data analysis system calculates the sum of squares between groups and the sum of squares within groups respectively, which involves calculating the deviation of the data from the overall mean and the group mean. Next, the data analysis system calculates the degrees of freedom between groups and the degrees of freedom within groups, and obtains the mean square value based on this. Then, the data analysis system calculates the F statistic and determines the corresponding critical value based on the degrees of freedom. Finally, the data analysis system determines the P value and organizes the statistics such as the F value, degrees of freedom, and P value into a second verification result.

[0082] S205. Calculate the effect value based on the first verification result or the second verification result, and generate a statistical test report.

[0083] Among them, the effect value refers to the actual impact of the statistical test results, which is used to measure the size of the effect; the statistical test report refers to the technical document containing the complete statistical analysis results.

[0084] After the data analysis system completes the statistical test, it needs to provide an in-depth explanation of the verification results and form a report. Specifically, first, the data analysis system selects the corresponding effect value calculation formula based on the selected test method (chi-square test or analysis of variance). Then, the data analysis system calculates the size of the effect value. This process may involve the standardization of multiple statistics. Next, the data analysis system determines the degree of effect based on the size of the effect value and gives corresponding explanations. The data analysis system also integrates all statistical analysis results, including descriptive statistics, test statistics, effect values and other information. Finally, the data analysis system generates a standardized statistical test report, which includes complete content such as data overview, analysis method description, test results, effect analysis, and conclusion explanation.

[0085] S206. Obtain the inherent delay of calling the R function based on rpy2 and the time overhead of memory data sharing, and calculate the total delay of single-process calling; obtain the network delay of data transmission based on Rserve and the time overhead of actual data processing in the R process, and calculate the total delay of multi-process calling.

[0086] Among them, rpy2 is used to represent the interface tool for Python to call R language; inherent delay refers to the inevitable system overhead when calling R functions; the time overhead of memory data sharing refers to the time consumed in transferring and converting data in memory; the total delay of single-process call refers to the total time consumed in processing data using the single-process mode; Rserve is used to represent the network service interface of R language; network delay refers to the time loss of data during network transmission; the actual time overhead of processing data refers to the actual time consumed by the R process to perform data analysis; the total delay of multi-process call refers to the total time consumed in processing data using the multi-process mode.

[0087] Before a data analysis system selects a processing mode, it needs to evaluate the time efficiency of different processing methods. Specifically, the data analysis system first uses a test program to measure the basic latency of calling R functions through the rpy2 interface. This includes the time spent on function call initialization, parameter passing, and result return, thereby obtaining the inherent latency of calling R functions through rpy2. The data analysis system then measures the time overhead of data sharing and conversion in memory, adding these two delays to obtain the total latency of a single process call. Next, the data analysis system measures the network transmission latency based on Rserve, including the time spent on connection establishment, data serialization, network transmission, and data reception, thereby obtaining the network latency of data transmission based on Rserve. Finally, the data analysis system measures the time overhead of the R process actually processing data, and adds these two delays to obtain the total latency of multi-process calls.

[0088] S207: Compare the total delay of a single process call and the total delay of multiple process calls.

[0089] After the data analysis system obtains latency data for both processing modes, it needs to perform a performance comparison. Specifically, the data analysis system first ensures that the latency data for both processing modes are comparable and measured under the same conditions. Then, the data analysis system compares the total latency of a single-process call with the total latency of multiple-process calls. This comparison takes into account measurement errors and system fluctuations, and may require averaging multiple measurements.

[0090] S208: If the total delay of a single-process call is less than the total delay of a multi-process call, the single-process mode is adopted.

[0091] Among them, the single-process mode refers to the operating mode of using a single process to complete data processing.

[0092] When the single-process mode demonstrates better time efficiency, the data analysis system needs to configure the corresponding processing environment. Specifically, the data analysis system first confirms that it is operating in single-process mode, shutting down or recycling the created multi-process resources. The data analysis system then initializes the single-process operating environment, including configuring memory allocation and loading necessary function libraries. Next, the data analysis system establishes the rpy2 calling interface to ensure normal communication between the Python and R language environments. Finally, the data analysis system prepares the data processing flow, including the specific implementation plans for steps such as data loading, conversion, and analysis.

[0093] S209: If the total delay of a single process call is greater than or equal to the total delay of a multi-process call, the multi-process mode is adopted.

[0094] Among them, the multi-process mode refers to an operating mode that uses multiple parallel processes to complete data processing.

[0095] When the multi-process model demonstrates improved time efficiency, the data analysis system needs to configure a parallel processing environment. Specifically, the data analysis system first confirms that it will operate in multi-process mode and initiates process creation and resource allocation. Then, based on the current number of CPU cores and memory availability, the data analysis system determines the optimal number of processes and resource allocation. Next, the data analysis system initializes the operating environment for each process, including configuring the Rserve service and establishing network connections. Finally, the data analysis system prepares the parallel processing flow, including specific implementation plans for steps such as data sharding, task allocation, and result merging, and establishes a coordination mechanism between processes.

[0096] S210. If the resource occupancy ratio is greater than or equal to the preset ratio threshold, the number of child processes is determined based on the current number of CPU cores, R processes equal to the number of child processes are created, and a TCP / IP connection pool is established for the R processes; based on the current available physical memory size, the maximum allowable size of a single data slice is calculated, and the data set to be analyzed is divided into multiple data blocks according to the maximum allowable size.

[0097] For details, please refer to steps S103 and S104, which will not be described in detail here.

[0098] S211. Send the data blocks to the corresponding R processes through the TCP / IP connection pool for parallel computing.

[0099] For details, please refer to step S105, which will not be described again here.

[0100] S212: Calculate the network transmission time of a single data block.

[0101] The network transmission time refers to the time required for a data block to be transmitted in the network.

[0102] Specifically, first, the data analysis system selects a representative data block as a test sample. Then, the data analysis system records the timestamp when the data block transmission starts. This process includes data serialization (data serialization refers to the conversion of data into a transmittable format) and the time of establishing the network connection. Then, the data analysis system monitors the entire process of data transmission, recording the various time points of data packet sending, transmission and reception confirmation. Finally, the data analysis system calculates the total time from the start of sending to the completion of receiving. This total time includes the time consumed by all links such as data serialization, network transmission, and data deserialization.

[0103] S213: Calculate the total data transmission delay based on the total size of the data set to be analyzed and the network transmission time.

[0104] The total size of the dataset to be analyzed refers to the capacity of the complete dataset that needs to be processed; the network transmission time refers to the time it takes to transmit a single data block; and the total data transmission delay refers to the total time required to transmit the complete dataset to be analyzed.

[0105] Specifically, the data analysis system first calculates the number of required transmission batches based on the total size of the dataset to be analyzed and the size of each data block. Next, the data analysis system considers the network's parallel transmission capabilities and determines the number of data streams that can be transmitted simultaneously. Next, the data analysis system multiplies the network transmission time for a single data block by the number of transmission batches, taking into account the time savings achieved through parallel transmission. Finally, the data analysis system calculates the total delay required to transmit the complete dataset to be analyzed, i.e., the total data transmission delay, taking into account factors such as network load and transmission contention.

[0106] S214. When the current number of CPU cores is greater than a preset value, the total data transmission delay is divided by the current number of CPU cores to obtain the average delay of the R process.

[0107] The current number of CPU cores refers to the number of processor cores available in the current system; the preset value refers to the predefined threshold for the number of CPU cores; and the average latency refers to the average transmission delay when each R process processes data.

[0108] Specifically, the data analysis system first compares the current number of CPU cores with a preset value to determine whether it is necessary to calculate the average latency of the R process. The data analysis system then considers the actual availability and performance characteristics of the CPU cores and may need to exclude system-reserved cores or performance-limited cores. The data analysis system then divides the previously calculated total data transmission latency by the number of available CPU cores to obtain the average latency for each R process. This value reflects the expected data transmission time cost for each process when multiple processes are processed in parallel.

[0109] S215: Obtain the processing success rate, memory usage rate, and calculation time required to complete data analysis for each data block of the R process.

[0110] The processing success rate indicates the proportion of R processes that successfully complete data analysis tasks; the memory usage rate refers to the extent to which the R process occupies system memory resources; and the calculation duration indicates the actual time required for the R process to process a single data block.

[0111] Specifically, the data analysis system first establishes a performance monitoring mechanism to track the running status of each R process in real time and record the number of successes and failures of each R process in processing data tasks to calculate the processing success rate. Next, the data analysis system monitors the memory usage of each R process through the system interface, recording the peak and average memory usage. Furthermore, the data analysis system accurately records the start and end times of each R process processing a data block to calculate the actual computation time.

[0112] S216. Divide the R process into a high-performance process group and a low-performance process group based on the processing success rate, memory usage, and calculation time.

[0113] Among them, the high-performance process group is used to represent a set of R processes with good running status; the low-performance process group is used to represent a set of R processes with poor performance that needs to be optimized.

[0114] Specifically, the data analysis system first determines the weights of various performance indicators to establish a comprehensive scoring mechanism. The data analysis system then normalizes the processing success rate, memory usage, and computation time of each R process to calculate a weighted score. Next, the data analysis system sets thresholds for performance grouping, potentially taking into account system load and task requirements. Finally, the data analysis system assigns processes to corresponding performance groups based on the scoring results. This grouping process is dynamic and updates as process performance changes.

[0115] S217. Reclaim the memory space of the R process in the low-performance process group, initialize the operating environment of the R process in the low-performance process group, and reload the function library of the R process in the low-performance process group.

[0116] Among them, memory space is used to represent the system memory resources occupied by the process; the operating environment is used to represent the configuration and parameters required for the process to run; the function library is used to represent the set of functional modules of the R language; environment initialization refers to resetting the running status of the process; resource recovery refers to releasing the system resources occupied by the process.

[0117] Specifically, the data analysis system first forcibly reclaims the memory resources occupied by processes in the low-performance process group through the operating system interface, including clearing memory caches and temporary data. The data analysis system then reinitializes the operating environment of these processes, including resetting environment variables, clearing session state, and restoring default configurations. Finally, the data analysis system reloads the necessary R language function libraries, ensuring the latest versions of the library files are loaded, and verifies the integrity and availability of the library files.

[0118] S218. Periodically check the running status of the R process; when the R process exits abnormally, obtain the data processing progress of the R process, save the completed data analysis results to the intermediate result buffer, and record the error type of the abnormal exit.

[0119] Among them, abnormal exit refers to the situation where the R process terminates abnormally; data processing progress refers to the proportion of completed analysis tasks; intermediate result buffer refers to the storage space for temporarily storing analysis results; error type refers to the specific cause classification that causes the R process to exit.

[0120] When the data analysis system monitors the running status of the R process at preset intervals and detects a process anomaly, emergency response is required. Specifically, the data analysis system first captures the signal indicating the R process's abnormal exit and quickly saves the current processing status. The data analysis system then uses a progress recording mechanism to obtain the amount of data processed by the R process and the current processing stage, and quickly transfers the completed analysis results to a dedicated cache area. This process ensures data integrity and consistency. At the same time, the data analysis system will record in detail the specific cause of the anomaly, including error codes, error messages, system status, and other information.

[0121] The data analysis system in the embodiment of the present invention is described below from the perspective of hardware processing. Figure 3 , is a schematic diagram of a physical device structure of a data analysis system in an embodiment of the present application.

[0122] It should be noted that Figure 3 The structure of the data analysis system shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0123] like Figure 3As shown, the data analysis system includes a CPU 301, which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 302 or programs loaded from a storage unit 308 into a random access memory (RAM) 303, such as executing the methods described in the above embodiments. RAM 303 also stores various programs and data required for system operation. CPU 301, ROM 302, and RAM 303 are interconnected via a bus 304. An I / O interface 305 is also connected to bus 304.

[0124] The following components are connected to the I / O interface 305: an input section 306 including an audio input device, push button switches, and the like; an output section 307 including a liquid crystal display (LCD), an audio output device, indicator lights, and the like; a storage section 308 including a hard disk and the like; and a communication section 309 including a network interface card such as a LAN (Local Area Network) card or a modem. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. Removable media 311, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 310 as needed, so that computer programs read from the removable media can be installed in the storage section 308 as needed.

[0125] In particular, according to an embodiment of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, an embodiment of the present invention includes a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 309 and / or installed from the removable medium 311. When the computer program is executed by the CPU 301, the various functions defined in the present invention are performed.

[0126] It should be noted that specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0127] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. Each box in the flowchart or block diagram can represent a module, program segment, or part of the code, and the above-mentioned module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings.

[0128] Specifically, the data analysis system of this embodiment includes a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, the data analysis method provided in the above embodiment is implemented.

[0129] As another aspect, the present invention further provides a computer-readable storage medium, which may be included in the data analysis system described in the above embodiments, or may exist independently and not incorporated into the data analysis system. The storage medium carries one or more computer programs, which, when executed by a processor of the data analysis system, enable the data analysis system to implement the data analysis methods provided in the above embodiments.

[0130] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0131] As used in the above embodiments, the term “when” may be interpreted to mean “if” or “after” or “in response to determining that” or “in response to detecting that”, depending on the context. Similarly, the phrases “upon determining that” or “if (stated condition or event) is detected” may be interpreted to mean “if determining that” or “in response to determining that” or “upon detecting (stated condition or event)” or “in response to detecting (stated condition or event)”, depending on the context.

[0132] Those skilled in the art will appreciate that all or part of the process steps in the above-described method embodiments can be implemented by a computer program instructing the relevant hardware. The program can be stored in a computer-readable storage medium, and when executed, the program can include the process steps in the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A data analysis method, characterized in that: Applied to a data analysis system, the method includes: Obtain the current number of CPU cores and a task analysis request input by the user, wherein the task analysis request includes a dataset to be analyzed, and divide the physical memory size occupied by the dataset to be analyzed by the current available physical memory size to obtain a resource occupancy ratio; If the resource occupancy ratio is less than a preset ratio threshold, converting the data set to be analyzed into an R language data frame, calling an R function to process the R language data frame, and obtaining a first analysis result; If the resource occupancy ratio is greater than or equal to a preset ratio threshold, determining the number of child processes based on the current number of CPU cores, creating R processes equal to the number of child processes, and establishing a TCP / IP connection pool for the R processes; Calculating the maximum allowable size of a single data slice based on the currently available physical memory size, and dividing the data set to be analyzed into multiple data blocks according to the maximum allowable size; Sending the data blocks to corresponding R processes through the TCP / IP connection pool for parallel computing, obtaining multiple partial computing results, and merging the multiple partial computing results to obtain a second analysis result; Visually displaying the first analysis result or the second analysis result; The calculating of the maximum allowable size of a single data shard based on the currently available physical memory size specifically includes: inputting a preset shard size, an empirical coefficient, and the currently available physical memory size into a shard size calculation formula to obtain the maximum allowable size of the single data shard; the shard size calculation formula is: B=min(Bmax, k×M); wherein B is used to represent the maximum allowable size of the single data shard, Bmax is used to represent the preset shard size, k is used to represent the empirical coefficient, and M is used to represent the currently available physical memory size.

2. The method according to claim 1, characterized in that After the step of obtaining the current number of CPU cores and the task analysis request input by the user, the method further includes: Obtain the inherent latency and memory data sharing overhead of calling R functions based on rpy2, and calculate the total latency of a single process call. Obtain the network delay of data transmission based on Rserve and the actual time overhead of data processing in the R process, and calculate the total delay of multi-process calls; Comparing the total delay of the single-process call and the total delay of the multi-process call; If the total delay of the single-process call is less than the total delay of the multi-process call, the single-process mode is adopted; If the total delay of the single-process call is greater than or equal to the total delay of the multi-process call, the multi-process mode is adopted.

3. The method according to claim 1, characterized in that After the step of sending the data blocks to the corresponding R processes for parallel computing through the TCP / IP connection pool, the method further includes: Calculate the network transmission time of a single data block; Calculating a total data transmission delay based on the total size of the data set to be analyzed and the network transmission time; When the current number of CPU cores is greater than a preset value, the total data transmission delay is divided by the current number of CPU cores to obtain the average delay of the R process.

4. The method according to claim 3, characterized in that When the current number of CPU cores is greater than a preset value, after the step of dividing the total data transmission delay by the current number of CPU cores to obtain the average delay of the R process, the method further includes: Obtain the processing success rate, memory usage, and computing time of the R process to complete data analysis for each data block; Dividing the R process into a high-performance process group and a low-performance process group according to the processing success rate, the memory usage rate, and the calculation duration; Reclaim the memory space of the R process in the low-performance process group, initialize the operating environment of the R process in the low-performance process group, and reload the function library of the R process in the low-performance process group.

5. The method according to claim 1, wherein After the step of obtaining the current number of CPU cores and the task analysis request input by the user, the method further includes: Obtaining the variable type of the data set to be analyzed; Determining whether the data set to be analyzed includes categorical variables; If categorical variables are included, the chi-square test path is automatically selected, and the chi-square independence test is performed to obtain the first verification result; If categorical variables are not included, the variance analysis path is automatically selected, and a one-way variance analysis is performed to obtain the second verification result; Based on the first verification result or the second verification result, the effect value is calculated and a statistical test report is generated.

6. The method according to claim 1, characterized in that After the step of sending the data blocks to the corresponding R processes for parallel computing through the TCP / IP connection pool, the method further includes: Periodically check the running status of the R process; When the R process exits abnormally, the data processing progress of the R process is obtained, and the completed data analysis results are saved to the intermediate result buffer area, and the error type of the abnormal exit is recorded.

7. A data analysis system, characterized in that: The data analysis system includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the data analysis system to perform the method as described in any one of claims 1 to 6.

8. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on a data analysis system, the data analysis system is caused to execute the method according to any one of claims 1 to 6.

9. A computer program product, characterized in that When the computer program product is run on a data analysis system, the data analysis system is caused to perform the method according to any one of claims 1 to 6.

Images