A data analysis software architecture design method

Abstract

The present invention belongs to the field of data analysis technology, and in particular to a data analysis software architecture design method, comprising the following steps: S1, building a statistical analysis system, the statistical analysis system comprising a model layer, a view layer, and a controller layer, separating data processing, business logic, and user interface into independent layers, the present invention significantly reduces the delay and memory usage of cross-language communication, maximizes the use of computing resources while ensuring calculation accuracy, breaks through the limitations of fixed communication modes, enables heterogeneous language collaborative analysis to run efficiently in various scenarios, realizes interface operations and background calculations simultaneously, thoroughly solves the problem of interface freezes caused by long-term calculations, constructs a layered and modular statistical analysis process, and decomposes complex data analysis tasks into reusable and replaceable functional modules, and users can freely combine analysis processes or add new modules according to their needs to ensure that the system has long-term evolution capabilities.

Description

Technical Field

[0001] The present invention relates to the field of data analysis technology, and in particular to a data analysis software architecture design method. Background Art

[0002] Currently, the statistical analysis tools widely used in clinical research and data analysis fall into the following categories: First, commercial statistical software such as SPSS and SAS offer comprehensive graphical user interfaces, which are convenient but expensive and have limited scalability. Second, open-source statistical programming languages ​​such as R and Python offer powerful functionality but require specialized programming skills, making their direct use difficult for clinical researchers. Furthermore, with the growing demand for big data and complex analysis, a single language or tool often struggles to balance performance and usability. For example, R boasts a rich statistical library, but is often limited by memory and single-threaded performance when processing large-scale data. Data analysis libraries within the Python ecosystem are widely used, but implementation of certain specialized statistical methods is limited. Some existing solutions attempt to combine the strengths of multiple technologies, such as calling R functions from Python (e.g., through the Rserve service or the rpy2 interface) to achieve enhanced statistical computing capabilities. However, these cross-language calls typically rely on a single communication method and lack adaptive optimization based on data size and computing environment. This results in unnecessary overhead for small datasets and memory or communication bottlenecks for large datasets. In addition, most traditional statistical analysis software uses a single process for sequential execution. When performing time-consuming analysis, the interface often freezes, resulting in a poor user experience. Overall, the existing technology has difficulty in achieving a balance between user-friendliness and high-performance and flexible analysis. There is a lack of a new statistical analysis system that is both oriented towards non-programmer users and can handle large data and is easily scalable. 1. Performance bottleneck of cross-language calls: Existing solutions usually use embedded calls or network service calls when interacting with Python and R, which cannot take into account the performance requirements of data of different scales. For small-scale data, enabling network services brings unnecessary communication overhead; and for very large-scale data, relying solely on embedded calls will cause excessive memory usage or call failures. The present invention needs to solve how to adaptively select the optimal cross-language communication method based on data characteristics to ensure call efficiency and stability in different situations. 2. Interface response and computational parallelism problem: Traditional statistical software is mostly single-threaded, and the interface is blocked when performing complex analysis, affecting user operations. How to ensure that the analysis and calculation fully utilize multi-core to increase speed while maintaining a smooth response of the user interface is another issue that the present invention focuses on. This requires a multi-process or multi-threaded task scheduling mechanism to avoid resource competition between the GUI and computing, and to ensure that long-running analysis tasks do not cause the application to become unresponsive. 3. Insufficient module expansion and maintainability: Existing statistical analysis tools have difficulties in functional expansion. Many software programs hard-code the analysis process into a single program and lack a good modular design. As a result, adding new analysis methods or improving algorithms requires significant code modifications, which is high-risk and difficult to maintain. In addition, there is a lack of unified interfaces for different analysis steps, and the coupling between modules is high, which limits the reuse and combination of functions.The problem to be solved by the present invention is how to design a modular and extensible architecture, which breaks down the data analysis process into loosely coupled functional units according to stages, which is convenient for plugging and unplugging functions and maintains overall collaborative work. 4. Balance between user-friendliness and professionalism: For non-computer professional users such as clinical researchers, the use of statistical tools in a purely programming manner is not friendly, and graphical interface tools often have limited functions. The technical problem that needs to be solved is to provide a friendly and intuitive user interface without reducing the professionalism and flexibility of statistical analysis. In other words, the system must allow users to conveniently complete advanced statistical analysis configurations through the interface, while ensuring that the underlying calculations use rigorous statistical methods and output reliable results. Summary of the Invention

[0003] The purpose of the present invention is to solve the shortcomings of the background technology and to propose a data analysis software architecture design method.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions:

[0005] A data analysis software architecture design method includes the following steps:

[0006] S1. Build a statistical analysis system. The statistical analysis system includes the model layer, view layer, and controller layer. It separates data processing, business logic, and user interface into independent layers, with clear responsibilities and collaborative work at each layer.

[0007] S2. Cross-language interaction and adaptive data transmission mechanism: The model layer communicates with the R language environment through an interface. The system also integrates two R calling methods: embedded rpy2 interface calling and server-based Rserve interface calling, using an adaptive selection strategy to optimize the interface;

[0008] S3. Multi-process Parallel Analysis and Task Scheduling: This introduces a multi-process analysis execution mechanism. The system architecture places data analysis tasks into independent subprocesses. The main process is primarily responsible for interface interaction and task control. The task scheduling manager is built using Python's multiprocessing library. When a user initiates an analysis on the interface, the controller creates a new analysis process and passes the corresponding data and parameters to the process to execute the model layer's computational code.

[0009] S4. Modular functional design and expansion: Adopting modular design ideas, the statistical analysis process is divided into multiple loosely coupled modules according to different stages and functions, providing good scalability and customization, including multi-level analysis module division, module independent communication, module registration and dynamic loading.

[0010] Preferably, the model layer is responsible for data storage and analysis processing, and is the core computing unit of the system, covering data reading and cleaning, field extraction, automatic identification of variable types, statistical description calculations, hypothesis testing and model building functions. The model layer is implemented using object-oriented and factory method patterns, and dynamically creates corresponding data processing objects according to different data analysis requirements. For different types of data sets or statistical analysis methods, the model layer can select appropriate algorithm module instances through the factory pattern. The model layer also encapsulates the calling logic of R language statistical functions, and performs specific statistical calculations through a custom R function interface. After obtaining the analysis results, the model layer will generate a structured result object and return the result to the controller layer.

[0011] Preferably, the view layer provides a graphical user interface for direct interaction with the user. This layer is responsible for displaying the analysis results generated by the model layer in a friendly manner and receiving user input operations. The PySide6 framework is used to build a cross-platform GUI. The interface includes a data table display area, a statistical chart drawing area, and a log output area component. In order to adapt to various data formats and display forms, the view layer introduces an adapter mode: by writing an adapter, the underlying different types of data are converted into a form that can be directly displayed by the interface controls. The view interface design focuses on intuitiveness and ease of use, and provides drop-down menus and forms for users to select statistical methods, set parameters, and trigger analysis runs through buttons. The results are presented intuitively in the form of charts and tables, which is convenient for users to understand the analysis output.

[0012] Preferably, the controller layer is the scheduling center of the system, responsible for responding to user operations and coordinating models and views. The controller receives user requests from the view layer, and calls the corresponding analysis function module of the model layer to perform calculations accordingly, and then distributes the results returned by the model layer to the view layer for display. In order to ensure the orderliness and efficiency of the entire system process, the controller is designed as a singleton mode, that is, there is only one controller instance in the entire application for unified management. This can avoid the waste of resources caused by repeated instantiation and ensure that calls to the model layer and the view layer are coordinated. The controller also assumes the management role of module loading and extension: through the decorator mode and plug-in mechanism, the controller can dynamically load different analysis function modules at runtime or add new features to existing functions. When the user needs to add a new statistical analysis method, it can be encapsulated into a module and registered to the controller. The controller can then recognize and call the new module without changing the core code.

[0013] Preferably, the rpy2 interface call: rpy2 allows R functions and objects to be called directly in the Python process, which is equivalent to embedding an R interpreter in the memory. For small and medium-sized data sets and faster statistical calculations, using rpy2 can avoid starting additional processes or performing network communications, with low overhead and small call latency. Data is passed directly to R through memory sharing, and the results are also returned to Python synchronously.

[0014] Preferably, the Rserve interface call: Rserve provides a service interface through the TCP / IP protocol in an independent R process. When the data set is large or it is necessary to utilize the multi-threaded / multi-process computing advantages of R, the system will switch to Rserve mode. Through Rserve, Python sends the data to the R service process for processing, and the results are returned through the network.

[0015] Preferably, the adaptive selection strategy is as follows: by monitoring the size of the data set to be analyzed and the resource utilization of the current system, dynamically switch between the two calling methods: when the data volume is small and the system memory is sufficient, the direct memory connection method of rpy2 is preferred to obtain the fastest response; when the data volume is huge or the memory is tight, Rserve is used to share the memory pressure. At the same time, a data sharding transmission mechanism is introduced in the Rserve mode: for extremely large data, instead of transmitting the entire data at one time, the data is automatically split into several blocks for batch transmission and processing, reducing the amount of data transmitted in a single time. This batch processing mechanism significantly reduces network transmission delay and peak memory usage. A dynamic R environment management strategy is also designed: the controller will dynamically decide whether to start multiple Rserve working processes to process different data blocks or multiple tasks in parallel based on the current number of CPU cores and task requirements, ensuring that computing resources are fully utilized and not wasted. Through the above mechanism, the collaborative calling of Python and R achieves adaptive optimization, and can obtain near-optimal performance in data analysis tasks of various scales.

[0016] Preferably, the interface optimization is as follows: in order to further reduce the cross-language call overhead, parameter passing and result return are optimized at the interface layer where Python and R interact. On the one hand, efficient binary data serialization and shared memory technology are used to reduce the number of data copies; on the other hand, a unified data result format is agreed upon to simplify result parsing. Under this optimization, each time Python calls an R function, it carries the necessary parameters and minimizes data redundancy. The R return result is also compressed and packaged into a form that is easy for Python to process. This reduces the extra time consumed in cross-language communication, so that even if the R function is called frequently, the overall performance loss is very small.

[0017] Preferably, the multi-level analysis module is divided as follows: the system divides the functions into three main levels according to the general process of statistical analysis: data preprocessing, data analysis, and result visualization. Each level is further subdivided into several independent functional modules. The data preprocessing level includes "missing value processing", "outlier detection", and "variable type conversion" modules; the data analysis level includes "descriptive statistics", "hypothesis testing", "regression analysis", and "principal component analysis" modules; and the visualization level includes "distribution histogram drawing" and "correlation matrix drawing" modules. Each module has its own function and can be independently developed and tested.

[0018] Preferably, the independent communication of the modules is as follows: in order to reduce the coupling between modules, each functional module is connected through a uniformly defined data interface, the data transmitted between modules adopts a standard format, and the modules agree on input and output: the data output by the previous module is directly used as the input of the next module. Due to the use of standard data format, the modules do not rely on each other's specific implementation, as long as they follow the interface contract, which means that developers can replace the internal implementation of a module without affecting other parts. There can be a variety of "outlier processing" modules with different algorithms. As long as their input and output formats are consistent, they can be registered and used in the system. The modules are connected through controller scheduling. After the user configures the analysis process in the interface, the controller calls the corresponding modules in sequence, passes data in sequence, and completes the entire pipeline; module registration and dynamic loading are as follows: a module management mechanism based on factory mode and multi-layer dictionary mapping is designed. A module registry is maintained internally in the system. The first layer distinguishes the modules. At the attribute level, the second layer uses the module name or function key to index the specific module implementation object. When a new module needs to be added, the developer registers the module class or function in this dictionary mapping. When the controller calls the module, it quickly finds the corresponding implementation through the module name and instantiates it for operation. Due to the use of the factory mode, the instantiation of the module is highly abstracted. The controller does not need to understand the internal details of the module, but only needs to request the module object according to the name. This dynamic loading mechanism ensures the high scalability of the system: without modifying the core code of the system, users or developers can add new statistical method modules at any time in the form of plug-ins, and the system immediately supports the method. In addition, the module management dictionary can also store the mapping rules between modules and applicable data types or scenarios. When called, the system can automatically match the optimal analysis module according to the type or size of the input data. For data sets with categorical variables, the statistical description module will automatically select the chi-square test path, while for continuous variables, it will select the t-test or variance analysis path.

[0019] Before the present invention was proposed, there were some alternative solutions that attempted to achieve similar functional goals, but each had its own limitations:

[0020] Solution 1: Statistical software based on a single language. For example, only use R language to develop a graphical user interface (with the help of frameworks such as Shiny) to implement statistical analysis functions, or only use Python (combined with its data science library) to develop an analysis application. The advantage of this type of solution is that the architecture is relatively simple, avoiding cross-language calls. However, the disadvantages are also obvious: using only R will encounter problems such as difficulty in building complex GUIs and insufficient smoothness of front-end interaction, and using only Python will not be as good as R in the richness and reliability of statistical professional algorithms. The present invention combines Python + R to take into account both friendly interface interaction and powerful statistical calculations. The two complement each other's advantages, which cannot be met simultaneously by a single language solution.

[0021] Solution 2: Cross-language calling with fixed communication mode. There are already some tools that allow Python to call R (for example, simply using rpy2 or with the help of Rserve service), and developers can also use one of the solutions to implement cross-language analysis. However, fixedly adopting one calling method means inefficiency in some cases: using rpy2 to process very large data will fail or be extremely slow due to memory limitations, and conversely, using Rserve to process small data will increase overhead. The adaptive communication mechanism of the present invention combines the advantages of both, and can be flexibly switched as needed, greatly improving performance and reliability. In comparison, the fixed mode solution is inferior in terms of versatility and stable performance.

[0022] Solution three: Implementation of single-process multi-threading. Another alternative path is to use multi-threading within a single process to try to parallelize computing and interface responses. However, due to Python's GIL restrictions, multi-threading is difficult to truly execute CPU-intensive tasks in parallel, and the R language itself has limited support for multi-threading. In addition, running R calls in multiple threads in the same process may cause conflicts and easily cause the program to crash. In comparison, the present invention adopts a multi-process architecture to completely avoid these problems and achieve true parallelism and stability. Although the multi-process implementation is relatively complex, it ensures that the calculation and interface each occupy independent resources, and the performance and robustness are better than the single-process multi-threaded solution.

[0023] Solution 4: Non-modular integrated design. Some solutions may write all functions in one large code block or a few modules, eliminating the need for module division. However, this integrated design is very cumbersome when it comes to functional expansion. Adding or deleting a function may affect other parts, and the testing and maintenance costs are high. The present invention decouples each functional unit through a modular layered design. Replacing or adding a new module will not interfere with other functions. The system can evolve as demand evolves, which is significantly better than the lack of flexibility of non-modular solutions.

[0024] Compared with the prior art, the advantages of the present invention are:

[0025] Cross-language Adaptive Interaction Mechanism: This paper proposes an adaptive data transmission protocol for Python and R collaboration. The system intelligently determines data size and dynamically selects between embedded calls (rpy2) and network service calls (Rserve). Specifically, the introduction of data sharding and multi-R process parallel computing for big data significantly reduces cross-language communication latency and memory usage, maximizing computational resource utilization while ensuring computational accuracy. This mechanism transcends the limitations of fixed communication modes and enables efficient heterogeneous language collaborative analysis in a variety of scenarios.

[0026] Multi-process Architecture and Task Scheduling: A multi-process architecture has been designed that separates the GUI from computation. By using a master process and analysis subprocess model, user interface operations and background computations can be performed simultaneously, completely eliminating the issue of interface lag caused by prolonged computations. The task scheduling module manages multiple analysis tasks in parallel and provides process isolation protection, improving system reliability. This architecture fully leverages the advantages of multiple cores, significantly improving the efficiency and user experience of large-scale statistical analyses.

[0027] Modular, layered design and flexible expansion: A layered, modular statistical analysis process is constructed, breaking down complex data analysis tasks into reusable and replaceable functional modules. Multi-layer dictionary mapping and a factory pattern enable dynamic module registration and invocation, supporting on-demand expansion of analysis functionality without modifying the existing system. Standard data interfaces between modules ensure low coupling and facilitate independent development and testing. This design enables a high degree of customization, allowing users to freely combine analysis processes or add new modules as needed, ensuring the system's long-term evolution.

[0028] Comprehensive application of multiple design patterns optimizes the architecture: The system architecture cleverly incorporates classic design patterns such as singletons, factory methods, adapters, and decorators. The singleton pattern ensures global uniqueness of the core controller, avoiding resource conflicts; the factory and decorator patterns enable on-demand production and enhancement of functional modules, facilitating the expansion of new algorithms or the packaging and upgrading of existing algorithms; and the adapter pattern ensures the decoupling of data and interface presentation. The integrated application of multiple design patterns ensures that the system maintains code simplicity and high cohesion while enabling flexible expansion of various components, representing an innovative improvement to the software architecture. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a schematic diagram of the overall system architecture of a data analysis software architecture design method proposed by the present invention;

[0030] Figure 2 A cross-language data transmission and processing flow chart of a data analysis software architecture design method proposed in the present invention;

[0031] Figure 3A modular hierarchical analysis flow chart of a data analysis software architecture design method proposed by the present invention;

[0032] Figure 4 This is a block diagram of the view layer, control layer, and model layer of the data analysis software architecture design method proposed by the present invention. DETAILED DESCRIPTION

[0033] The technical solution of this embodiment will be clearly and completely described below in conjunction with the drawings in this embodiment. Obviously, the described embodiment is only a part of this embodiment, rather than all the embodiments.

[0034] Reference Figure 1-Figure 4 , a data analysis software architecture design method, comprising the following steps:

[0035] S1. Build a statistical analysis system. The statistical analysis system includes the model layer, view layer, and controller layer. It separates data processing, business logic, and user interface into independent layers, with clear responsibilities and collaborative work at each layer.

[0036] S2. Cross-language interaction and adaptive data transmission mechanism: The model layer communicates with the R language environment through an interface. The system also integrates two R calling methods: embedded rpy2 interface calling and server-based Rserve interface calling, using an adaptive selection strategy to optimize the interface;

[0037] S3. Multi-process Parallel Analysis and Task Scheduling: This introduces a multi-process analysis execution mechanism. The system architecture places data analysis tasks into independent subprocesses. The main process is primarily responsible for interface interaction and task control. The task scheduling manager is built using Python's multiprocessing library. When a user initiates an analysis on the interface, the controller creates a new analysis process and passes the corresponding data and parameters to the process to execute the model layer's computational code.

[0038] S4. Modular functional design and expansion: Adopting modular design ideas, the statistical analysis process is divided into multiple loosely coupled modules according to different stages and functions, providing good scalability and customization, including multi-level analysis module division, module independent communication, module registration and dynamic loading.

[0039] In this embodiment, the model layer (Model): The model layer is responsible for the storage and analysis of data and is the core computing unit of the system. It covers functions such as data reading and cleaning, field extraction, automatic identification of variable types, statistical description calculations, hypothesis testing, and model building. The model layer is implemented using object-oriented and factory method patterns, and dynamically creates corresponding data processing objects according to different data analysis requirements. For example, for different types of data sets or statistical analysis methods, the model layer can select the appropriate algorithm module instance through the factory pattern. The model layer also encapsulates the calling logic of the R language statistical function and performs specific statistical calculations through a custom R function interface. After obtaining the analysis results, the model layer generates a structured result object (such as a data structure containing an analysis result table, a visual chart, and an analysis log, etc.), and returns the result to the controller layer.

[0040] View layer (View): The view layer provides a graphical user interface and interacts directly with the user. This layer is responsible for displaying the analysis results generated by the model layer in a friendly manner and receiving user input operations (such as parameter settings, analysis commands, etc.). The present invention uses the PySide6 framework to build a cross-platform GUI. The interface includes components such as a data table display area, a statistical chart drawing area, and a log output area. In order to adapt to various data formats and display forms, the view layer introduces an adapter pattern: by writing adapters, different types of underlying data (such as Pandas DataFrame, Numpy arrays, or data frames returned by R) are converted into forms that can be directly displayed by interface controls (such as table models, image objects, etc.). This design ensures that no matter what format the data is returned by the model layer, the view layer can be uniformly processed and presented through the adapter. The view interface design focuses on intuitiveness and ease of use. For example, drop-down menus and forms are provided for users to select statistical methods, set parameters, and trigger analysis runs through buttons. The results are presented intuitively in the form of charts and tables, making it easier for users to understand the analysis output.

[0041] Controller layer: The controller is the scheduling center of the system, responsible for responding to user operations and coordinating models and views. The controller receives user requests from the view layer (for example, the user chooses to perform a certain statistical analysis on the interface), and accordingly calls the corresponding analysis function module of the model layer to perform calculations, and then distributes the results returned by the model layer to the view layer for display. In order to ensure the orderliness and efficiency of the entire system process, the controller is designed as a singleton mode in the present invention, that is, there is only one controller instance in the entire application for unified management. This can avoid the waste of resources caused by repeated instantiation and ensure that calls to the model layer and view layer are coordinated and consistent. The controller also assumes the management role of module loading and expansion: through the decorator mode and plug-in mechanism, the controller can dynamically load different analysis function modules at runtime or add new features to existing functions. For example, when a user needs to add a new statistical analysis method, it can be encapsulated into a module and registered with the controller. The controller can then recognize and call the new module without changing the core code. This design greatly enhances the flexible expansion capability of the system.

[0042] Through the MVC layered architecture, the present invention decouples interface presentation from data processing logic: the view and model layers interact only through the controller, remaining independent of each other. This benefits future algorithm updates or interface framework changes without impacting other layers, resulting in a highly cohesive, low-coupling software structure and improved system maintainability.

[0043] Cross-language interaction and adaptive data transmission mechanism: To fully leverage the powerful statistical capabilities of the R language, the model layer of this system communicates with the R language environment through an interface. Specifically, the system integrates two R calling methods: the embedded rpy2 interface and the server-based Rserve interface. The innovation of this invention lies in the design of a cross-language adaptive data transmission protocol that intelligently selects the optimal communication mechanism between rpy2 and Rserve based on the size of the currently analyzed dataset and the availability of computing resources, significantly improving the efficiency and stability of Python-R interaction.

[0044] Rpy2 interface calls: rpy2 allows you to call R functions and objects directly from within a Python process, effectively embedding an R interpreter in memory. For small to medium-sized datasets and fast statistical calculations, using rpy2 avoids launching additional processes or performing network communication. This approach offers low overhead and minimal call latency, and is therefore preferred in this system. Data is passed directly to R via shared memory, and results are synchronously returned to Python. This approach is suitable for scenarios with moderate data volumes and frequent interactions.

[0045] Rserve interface call: Rserve provides a service interface via the TCP / IP protocol in an independent R process. When the dataset is large (for example, exceeding the memory limit of a single process) or when it is necessary to utilize R's multi-threaded / multi-process computing advantages, the system switches to Rserve mode. Through Rserve, Python sends the data to the R service process for processing, and the results are returned over the network. Although this loosely coupled approach has certain communication overhead, it is more robust for very large data sets, and the R-side calculations can be performed independently without blocking the main Python process. Adaptive selection strategy: The present invention dynamically switches between two calling methods by monitoring the size of the dataset to be analyzed and the current system resource utilization: When the data volume is small and the system memory is sufficient, the direct memory connection method of rpy2 is preferred for the fastest response; when the data volume is large or memory is limited, Rserve is used instead to share memory pressure. At the same time, the present invention introduces a data segmentation transmission mechanism in Rserve mode: for very large data, instead of transmitting the entire data at once, the data is automatically split into several blocks for batch transmission and processing, reducing the amount of data transmitted in a single batch. This batch processing mechanism significantly reduces network transmission latency and peak memory usage. A dynamic R environment management strategy has also been designed: the controller dynamically decides whether to launch multiple Rserve worker processes to process different data blocks or multiple tasks in parallel, based on the current number of CPU cores and task requirements, ensuring full utilization of computing resources without waste. This mechanism enables adaptive optimization of Python and R collaborative calls—achieving near-optimal performance for data analysis tasks of all sizes.

[0046] Interface optimization: In order to further reduce the cross-language call overhead, the present invention optimizes parameter passing and result returning at the interface layer where Python and R interact. On the one hand, efficient binary data serialization and shared memory technology (such as directly passing Numpy array pointers for rpy2) are used to reduce the number of data copies; on the other hand, a unified data result format (such as JSON or DataFrame) is agreed upon to simplify result parsing. Under this optimization, each time Python calls an R function, it carries the necessary parameters and minimizes data redundancy, and the R return result is also compressed and packaged into a form that is easy for Python to handle. This reduces the extra time consumed in cross-language communication, so that even if the R function is called frequently, the overall performance loss is very small.

[0047] Multi-process parallel analysis and task scheduling: This paper introduces a multi-process analysis execution mechanism for time-consuming statistical calculations. The system architecture places data analysis tasks in independent subprocesses, with the main process primarily responsible for interface interaction and task control. Specifically, a task scheduling manager is established using Python's multiprocessing library. When a user initiates an analysis in the interface, the controller creates a new analysis process and passes the corresponding data and parameters to the computational code in the execution model layer. This design has the following advantages:

[0048] Avoiding UI blocking: Because time-consuming calculations are performed in a child process, the main process's GUI thread is not occupied for long periods of time, allowing the user interface to remain responsive (for example, allowing users to browse partial results or prepare for the next step of analysis). This solves the issue of suspended animation in traditional single-threaded programming interfaces and significantly improves the user experience.

[0049] Improved Computing Efficiency: The multi-process mechanism enables the system to leverage multi-core CPUs to execute multiple tasks in parallel. If a user submits multiple analysis tasks simultaneously, the scheduler can launch multiple processes concurrently to handle them separately, thus reducing overall runtime. Furthermore, through effective inter-process communication, sub-processes notify the main process of their results upon completion, which are then transmitted back to the main interface, instantly updating the results.

[0050] Security and Stability: Using independent processes also enhances system robustness. Even if an analysis process fails or crashes due to abnormal data or extreme circumstances, it only affects that subprocess and does not bring down the entire main application. The main process can detect abnormal subprocess conditions and provide user notifications. This isolation ensures system stability and data security, preventing problems such as a single calculation failure leading to the loss of all unsaved data.

[0051] Task Management: The controller includes a simple task management module that monitors the execution status of each analysis process (in progress, completed, exception, etc.) and collects results. When a task completes, the result is transmitted back to the main process through a pre-established pipeline or queue. Upon receiving the result, the controller calls the view to update the result display and store the result. On the interface, users can see a list of currently running tasks and their progress. For time-consuming tasks, log output can be viewed in real time. This allows users to easily manage and track the execution of multiple analysis tasks.

[0052] Modular functional design and expansion: This invention adopts a modular design concept, dividing the statistical analysis process into multiple loosely coupled modules according to different stages and functions, providing good scalability and customization.

[0053] Multi-level analysis module division: Based on the general statistical analysis process, the system divides its functions into three main levels: data preprocessing, data analysis, and results visualization. Each level is further subdivided into several independent functional modules. For example, the data preprocessing level includes modules such as "missing value handling," "outlier detection," and "variable type conversion"; the data analysis level includes modules such as "descriptive statistics," "hypothesis testing," "regression analysis," and "principal component analysis"; and the visualization level includes modules such as "distribution histogram plotting" and "correlation matrix plotting." Each module performs its own functions and can be independently developed and tested.

[0054] Independent module communication: To reduce inter-module coupling, each functional module is connected through a uniformly defined data interface. Specifically, data transferred between modules uses a standard format (such as a Pandas DataFrame or a common format file like CSV), and the modules agree on input and output: the output of the previous module directly serves as the input of the next module. Because of the use of a standard data format, modules do not depend on each other's specific implementations; they only need to follow the interface contract. This means that developers can replace the internal implementation of a module without affecting other parts. For example, there can be multiple "outlier processing" modules with different algorithms, and as long as their input and output formats are consistent, they can all be registered and used in the system. Modules are connected through a controller. Once the user configures the analysis process in the interface (for example, choosing to fill in missing values ​​first, then perform regression analysis, and then generate charts), the controller calls the corresponding modules in sequence, passing data in turn to complete the entire pipeline.

[0055] Module Registration and Dynamic Loading: This invention designs a module management mechanism based on a factory pattern and multi-layer dictionary mapping. The system maintains a module registry (which can be understood as a multi-layered nested dictionary structure). The first layer distinguishes the module's level (preprocessing / analysis / visualization), and the second layer indexes the specific module implementation objects by module name or function key. When a new module is needed, the developer registers the module class or function in this dictionary mapping. When the controller calls the module, it quickly finds the corresponding implementation by module name and instantiates it. Due to the factory pattern, module instantiation is highly abstracted; the controller does not need to understand the module's internal details; it simply requests the module object by name. This dynamic loading mechanism ensures the system's high scalability: without modifying the core system code, users or developers can add new statistical method modules at any time via plug-ins, and the system will immediately support the method. In addition, the module management dictionary also stores mapping rules between modules and applicable data types or scenarios. When called, the system automatically matches the optimal analysis module based on the type and size of the input data. For example, for a dataset with categorical variables, the statistical description module automatically selects the chi-square test path, while for continuous variables, it selects the t-test or analysis of variance path. This intelligent matching improves the accuracy and efficiency of analysis.

[0056] The above is only a preferred specific implementation method of this embodiment, but the protection scope of this embodiment is not limited to this. Any technician familiar with this technical field can make equivalent replacements or changes based on the technical solution and inventive concept of this embodiment within the technical scope disclosed in this embodiment, and they should be covered by the protection scope of this embodiment.

Claims

1. A data analysis software architecture design method, characterized in that: The following steps are involved: S1. Build a statistical analysis system. The statistical analysis system includes the model layer, view layer, and controller layer. It separates data processing, business logic, and user interface into independent layers, with clear responsibilities and collaborative work at each layer. S2. Cross-language interaction and adaptive data transmission mechanism: The model layer communicates with the R language environment through an interface, integrating two R calling methods: embedded rpy2 interface calling and server-based Rserve interface calling, and adopting an adaptive selection strategy to optimize the interface; S3. Multi-process parallel analysis and task scheduling: This introduces a multi-process analysis execution mechanism, placing data analysis tasks into independent subprocesses. The main process is responsible for interface interaction and task control. The task scheduling manager is established using Python's multiprocessing library. When a user initiates an analysis on the interface, the controller creates a new analysis process and passes the corresponding data and parameters to the process to execute the model layer's calculation code. S4. Modular Function Design and Expansion: Using modular design, the statistical analysis process is divided into multiple loosely coupled modules according to different stages and functions, providing good scalability and customization, including multi-level analysis module division, independent module communication, module registration and dynamic loading; Specifically, the rpy2 interface allows you to call R functions and objects directly from within a Python process, effectively embedding an R interpreter in memory. For small to medium-sized datasets and fast statistical calculations, using rpy2 avoids starting additional processes or performing network communications, resulting in low overhead and minimal call latency. Data is passed directly to R via shared memory, and the results are synchronously returned to Python. Specifically, Rserve provides a service interface in an independent R process via the TCP / IP protocol. When the data set is large or the multi-threaded / multi-process computing advantages of R are needed, the system switches to Rserve mode. Through Rserve, Python sends the data to the R service process for processing, and the results are returned via the network. The interface optimization is as follows: Parameter passing and result return are optimized at the Python-R interaction interface layer. On the one hand, efficient binary data serialization and shared memory technology are used to reduce the number of data copies; On the other hand, a unified data result format is agreed upon to simplify result parsing. Each time Python calls an R function, it carries the necessary parameters and minimizes data redundancy. The R return result is also compressed and packaged into a form that is easy for Python to handle.

2. A data analysis software architecture design method according to claim 1, characterized in that: The steps for building a statistical analysis system are as follows: data collection: ensure that the data source is reliable and relevant to the analysis objectives, data cleaning: handle missing values, outliers and duplicate data to ensure data quality, data exploration: use statistical description and visualization methods to gain a preliminary understanding of data characteristics, discover patterns and anomalies in the data, select appropriate statistical models based on the analysis objectives and data characteristics, including linear regression and decision trees, use the selected models to fit the data, including parameter estimation and model training, evaluate the performance of the model through cross-validation methods to ensure the stability and predictive ability of the model, interpret the analysis results, and use chart tools to visualize the results to ensure that the results are easy to understand and communication, the model layer is responsible for data storage and analysis and processing, and is the core computing unit of the system, covering data reading and cleaning, field extraction, automatic identification of variable types, statistical description calculations, hypothesis testing and model building functions. The model layer is implemented using object-oriented and factory method patterns, and dynamically creates corresponding data processing objects according to different data analysis needs. For different types of data sets or statistical analysis methods, the model layer selects the appropriate algorithm module instance through the factory pattern. The model layer has the calling logic for R language statistical functions, and performs specific statistical calculations through the custom R function interface. After obtaining the analysis results, the model layer generates a structured result object and returns the result to the controller layer.

3. A data analysis software architecture design method according to claim 2, characterized in that: The view layer provides a graphical user interface, interacts directly with the user, displays the analysis results generated by the model layer in a friendly manner, and receives user input operations. The PySide6 framework is used to build a cross-platform GUI. The interface includes a data table display area, a statistical chart drawing area, and a log output area component. The adapter mode is introduced: by writing an adapter, different types of underlying data are converted into a form that is directly displayed by interface controls. The view interface design focuses on intuitiveness and ease of use, providing drop-down menus and forms for users to select statistical methods, set parameters, and trigger analysis runs through buttons. The results are presented intuitively in the form of charts and tables, which is convenient for users to understand the analysis output.

4. A data analysis software architecture design method according to claim 3, characterized in that: The controller layer is responsible for responding to user operations and coordinating models and views. The controller receives user requests from the view layer, and calls the corresponding analysis function module of the model layer to perform calculations. The results returned by the model layer are then distributed to the view layer for display. In order to ensure the orderliness and efficiency of the entire system process, the controller is designed as a singleton mode, that is, there is only one controller in the entire application for unified management, avoiding the waste of resources caused by repeated instantiation, and ensuring that calls to the model layer and the view layer are coordinated. The controller assumes the management role of module loading and expansion: through the decorator mode and plug-in mechanism, the controller dynamically loads different analysis function modules at runtime or adds new features to existing functions. When the user needs to add a new statistical analysis method, it is encapsulated into a module and registered with the controller. The controller recognizes and calls the encapsulated module without changing the core code.

5. A data analysis software architecture design method according to claim 4, characterized in that: The adaptive selection strategy is as follows: by monitoring the size of the data set to be analyzed and the resource utilization of the current system, dynamically switch between the two calling methods: when the data volume is small and the system memory is sufficient, use rpy2's direct memory connection method to obtain the fastest response; when the data volume is huge or the memory is tight, use Rserve to share the memory pressure. At the same time, introduce a data sharding transmission mechanism in Rserve mode: for extremely large data, instead of transmitting the entire data at one time, the data is automatically split into several blocks for batch transmission and processing, reducing the amount of data transmitted in a single time, reducing network transmission delay and peak memory usage, and setting a dynamic R environment management strategy: the controller will dynamically decide whether to start multiple Rserve working processes to process different data blocks or multiple tasks in parallel based on the current number of CPU cores and task requirements, ensuring that computing resources are fully utilized and not wasted. Through the above mechanism, the collaborative calling of Python and R achieves adaptive optimization, and can obtain near-optimal performance in data analysis tasks of various scales.

6. A data analysis software architecture design method according to claim 5, characterized in that: The multi-level analysis module division is specifically as follows: the system divides the functions into three levels according to the general process of statistical analysis: data preprocessing, data analysis, and result visualization. Each level is further subdivided into several independent functional modules. The data preprocessing level includes "missing value processing", "outlier detection", and "variable type conversion" modules; the data analysis level includes "descriptive statistics", "hypothesis testing", "regression analysis", and "principal component analysis" modules; and the visualization level includes "distribution histogram drawing" and "correlation matrix drawing" modules. Each module performs its own duties and is independently developed and tested.

7. A data analysis software architecture design method according to claim 6, characterized in that: The independent communication of the modules is as follows: each functional module is connected through a uniformly defined data interface, the data transmitted between modules adopts a standard format, and the modules agree on input and output: the data output by the previous module is directly used as the input of the next module, using a standard data format, and the modules do not rely on each other for specific implementation. As long as the interface contract is followed, the internal implementation of a module can be replaced without affecting other parts. There are "outlier processing" modules with different algorithms. As long as their input and output formats are consistent, they are all registered and used in the system. The modules are connected through controller scheduling. After the user configures the analysis process in the interface, the controller calls the corresponding modules in sequence, passes data in sequence, and completes the entire pipeline; module registration and dynamic loading are as follows: a module management mechanism based on factory mode and multi-layer dictionary mapping is designed. A module registry is maintained internally in the system. The first layer distinguishes the level to which the modules belong, and the second layer is based on the module. The name or function key indexes the specific module implementation object. When a new module is needed, the module class or function is registered in this dictionary mapping. When the controller calls the module, it quickly finds the corresponding implementation through the module name and instantiates it for operation. Due to the use of the factory mode, the instantiation of the module is highly abstracted. The controller does not need to understand the internal details of the module and only needs to request the module object according to the name. This dynamic loading mechanism ensures the high scalability of the system: without modifying the core code of the system, users or developers can add new statistical method modules at any time in the form of plug-ins, and the system immediately supports this method. The module management dictionary stores the mapping rules between modules and applicable data types or scenarios. When called, the system automatically matches the optimal analysis module according to the type or size of the input data. For data sets with categorical variables, the statistical description module will automatically select the chi-square test path, while for continuous variables, it will select the t-test or variance analysis path.

Images