Automatic test platform for data analysis

Abstract

The invention relates to an automatic test platform, and discloses an automatic test platform for data analysis, which comprises a data acquisition module for concurrently acquiring data from a plurality of data sources based on a composite means; the data storage module is used for storing the collected data in a partitioned manner according to characteristics and enabling the data to be safely shared and exchanged through an intelligent contract and an encryption algorithm; and the data processing module is used for processing the collected data based on a deep learning algorithm to obtain an analysis result, and the data testing module is used for verifying and testing the analysis result according to a preset strategy, and recording and feeding back a testing result and a process in an excl. According to the method, the block chain technology and the deep learning algorithm are combined, so that high-efficiency collection, safe storage and intelligent processing of high-capacity concurrent synchronous data are realized.

Description

Technical Field

[0001] The invention relates to an automated testing platform, and in particular to an automated testing platform for data analysis. Background Art

[0002] With the rapid development of information technology, big data processing and analysis has become an indispensable part of all industries. Especially in the collection and management of large-capacity concurrent synchronous data, traditional methods face many challenges, such as data loss, tampering risks, and low processing efficiency.

[0003] At present, the big data processing process includes key links such as data collection, storage, processing and analysis. In the data collection stage, commonly used technologies include obtaining data from channels such as databases, API interfaces, file systems, and web crawlers. As disclosed in Publication No.: CN113792029B, Publication Date: 2023-12-01, a rapid development framework and construction method for big data processing and analysis models are disclosed, the framework includes: a model management component, which is used to receive an analysis model that meets the definition according to the provided unified model interface, and register and store the analysis model for query, and also use the general analysis model as a general model for calling; a task architecture component, which is used to receive and manage multiple tasks corresponding to the analysis model, and execute the analysis model after binding the task to the corresponding analysis model; a page display component, which is used to display relevant information in the model management component and the task architecture component; an infrastructure component, which is used to provide basic functions and external service interfaces and API interfaces. The present invention can uniformly develop, manage, and run a large number of models, reduce the workload of model developers, and effectively improve the efficiency of model development.

[0004] In the prior art based on the above patent, when processing large-capacity concurrent data, problems such as inconsistent data synchronization and processing delays often occur. It is difficult to ensure the security and reliability of data storage, and the risk of data loss and tampering during data processing always exists. Summary of the invention

[0005] The purpose of the present invention is to provide an automated testing platform for data analysis to solve the above problems.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] An automated testing platform for data analysis, comprising:

[0008] Data collection module, which collects data from multiple data sources concurrently based on composite means;

[0009] The data storage module partitions and stores the collected data according to its characteristics and uses smart contracts and encryption algorithms to enable secure sharing and exchange of data;

[0010] Data processing module, which processes the collected data based on deep learning algorithms to obtain analysis results

[0011] The data testing module verifies and tests the analysis results according to a predetermined strategy, and records and feeds back the test results and process in execl.

[0012] Preferably, the data acquisition module includes:

[0013] The concurrent data acquisition submodule is configured to receive data from multiple data sources simultaneously and ensure the consistency of the acquired data based on the timestamp and hash verification mechanism;

[0014] The data preprocessing submodule performs cleaning, format conversion, outlier detection, and extraction on the received data to obtain a first data packet and a second data packet, wherein the second data is the extracted outlier data;

[0015] The data flow control submodule dynamically adjusts the data acquisition rate based on the network status.

[0016] Preferably, the data flow control submodule establishes a prediction model through historical data and network quality data environment in adjacent cycle windows, adjusts the data collection frequency according to the prediction results, and finally predicts the data changes in the future through the prediction model.

[0017] Preferably, the data storage module comprises:

[0018] The distributed blockchain storage submodule shards the data according to specific rules and stores it in multiple sub-cloud disks created under the cloud disk;

[0019] Smart contract and encryption submodule, which defines data access rules through smart contracts and uses asymmetric encryption algorithms to ensure secure data transmission and sharing;

[0020] Preferably, the data processing module includes:

[0021] The deep learning model training submodule integrates multiple deep learning frameworks and supports model training, tuning, and deployment to adapt to different data processing tasks.

[0022] The processing result feedback submodule collects processing results in real time, analyzes processing efficiency through machine learning algorithms, and dynamically adjusts data collection and storage strategies.

[0023] Preferably, the step of adapting the deep learning model training submodule to different data processing tasks includes:

[0024] S11, classify the data based on the support vector machine, extract the features of the input data, and divide it into data subsets of different categories;

[0025] S12, performing regression prediction on the data in each of the data subsets according to ridge regression;

[0026] S13, cluster similar data points together to form different data clusters;

[0027] S14: Eliminate the data subsets with the same data clusters.

[0028] Preferably, the data testing module includes:

[0029] The test case automatic generation submodule automatically designs and executes test cases based on data processing requirements;

[0030] The test result analysis submodule uses statistical analysis and machine learning techniques to evaluate test results and identify potential platform problems.

[0031] Preferably, the test cases automatically designed and executed in the test case automatic generation submodule are entries stored in the rule library, and a short sentence consisting of at least no more than ten characters composed of multiple entries is used as a test case.

[0032] In the above technical scheme, the present invention provides an automated testing platform for data analysis, which has the following beneficial effects: by combining blockchain technology and deep learning algorithms, it realizes efficient collection, secure storage and intelligent processing of large-capacity concurrent synchronous data.

[0033] At the same time, through the setting of automated testing modules and data sovereignty protection modules, the stability and reliability of the platform are further improved, and the compliance and security of the data are ensured. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0035] Figure 1 A module diagram provided for an embodiment of the present invention;

[0036] Figure 2 A diagram of submodules of a data acquisition module provided by an embodiment of the present invention;

[0037] Figure 3A submodule diagram of a data storage module provided by an embodiment of the present invention;

[0038] Figure 4 A diagram of submodules of a data processing module provided by an embodiment of the present invention;

[0039] Figure 5 A submodule diagram of a data test module provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] like Figure 1-5 As shown, an automated testing platform for data analysis includes:

[0042] Data collection module, which collects data from multiple data sources concurrently based on composite means;

[0043] The data storage module partitions and stores the collected data according to its characteristics and uses smart contracts and encryption algorithms to enable secure sharing and exchange of data;

[0044] Data processing module, which processes the collected data based on deep learning algorithms to obtain analysis results

[0045] The data testing module verifies and tests the analysis results according to the predetermined strategy, and records and feeds back the test results and processes in execl.

[0046] Among the above technologies, by combining blockchain technology and deep learning algorithms, efficient collection, secure storage and intelligent processing of large-capacity concurrent synchronous data are achieved.

[0047] At the same time, through the setting of automated testing modules and data sovereignty protection modules, the stability and reliability of the platform are further improved, and the compliance and security of the data are ensured.

[0048] As an embodiment further provided by the present invention, the data acquisition module includes:

[0049] The concurrent data acquisition submodule is configured to receive data from multiple data sources simultaneously and ensure the consistency of the acquired data based on the timestamp and hash verification mechanism;

[0050] The data preprocessing submodule performs cleaning, format conversion, outlier detection, and extraction on the received data to obtain a first data packet and a second data packet, wherein the second data is the extracted outlier data;

[0051] The data flow control submodule dynamically adjusts the data acquisition rate based on the network status.

[0052] Secondly, the data flow control submodule in the above embodiment establishes a prediction model through the historical data and network quality data environment in adjacent cycle windows, adjusts the data collection frequency according to the prediction results, and finally predicts the data changes in the future through the prediction model.

[0053] As another embodiment further provided by the present invention, the data storage module includes:

[0054] The distributed blockchain storage submodule shards the data according to specific rules and stores it in multiple sub-cloud disks created under the cloud disk;

[0055] Smart contract and encryption submodule, which defines data access rules through smart contracts and uses asymmetric encryption algorithms to ensure secure data transmission and sharing;

[0056] As another embodiment further provided by the present invention, the data processing module includes:

[0057] The deep learning model training submodule integrates multiple deep learning frameworks and supports model training, tuning, and deployment to adapt to different data processing tasks.

[0058] The processing result feedback submodule collects processing results in real time, analyzes processing efficiency through machine learning algorithms, and dynamically adjusts data collection and storage strategies.

[0059] The steps of adapting the above-mentioned deep learning model training submodule to different data processing tasks include:

[0060] S11, classify the data based on the support vector machine, extract the features of the input data, and divide it into data subsets of different categories;

[0061] S12, performing regression prediction on the data in each data subset according to ridge regression;

[0062] S13, cluster similar data points together to form different data clusters;

[0063] S14, eliminating data subsets with the same data clusters.

[0064] As another embodiment further provided by the present invention, the data testing module includes:

[0065] The test case automatic generation submodule automatically designs and executes test cases based on data processing requirements;

[0066] The test result analysis submodule uses statistical analysis and machine learning techniques to evaluate test results and identify potential platform problems.

[0067] As another embodiment further provided by the present invention, the test cases automatically designed and executed in the test case automatic generation submodule are entries stored in the rule library, and a short sentence consisting of at least no more than ten characters composed of multiple entries is used as a test case.

[0068] In summary, the above-mentioned technologies realize the automation of the entire process from data collection to testing by integrating multiple advanced technologies, improve the efficiency and accuracy of data analysis, and ensure the security and sharing of data.

[0069] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0070] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0071] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0072] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0073] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0074] The embodiments of the present application also provide a specific implementation of an electronic device capable of implementing all steps of the method in the above embodiments, and the electronic device specifically includes the following contents:

[0075] Processor, memory, communications interface and bus;

[0076] Wherein, the processor, memory, and communication interface communicate with each other via the bus;

[0077] The processor is used to call the computer program in the memory, and when the processor executes the computer program, all the steps in the method in the above embodiment are implemented.

[0078] The embodiments of the present application also provide a computer-readable storage medium capable of implementing all the steps of the method in the above embodiments, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, all the steps of the method in the above embodiments are implemented.

[0079] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the hardware + program embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. Although the embodiment of this specification provides the method operation steps described in the embodiment or flow chart, more or less operation steps can be included based on conventional or non-creative means. The order of steps listed in the embodiment is only one way of executing the order of many steps, and does not represent the only execution order. When the device or terminal product in practice is executed, it can be executed in sequence or in parallel according to the method shown in the embodiment or the accompanying drawings (for example, a parallel processor or a multi-threaded processing environment, or even a distributed data processing environment). The term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, product or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such a process, method, product or device. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or device including the elements. For the convenience of description, the above device is described by dividing it into various modules according to its functions. Of course, when implementing the embodiments of this specification, the functions of each module can be implemented in the same one or more software and / or hardware, or the module implementing the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The present invention is described with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the process and / or box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0080] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, the embodiments of this specification may be in the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the embodiments of this specification may be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts refer to the partial description of the method embodiment. In the description of this specification, the description of the reference term "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of this specification.

[0081] In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradicting each other. The above is only an embodiment of the embodiment of this specification and is not intended to limit the embodiment of this specification. For those skilled in the art, the embodiment of this specification may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiment of this specification shall be included in the scope of the claims of the embodiment of this specification.

Claims

1. An automated testing platform for data analysis, characterized in that: include: Data collection module, which collects data from multiple data sources concurrently based on composite means; The data storage module partitions and stores the collected data according to its characteristics and uses smart contracts and encryption algorithms to enable secure sharing and exchange of data; Data processing module, which processes the collected data based on deep learning algorithms to obtain analysis results The data testing module verifies and tests the analysis results according to a predetermined strategy, and records and feeds back the test results and process in execl.

2. The automated testing platform for data analysis according to claim 1, characterized in that: The data acquisition module comprises: The concurrent data acquisition submodule is configured to receive data from multiple data sources simultaneously and ensure the consistency of the acquired data based on the timestamp and hash verification mechanism; The data preprocessing submodule performs cleaning, format conversion, outlier detection, and extraction on the received data to obtain a first data packet and a second data packet, wherein the second data is the extracted outlier data; The data flow control submodule dynamically adjusts the data acquisition rate based on the network status.

3. The automated testing platform for data analysis according to claim 2, characterized in that: The data flow control submodule establishes a prediction model through historical data and network quality data environment in adjacent cycle windows, adjusts the data collection frequency according to the prediction results, and finally predicts the data changes in the future through the prediction model.

4. The automated testing platform for data analysis according to claim 1, characterized in that: The data storage module comprises: The distributed blockchain storage submodule divides the data into multiple sub-cloud disks created under the cloud disk according to specific rules; The smart contract and encryption sub-module defines data access rules through smart contracts and uses asymmetric encryption algorithms to ensure secure data transmission and sharing.

5. The automated testing platform for data analysis according to claim 1, characterized in that: The data processing module comprises: The deep learning model training submodule integrates multiple deep learning frameworks and supports model training, tuning, and deployment to adapt to different data processing tasks. The processing result feedback submodule collects processing results in real time, analyzes processing efficiency through machine learning algorithms, and dynamically adjusts data collection and storage strategies.

6. The automated testing platform for data analysis according to claim 5, characterized in that: The steps of adapting the deep learning model training submodule to different data processing tasks include: S11, classify the data based on the support vector machine, extract the features of the input data, and divide it into data subsets of different categories; S12, performing regression prediction on the data in each of the data subsets according to ridge regression; S13, cluster similar data points together to form different data clusters; S14: Eliminate the data subsets with the same data clusters.

7. The automated testing platform for data analysis according to claim 1, characterized in that: The data testing module includes: The test case automatic generation submodule automatically designs and executes test cases based on data processing requirements; The test result analysis submodule uses statistical analysis and machine learning techniques to evaluate test results and identify potential platform problems.

8. The automated testing platform for data analysis according to claim 1, characterized in that: The test cases automatically designed and executed in the test case automatic generation submodule are entries stored in the rule library, and are composed of multiple entries and a short sentence of at least no more than ten characters as a test case.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the automated testing platform for data analysis described in any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the automated testing platform for data analysis described in any one of claims 1 to 8 are implemented.

Images