Data simulation test method, device, equipment and medium

Abstract

The invention relates to the technical field of data processing, and discloses a data simulation test method, which comprises the following steps of: obtaining pipeline data corresponding to each data platform in a preset message queue, and performing data processing on the pipeline data according to a data state and a preset data set to obtain target data; screening out all target data corresponding to the same service information from all the target data, and performing data aggregation on all the target data corresponding to the same service information to obtain aggregated data of all the target data; performing a simulation test on all the aggregated data to obtain a simulation test result; and carrying out statistical analysis on all simulation test results to obtain a simulation statistical result. According to the method, data processing is carried out on all the pipeline data, it is ensured that the granularity of all the data meets the requirement, generation of the test case is achieved, and therefore testing and inspection of the data are achieved, and then the efficiency of data simulation testing and the accuracy of simulation statistical results are improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data simulation test method, device, equipment and medium. Background Art

[0002] In the prior art, simulation technology is widely used to simulate the behavior of complex systems for the purpose of research, testing and optimization. Existing simulation test methods are not sophisticated enough to simulate real scenarios, including but not limited to the scenarios and data at the time, and the granularity of collected data cannot support simulation accuracy, and cannot simulate quickly and accurately. If simulation is to meet real application scenarios, it is necessary to ensure that the simulation capability has a high degree of accuracy, which results in the inability of existing simulation tests to meet the requirements. Summary of the invention

[0003] The embodiments of the present invention provide a data simulation test method, device, equipment and medium to solve the problems of low simulation efficiency and low accuracy in the prior art.

[0004] A data simulation test method, comprising: Obtaining the pipeline data corresponding to each data platform in the preset message queue, and performing data processing on all the pipeline data according to the data status corresponding to each pipeline data and the preset data set, to obtain the target data corresponding to each pipeline data; All the target data corresponding to the same business information are screened out from all the target data, and data aggregation is performed on all the target data corresponding to the same business information to obtain aggregated data of all the target data corresponding to the same business information; the business information refers to a business field used to associate each of the data platforms; Perform simulation tests on all the aggregated data through a preset test process to obtain simulation test results corresponding to each of the aggregated data; A statistical analysis is performed on all the simulation test results using a preset statistical tool to obtain simulation statistical results.

[0005] A data simulation test device, comprising: A data processing module is used to obtain the pipeline data corresponding to each data platform in the preset message queue, and perform data processing on all the pipeline data according to the data status corresponding to each pipeline data and the preset data set to obtain the target data corresponding to each pipeline data; A data aggregation module, used for filtering out all the target data corresponding to the same business information from all the target data, and performing data aggregation on all the target data corresponding to the same business information to obtain aggregated data of all the target data corresponding to the same business information; the business information refers to a business field used to associate each of the data platforms; A simulation test module, used to perform simulation tests on all the aggregated data through a preset test process to obtain simulation test results corresponding to each of the aggregated data; The statistical analysis module is used to perform statistical analysis on all the simulation test results through a preset statistical tool to obtain simulation statistical results.

[0006] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor is used to execute the above-mentioned data simulation test method.

[0007] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the data simulation test method is implemented.

[0008] The above-mentioned data simulation test method, device, equipment and medium, the method realizes the acquisition of target data by performing data processing on all pipeline data obtained from the preset message queue, thereby ensuring that the granularity of all data meets the requirements, thereby improving the accuracy of the simulation statistical results. By performing data aggregation on all target data, the acquisition of aggregated data is realized, thereby realizing the generation of test cases, thereby improving the efficiency of data simulation testing. By performing simulation testing on all aggregated data through a preset test process, the testing and verification of data is realized. Statistical analysis is performed on all simulation test results, the acquisition of simulation statistical results is realized, and the statistics of simulation test results are realized, thereby realizing the difference detection of simulation statistical results, thereby improving the accuracy of simulation statistical results. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0010] Figure 1 is a flow chart of a data simulation test method in one embodiment of the present invention; Figure 2 is a flow chart of step S10 of the data simulation test method in one embodiment of the present invention; Figure 3 is a flow chart of step S10 of a data simulation test method in another embodiment of the present invention; Figure 4 is a flow chart of step S104 of the data simulation test method in one embodiment of the present invention; Figure 5 is a flow chart of step S20 of the data simulation test method in one embodiment of the present invention; Figure 6 is a flow chart of step S30 of the data simulation test method in one embodiment of the present invention; Figure 7 is a flow chart of step S40 of the data simulation test method in one embodiment of the present invention; Figure 8 is a principle block diagram of a data simulation test device in one embodiment of the present invention; Fig. 9 is a schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0011] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0012] In one embodiment, if Figure 1 As shown, a data simulation test method is provided, comprising the following steps: S10: Obtain the pipeline data corresponding to each data platform in the preset message queue, and process all the pipeline data according to the data status corresponding to each pipeline data and the preset data set to obtain the target data corresponding to each pipeline data.

[0013] Understandably, a data platform refers to a system or platform that generates business data, such as a process engine, a decision engine, a field platform, and a model platform. A preset message queue refers to a message queue used to store data, such as a kafka message queue, a RabbitMQ message queue, and a RocketMQ message queue. A data platform refers to a platform that is set up in advance to obtain data, and each platform is interconnected. Flow data refers to business data that has changed in the platform, such as new business data, modified business data, and so on. Target data refers to processed flow data. The data state is used to characterize the operation type corresponding to the update operation of the flow data, such as adding new data, modifying data, and so on. The processing method refers to the way of processing data, such as deduplication, modification, and other operations. The preset data set refers to the collection of all external data.

[0014] Specifically, the pipeline data obtained from at least one data platform is obtained from the preset message queue. Then, data processing is performed on all the pipeline data, that is, all the pipeline data are first parsed, the state of each pipeline data is determined, and all the pipeline data corresponding to each state are clustered, so as to obtain all the pipeline data corresponding to each state. Then, the processing method of the pipeline data under the data state is matched by the data state of the pipeline data. When the match is successful, the processing method corresponding to the data state is obtained, and all the pipeline data of the state are processed by the processing method corresponding to the same state, so as to obtain the data of the processed data corresponding to each pipeline data. Then, data adjustment is performed on all the processed data through the preset data set, that is, some information in the processed data is changed, so as to obtain the target data corresponding to each pipeline data.

[0015] S20: Filter out all the target data corresponding to the same business information from all the target data, and aggregate all the target data corresponding to the same business information to obtain aggregated data of all the target data corresponding to the same business information; the business information refers to the business field used to associate each of the data platforms.

[0016] It can be understood that aggregate data refers to data obtained by merging related target data. Business information refers to business fields used to associate the data platforms.

[0017] Specifically, all data contents in each target data are identified through a preset identification tool, so as to obtain the business fields corresponding to the business information in each target data. And all target data are screened through the business fields corresponding to each target data, so as to obtain all target data corresponding to the same business information. Then, all target data corresponding to the same business information are merged, that is, all target data corresponding to the business information are sorted and spliced ​​according to the data order in the business information, so as to obtain the aggregated data corresponding to the business information. In this way, all target data corresponding to each business information are merged in the above manner to obtain the aggregated data corresponding to each target data.

[0018] S30: Perform simulation tests on all the aggregated data through a preset test process to obtain simulation test results corresponding to each of the aggregated data.

[0019] Understandably, the preset test process refers to the process of test data set in advance. The simulation test result refers to the result obtained through the simulation test, which is used to indicate whether the aggregated data meets the requirements of each stage in the test process.

[0020] Specifically, a preset test process is obtained, and all aggregated data are divided according to attributes, that is, divided according to time attributes, so as to obtain aggregated data of different batches. Then, all aggregated data in each batch are input into the preset test process according to batches, and all aggregated data are simulated through the preset test process, that is, the process operation is first performed on all aggregated data, and then the decision operation is performed according to the obtained process result, and then the obtained decision result is identified, and finally, the obtained identification result is simulated and tested, and the test result is compared with the initial data, so as to obtain the simulation test result corresponding to each aggregated data.

[0021] S40: Performing statistical analysis on all the simulation test results using a preset statistical tool to obtain simulation statistical results.

[0022] It can be understood that the simulation statistical results refer to the results obtained by testing and statistically analyzing the data, which are used to indicate whether the simulation test meets the standards.

[0023] Specifically, a preset statistical tool is obtained, and statistical analysis is performed on all simulation test results through the preset statistical tool, that is, result statistics are performed on all simulation test results through the preset statistical tool to obtain test statistical results; distribution statistics are performed on all simulation test results through the preset statistical tool to obtain distribution statistical results; abnormal statistics are performed on all simulation test results through the preset statistical tool to obtain abnormal statistical results. Finally, the test statistical results, distribution statistical results, and abnormal statistical results are determined as simulation statistical results.

[0024] In the data simulation test method of the present invention, by performing data processing on all pipeline data obtained from a preset message queue, the acquisition of target data is achieved, thereby ensuring that the granularity of all data meets the requirements, thereby improving the accuracy of the simulation statistical results. By performing data aggregation on all target data, the acquisition of aggregated data is achieved, thereby achieving the generation of test cases, thereby improving the efficiency of data simulation testing. By performing simulation testing on all aggregated data through a preset test process, the testing and verification of data is achieved. By performing statistical analysis on all simulation test results, the acquisition of simulation statistical results is achieved, and the statistics of simulation test results are achieved, thereby achieving the detection of differences in simulation statistical results, thereby improving the accuracy of simulation statistical results.

[0025] In one embodiment, if Figure 2 As shown, in step S10, the flow data corresponding to each data platform in the preset message queue is obtained, including: S101, monitor each of the data platforms, and when monitoring that the data in each of the data platforms is updated, obtain the flow data of the update operation in each of the data platforms, and send the flow data to a preset message queue.

[0026] It can be understood that an update operation refers to a change in data, such as modifying data or adding data.

[0027] Specifically, each data platform is monitored through a monitoring plug-in or a monitoring function. When the data in the data platform is updated, that is, the data information changes, the changed data is collected, and the collected data corresponding to each data platform is determined as flow data. Then, all flow data are stored in a preset message queue.

[0028] In a specific embodiment, first confirm whether the binary log in the data platform is turned on, that is, confirm by checking the log_bin option in the my.cnf or my.ini configuration file, and confirm that the format of the binlog is ROW (row level) or MIXED (mixed). Then, obtain the changed flow data from the binary log of each data platform through one or more monitoring methods of MySQLbinlog (MySQL's own command line tool, which can read and parse binlog files), Maxwell (Java-based real-time MySQL change data capture (CDC) framework), Debezium (open source distributed platform for capturing change data from multiple databases) and Canal (based on MySQL database incremental log parsing, providing incremental data subscription and consumption components), and then send all the flow data to the preset kafka message queue.

[0029] In this embodiment, by monitoring the data platform, the monitoring of data changes in the data platform is achieved, the real-time acquisition of the flow data is achieved, and then the sending of the flow data is achieved.

[0030] In one embodiment, if Figure 3 As shown, in the step S10, data processing is performed on all the pipeline data according to the data status corresponding to each pipeline data and the preset data set to obtain target data corresponding to each pipeline data, including: S102, parsing each of the pipeline data to obtain a data state corresponding to each of the pipeline data; the data state is used to characterize an operation type corresponding to an update operation of the pipeline data.

[0031] S103, determining a processing method corresponding to each data state from a processing database according to the data state.

[0032] S104, clustering all the pipeline data according to the data state to determine all the pipeline data associated with the same data state.

[0033] S105, obtaining a preset data set, performing data processing on all pipeline data corresponding to the data state through the preset data in the preset data set and the processing method corresponding to each data state, to obtain target data corresponding to each pipeline data.

[0034] It can be understood that the processing database refers to a database used to store processing methods.

[0035] Specifically, after obtaining the flow data, all the flow data are parsed to identify the field information in each flow data, and the data state corresponding to the flow data is determined according to the content corresponding to the field information. Then, the processing method corresponding to each data state is obtained from the processing database through the data state, that is, the data state and the preset state corresponding to the processing method are matched. When the data state and the preset state match successfully, the processing method corresponding to the preset state is determined as the processing method corresponding to the data state, so that the processing method corresponding to each data state can be obtained. At the same time, after determining the data state, the flow data corresponding to the same data state are clustered, that is, each data state is used as a cluster center, and each flow data belongs to the cluster center. All the flow data belonging to the same cluster center are clustered, so as to obtain all the flow data corresponding to the same data state. Furthermore, a preset data set is obtained from the database, and data processing is performed on all pipeline data corresponding to the same data state through the processing method and the preset data in the preset data set. That is, all pipeline data corresponding to the data state are processed in sequence according to the operations in the processing method through the processing method corresponding to the pipeline data, so as to obtain processed data, and the processed data is adjusted through the preset data in the preset data set, so as to obtain target data corresponding to each pipeline data.

[0036] In this embodiment, by analyzing the pipeline data, the data state is determined, thereby matching the processing method, processing all the pipeline data, and further acquiring the target data.

[0037] In one embodiment, if Figure 4 As shown, in the step S105, that is, all the pipeline data corresponding to the same data state are processed by the processing method and the preset data in the preset data set to obtain the target data corresponding to each of the pipeline data, including: S1051, when the processing method is deduplication processing, all the pipeline data are deduplicated using data information corresponding to each of the pipeline data to obtain deduplicated data.

[0038] S1052, adjusting all the deduplicated data using the preset data in the preset data set to obtain the deduplicated adjusted data in the target data.

[0039] S1053, when the processing method is mock processing, all the pipeline data are mock processed according to a preset standard to obtain simulated data.

[0040] S1054, adjusting all the simulation data by using the preset data in the preset data set to obtain simulation adjustment data in the target data.

[0041] Understandably, deduplication data refers to data that is not repeated after deduplication of pipeline data. Simulation data refers to data after field mocking of pipeline data. Deduplication adjustment data refers to data obtained by adjusting the information in deduplication data through external data. Simulation adjustment data refers to data obtained by adjusting the information in simulation data through external data. Mock processing refers to further processing, configuration or adjustment of information in pipeline data to turn it into simulation data.

[0042] Specifically, after obtaining the processing method, all the pipeline data and processing methods corresponding to the same data state are obtained. When the processing method is deduplication processing, all pipeline data are deduplicated through the data information corresponding to each pipeline data, that is, a set set is obtained, and all pipeline data are stored in the set set. At this time, the set set calculates the hash value of all pipeline data through a hash function to obtain a hash value, and deletes the repeated hash values ​​to obtain deduplication data. Then, all deduplication data are adjusted and processed through the preset data in the preset data set, that is, the data platform corresponding to the deduplication data is first determined, and the preset data corresponding to the data platform is screened out from the preset data set, and then the preset data corresponding to the data state of the deduplication data is screened out, and the information in the deduplication data is adjusted through the preset data, that is, part of the information in the deduplication data is adjusted, so as to obtain the deduplication adjustment data in the target data. Similarly, when the processing method is mock processing, all pipeline data are mocked through the preset standard, that is, the preset standard is obtained, and the information in all pipeline data is replaced and modified through the preset standard, so as to obtain the simulated data. Then, all the simulation data are adjusted and processed by the preset data in the preset data set, that is, the data platform corresponding to the simulation data is first determined, the preset data corresponding to the data platform is screened out from the preset data set, and then the preset data corresponding to the data state of the simulation data is screened out, and the information in the simulation data is adjusted by the preset data, so as to obtain the simulation adjustment data in the target data. It can be understood that the above two processing methods are only for illustration and are not intended to be limiting, and there may be other processing methods.

[0043] In this embodiment, each pipeline data is processed respectively by a processing method corresponding to each data state, thereby realizing the processing of deduplicated data and simulated data and avoiding subsequent dependence on external data.

[0044] In one embodiment, if Figure 5 As shown, in step S20, all the target data corresponding to the same business information are screened out from all the target data, and data aggregation is performed on all the target data corresponding to the same business information to obtain aggregated data of all the target data corresponding to the same business information, including: S201, identifying data fields corresponding to business information in each target data, and filtering out all the target data corresponding to the same business information according to the data fields.

[0045] S202, concatenating all the target data corresponding to the same business information to obtain aggregated data.

[0046] Understandably, data fields refer to field information of flow data, such as user data and business data, etc. Business information refers to fields that are interrelated between businesses on various data platforms.

[0047] Specifically, after obtaining the target data. Field identification is performed on all target data, that is, a preset identification tool is obtained, and all target data are input into the preset identification tool, and all data contents in each target data are identified by the preset identification tool, so as to obtain the data field corresponding to each target data. Then, business information is obtained, and the fields and data fields in each business information are matched. When the fields and data fields in the business information are successfully matched, the target data corresponding to the data field that successfully matches the fields in the business information is determined as the target data corresponding to the business information, and all target data corresponding to the same business information are screened out. Further, data aggregation is performed on all target data corresponding to the same business information, that is, multiple target data corresponding to the same business information are connected according to the order of the fields in the business information, or an aggregation function is obtained, and one or more target data corresponding to the same business information are aggregated by the aggregation function, and the aggregated data are spliced ​​in order, so as to obtain the aggregated data corresponding to each business information. Among them, all aggregated data are compressed and stored in a preset database.

[0048] In this embodiment, the data fields are identified, thereby achieving the screening of target data, and then achieving the acquisition of aggregated data, achieving the generation of test cases, and improving the efficiency of subsequent tests.

[0049] In one embodiment, if Figure 6 As shown, in the step S30, that is, a simulation test is performed on all the aggregated data through a preset test process to obtain a simulation test result corresponding to each of the aggregated data, including: S301, performing process detection on all the aggregated data through the platform process in the preset test process to obtain process detection data corresponding to each of the aggregated data; the process detection data includes decision information used by the aggregated data.

[0050] S302: Perform a decision operation on all the aggregated data using the decision information to obtain decision operation data corresponding to each of the aggregated data.

[0051] S303, performing field identification on all the decision operation data to obtain field data corresponding to each decision operation data.

[0052] S304, performing simulation test on all the field data to obtain simulation data corresponding to each field data, and performing comparison processing on the simulation data and initial data to obtain simulation test results corresponding to each simulation data.

[0053] Understandably, process detection data refers to data obtained by executing process operations on aggregated data, which is used to characterize the process information to be executed, including decision information used by aggregated data. Decision operation data refers to data obtained by executing decision operations, which is used to characterize information that can reflect aggregated data. Field data refers to fields obtained by identifying decision operation data, which refers to data information that needs to be tested. Simulation data refers to data obtained by processing field data through the model platform, which is used to characterize the test results of the entire process on aggregated data.

[0054] Specifically, after obtaining the aggregated data, the initial data corresponding to each data platform is obtained, and the test process is constructed through all the initial data. Then, the process platform of the first stage in the platform process of the preset test process is used to perform process detection on all aggregated data in batches to determine the decision information to be executed by the subsequent aggregated data, thereby obtaining the process detection data corresponding to each aggregated data in the batch. Next, the decision platform of the next stage in the preset test process is used to perform decision detection on the aggregated data according to the decision information in the process detection data, and it is determined whether the aggregated data conforms to the decision operation, thereby obtaining the decision operation data corresponding to each aggregated data. Further, the field platform of the next stage in the preset test process is used to perform field detection on all decision operation data, and it is determined whether the decision operation data conforms to the field requirements, thereby obtaining the field data corresponding to each decision operation data. Finally, all field data are simulated and tested through the model platform to determine whether the field data conforms to the test requirements, thereby obtaining the simulation data corresponding to each field data. And the simulation data corresponding to the same aggregated data is compared with the initial data, that is, the information in the simulation data is compared with the information in the initial data, thereby obtaining the simulation test results corresponding to each aggregated data.

[0055] In this embodiment, by sequentially performing process operations, decision operations, field identification and simulation tests on aggregated data, the acquisition of process detection data, decision operation data, field data and simulation data is achieved, thereby achieving the test of aggregated data. The simulation data and the initial data are compared and processed to achieve the determination of the difference between the simulation and the actual data, thereby achieving the acquisition of the simulation test results.

[0056] In one embodiment, if Figure 7 As shown, in step S40, all the simulation test results are statistically analyzed by a preset statistical tool to obtain simulation statistical results, including: S401, obtaining a preset statistical tool, and inputting all the simulation test results into the preset statistical tool.

[0057] S402: Performing statistics on all the simulation test results using the preset statistical tool to obtain test statistical results.

[0058] S403, performing distribution statistics on all the simulation test results using the preset statistical tool to obtain distribution statistical results.

[0059] S404: Perform abnormal statistics on all the simulation test results using the preset statistical tool to obtain abnormal statistical results.

[0060] S405, determining the simulation statistical result according to the test statistical result, the distribution statistical result and the abnormal statistical result.

[0061] Understandably, the preset statistical tool refers to a tool for performing statistics on simulation results, such as a BI (Business Intelligence) reporting tool.

[0062] Specifically, after obtaining the simulation test results, a preset statistical tool is obtained, and all simulation test results that have been simulated are input into the preset statistical tool. Then, the preset statistical tool is used to perform result statistics on all simulation test results, that is, the statistical tool is used to perform result statistics on the type of each simulation test result, so as to obtain the test statistical results. Next, the preset statistical tool is used to perform distribution statistics on all simulation test results, that is, the statistical tool is used to perform statistics on the distribution of each simulation test result, so as to obtain the distribution statistical results. Finally, the preset statistical tool is used to perform abnormal statistics on all simulation test results, that is, the statistical tool is used to perform statistics on the abnormal situation of each simulation test result, that is, the simulation test results with different simulation data and initial data are determined as abnormal simulation test results, so as to obtain abnormal statistical results. Finally, the simulation statistical results are determined according to the test statistical results, the distribution statistical results and the abnormal statistical results. Among them, the abnormal simulation test results can be parsed to determine the specific abnormal details, that is, to determine the specific different contents of the simulation data and the initial data.

[0063] In this embodiment, the acquisition of test statistical results is achieved through the preset statistical tool, thereby achieving statistics on result distribution and acquisition of distribution statistical results, and further achieving statistics on abnormal results and acquisition of abnormal statistical results.

[0064] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0065] In one embodiment, a data simulation test device is provided, and the data simulation test device corresponds one-to-one to the data simulation test method in the above embodiment. Figure 8 As shown, the data simulation test device includes a data processing module 10, a data aggregation module 20, a simulation test module 30 and a statistical analysis module 40. The functional modules are described in detail as follows: The data processing module 10 is used to obtain the pipeline data corresponding to each data platform in the preset message queue, and perform data processing on all the pipeline data according to the data state corresponding to each pipeline data and the preset data set to obtain the target data corresponding to each pipeline data; The data aggregation module 20 is used to filter out all the target data corresponding to the same business information from all the target data, and aggregate all the target data corresponding to the same business information to obtain aggregated data of all the target data corresponding to the same business information; the business information refers to the business field used to associate each of the data platforms; A simulation test module 30, configured to perform simulation tests on all the aggregated data through a preset test process to obtain simulation test results corresponding to each of the aggregated data; The statistical analysis module 40 is used to perform statistical analysis on all the simulation test results by using a preset statistical tool to obtain simulation statistical results.

[0066] Optionally, the data processing module 10 includes: The data acquisition unit is used to monitor each of the data platforms, and when it monitors that the data in each of the data platforms is updated, it acquires the flow data of the update operation in each of the data platforms, and sends the flow data to a preset message queue.

[0067] Optionally, the data processing module 10 further includes: A data status unit, used for parsing each of the pipeline data to obtain a data status corresponding to each of the pipeline data; the data status is used to characterize an operation type corresponding to an update operation of the pipeline data; A processing mode unit, used for determining a processing mode corresponding to each of the data states from a processing database according to the data states; A data clustering unit, used for clustering all the pipeline data according to the data state, and determining all the pipeline data associated with the same data state; The data processing unit is used to obtain a preset data set, and to process all the pipeline data corresponding to the data state through the preset data in the preset data set and the processing method corresponding to each data state, so as to obtain the target data corresponding to each pipeline data.

[0068] Optionally, the data aggregation module 20 includes: A data screening unit, used to identify the data fields corresponding to the business information in each of the target data, and screen out all the target data corresponding to the same business information according to the data fields; The data aggregation unit is used to perform data splicing on all the target data corresponding to the same business information to obtain aggregated data.

[0069] Optionally, the simulation test module 30 includes: A process operation unit, used to perform process detection on all the aggregated data through the platform process in the preset test process, and obtain process detection data corresponding to each of the aggregated data; the process detection data includes decision information used by the aggregated data; A decision operation unit, configured to perform a decision operation on all the aggregated data using the decision information to obtain decision operation data corresponding to each of the aggregated data; A field identification unit, used to perform field identification on all the decision operation data to obtain field data corresponding to each of the decision operation data; The simulation test unit is used to perform simulation tests on all the field data to obtain simulation data corresponding to each field data, and compare the simulation data with the initial data to obtain simulation test results corresponding to each simulation data.

[0070] Optionally, the statistical analysis module 40 includes: A statistical tool unit, used for obtaining a preset statistical tool and inputting all the simulation test results into the preset statistical tool; A result statistics unit, used for performing result statistics on all the simulation test results through the preset statistical tool to obtain test statistical results; A distribution statistics unit, used for performing distribution statistics on all the simulation test results by using the preset statistical tool to obtain distribution statistics results; An abnormality statistics unit, used for performing abnormality statistics on all the simulation test results through the preset statistical tool to obtain abnormality statistical results; A result determination unit is used to determine the simulation statistical result according to the test statistical result, the distribution statistical result and the abnormal statistical result.

[0071] Optionally, the data processing unit includes: A data deduplication subunit, used for, when the processing mode is deduplication processing, performing data deduplication on all the pipeline data by using data information corresponding to each of the pipeline data to obtain deduplication data; a deduplication data adjustment subunit, configured to adjust all the deduplication data by using the preset data in the preset data set to obtain the deduplication adjusted data in the target data; A mock processing subunit is used to perform mock processing on all the pipeline data according to a preset standard to obtain simulated data when the processing mode is mock processing; The simulation data adjustment subunit is used to adjust all the simulation data through the preset data in the preset data set to obtain the simulation adjustment data in the target data.

[0072] This application further provides Fig. 9 A computer device is shown, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is used to execute the data simulation test method.

[0073] The specific definition of computer equipment, processor and its units and modules can refer to the definition of data simulation test method above, which will not be repeated here. Each module in the above processor can be implemented in whole or in part by software, hardware and combination thereof. Understandably, the processor includes a processor, memory, network interface and database connected by a device bus. Each module of the processor can be embedded in or independent of the processor in hardware form, or can be stored in the memory in software form, so that the processor calls and executes the operations corresponding to the above modules. Among them, the processor is used to provide computing and control capabilities. The memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating device, a computer program and a database. The internal memory provides an environment for the operation of the operating device and the computer program in the non-volatile storage medium. The database is used to store the data used by the data simulation test method in the above embodiment. The network interface is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data simulation test method is implemented.

[0074] In one embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned data simulation test method is implemented.

[0075] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0076] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0077] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A data simulation test method, characterized in that: include: Obtaining the pipeline data corresponding to each data platform in the preset message queue, and performing data processing on all the pipeline data according to the data status corresponding to each pipeline data and the preset data set, to obtain the target data corresponding to each pipeline data; All the target data corresponding to the same business information are screened out from all the target data, and data aggregation is performed on all the target data corresponding to the same business information to obtain aggregated data of all the target data corresponding to the same business information; the business information refers to a business field used to associate each of the data platforms; Perform simulation tests on all the aggregated data through a preset test process to obtain simulation test results corresponding to each of the aggregated data; A statistical analysis is performed on all the simulation test results using a preset statistical tool to obtain simulation statistical results.

2. The data simulation test method according to claim 1, characterized in that: The obtaining of the flow data corresponding to each data platform in the preset message queue includes: Monitor each of the data platforms, and when monitoring that the data in each of the data platforms is updated, obtain the flow data of the update operation in each of the data platforms, and send the flow data to a preset message queue.

3. The data simulation test method according to claim 2, characterized in that: The step of processing all the pipeline data according to the data status corresponding to each pipeline data and the preset data set to obtain the target data corresponding to each pipeline data includes: Parsing each of the pipeline data to obtain a data state corresponding to each of the pipeline data; the data state is used to characterize an operation type corresponding to an update operation of the pipeline data; Determine, from a processing database according to the data status, a processing method corresponding to each of the data statuses; Performing clustering processing on all the pipeline data according to the data state to determine all the pipeline data associated with the same data state; A preset data set is obtained, and all pipeline data corresponding to the data state are processed using the preset data in the preset data set and the processing method corresponding to each data state to obtain target data corresponding to each pipeline data.

4. The data simulation test method according to claim 1, characterized in that: The step of selecting all the target data corresponding to the same business information from all the target data, and aggregating all the target data corresponding to the same business information to obtain aggregated data of all the target data corresponding to the same business information includes: Identify the data fields corresponding to the business information in each of the target data, and filter out all the target data corresponding to the same business information according to the data fields; All the target data corresponding to the same business information are concatenated to obtain aggregated data.

5. The data simulation test method according to claim 1, characterized in that: The performing simulation test on all the aggregated data through a preset test process to obtain simulation test results corresponding to each of the aggregated data includes: Performing process detection on all the aggregated data through the platform process in the preset test process to obtain process detection data corresponding to each of the aggregated data; the process detection data includes decision information used by the aggregated data; Performing a decision operation on all the aggregated data using the decision information to obtain decision operation data corresponding to each of the aggregated data; Performing field identification on all the decision operation data to obtain field data corresponding to each of the decision operation data; A simulation test is performed on all the field data to obtain simulation data corresponding to each field data, and the simulation data is compared with the initial data to obtain a simulation test result corresponding to each simulation data.

6. The data simulation test method according to claim 1, characterized in that: The statistical analysis of all the simulation test results is performed by a preset statistical tool to obtain simulation statistical results, including: Obtaining a preset statistical tool, and inputting all of the simulation test results into the preset statistical tool; Performing statistics on all the simulation test results by using the preset statistical tool to obtain test statistical results; Performing distribution statistics on all the simulation test results by using the preset statistical tool to obtain distribution statistical results; Perform abnormal statistics on all the simulation test results by using the preset statistical tool to obtain abnormal statistical results; The simulation statistical result is determined according to the test statistical result, the distribution statistical result and the abnormal statistical result.

7. The data simulation test method according to claim 3, characterized in that: The processing of all the pipeline data corresponding to the same data state by using the processing method and the preset data in the preset data set to obtain target data corresponding to each of the pipeline data includes: When the processing method is deduplication processing, all the pipeline data are deduplicated using data information corresponding to each of the pipeline data to obtain deduplicated data; Adjusting all the deduplicated data using the preset data in the preset data set to obtain deduplicated adjusted data in the target data; When the processing method is mock processing, all the pipeline data are mock processed according to a preset standard to obtain simulated data; All the simulation data are adjusted by using the preset data in the preset data set to obtain the simulation adjustment data in the target data.

8. A data simulation test device, characterized in that: include: A data processing module is used to obtain the pipeline data corresponding to each data platform in the preset message queue, and perform data processing on all the pipeline data according to the data status corresponding to each pipeline data and the preset data set to obtain the target data corresponding to each pipeline data; A data aggregation module, used for filtering out all the target data corresponding to the same business information from all the target data, and performing data aggregation on all the target data corresponding to the same business information to obtain aggregated data of all the target data corresponding to the same business information; the business information refers to a business field used to associate each of the data platforms; A simulation test module, used to perform simulation tests on all the aggregated data through a preset test process to obtain simulation test results corresponding to each of the aggregated data; The statistical analysis module is used to perform statistical analysis on all the simulation test results through a preset statistical tool to obtain simulation statistical results.

9. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor is used to execute the data simulation test method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the data simulation test method according to any one of claims 1 to 7 is implemented.

Images