Real-time data testing method and device, computer equipment and storage medium

Abstract

The invention relates to the field of data testing and financial science and technology, and discloses a real-time data testing method, device and equipment and a medium, and the method comprises the steps: configuring a derivative environment for a Kafka cluster, and after the Kafka cluster is logically classified, creating a derivative task of the Kafka cluster based on the derivative environment; performing rationality evaluation on the derivative task, when an evaluation result meets a preset result, querying an execution resource quantity of a task execution system corresponding to the derivative task, and starting to execute the derivative task when the execution resource quantity meets a demand quantity of an execution number task; and during execution of the derivative task, reading theme message data of the Kafka cluster, performing data desensitization and data storage on the theme message data, obtaining desensitization data, transmitting the desensitization data to a data test environment, obtaining test data, performing data integrity test on the test data, obtaining a test result, and completing the derivative task when the test result is complete. According to the invention, the accuracy of real-time data testing can be improved.

Description

Technical Field

[0001] The present invention relates to the fields of data testing and financial technology, and in particular to a real-time data testing method, device, computer equipment and storage medium. Background Art

[0002] With the accelerated development of big data and artificial intelligence technology, the amount of data in various industries and fields has been growing rapidly over time. Studies have found that in recent years, the amount of data has doubled about every three years. For example, as the financial and insurance industries develop, they have gradually accumulated a large amount of customer and related business information. If these data can be fully utilized, it will provide huge advantages for the development of enterprises, and real-time data application scenarios have emerged.

[0003] At present, the real-time scenarios in the industry are mainly based on the construction of enterprise ecosystems around the Kafka stream processing characteristics. Traditional real-time data processing scenario development needs, such as real-time indicator development, require a large amount of different types of data, which usually need to be generated in the test environment. If the data generated in the test environment is too different from the data in the production environment, the accuracy of indicator development cannot be guaranteed. At the same time, manual data generation in the test environment often faces many upstream and downstream data related parties, long data links, and involves multiple teams and systems, which makes data generation very difficult, requires a lot of manpower, and has low data accuracy. Summary of the invention

[0004] The present invention provides a real-time data testing method, device, computer equipment and storage medium to improve the accuracy of real-time data testing.

[0005] In a first aspect, a real-time data testing method is provided, comprising:

[0006] Selecting a Kafka cluster to be derived from a preconfigured data integration platform, configuring a derivative environment for the Kafka cluster, and logically classifying the Kafka cluster to obtain a logically classified Kafka cluster, and creating a derivative task for the logically classified Kafka cluster based on the derivative environment;

[0007] Performing a rationality evaluation on the derivative task to obtain an evaluation result, and when the evaluation result meets a preset result, querying the execution resource amount of the task execution system corresponding to the derivative task, and when the execution resource amount meets the demand for executing the derivative task, using the task execution system to execute the derivative task;

[0008] When the derivative task is executed, the topic message data of the Kafka cluster in production is read, the topic message data is desensitized, and the desensitized data after the data desensitization is stored;

[0009] The desensitized data is transmitted to a data testing environment to obtain test data, a data integrity test is performed on the test data to obtain a test result, and when the test result is complete, the derivative task is completed.

[0010] In a second aspect, a real-time data testing device is provided, comprising:

[0011] A derivative task creation module is used to select a Kafka cluster to be derived from a preconfigured data integration platform, configure a derivative environment for the Kafka cluster, and logically classify the Kafka cluster to obtain a logically classified Kafka cluster, and create a derivative task for the logically classified Kafka cluster based on the derivative environment;

[0012] A derivative task execution module, used to evaluate the rationality of the derivative task to obtain an evaluation result, and when the evaluation result meets the preset result, query the execution resource amount of the task execution system corresponding to the derivative task, and when the execution resource amount meets the demand for executing the derivative task, use the task execution system to execute the derivative task;

[0013] A data storage module, used for reading the topic message data of the Kafka cluster during production when the derivative task is executed, desensitizing the topic message data, and storing the desensitized data after desensitization;

[0014] The data testing module is used to transfer the desensitized data to the data testing environment, obtain test data, perform data integrity test on the test data, obtain test results, and complete the derivative task when the test results are complete.

[0015] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned objection speech mining method when executing the computer program.

[0016] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned real-time data testing method are implemented.

[0017] Compared with the problems described in the background technology, the present invention can enable users to quickly find and select the Kafka cluster to be derived from a large number of data sources by selecting the Kafka cluster to be derived from the preconfigured data integration platform, avoiding the tedious process of switching and searching between different systems and tools, and improving the operation efficiency. Among them, the preconfigured data integration platform refers to a software system specially designed to integrate different data sources, realize data flow and processing, and can provide a unified interface and management environment, which is convenient for users to centrally manage various data-related tasks. Different data integration processes can be viewed, configured and monitored on this platform, including derivative operations on Kafka clusters, etc. The Kafka cluster refers to a distributed message system that can process a large number of messages to meet the needs of large-scale data processing. In application scenarios with high real-time requirements, such as financial transactions, Internet of Things data processing, etc., the Kafka cluster can quickly receive, store and transmit messages to ensure timely processing of data, and configure the derivative environment for the Kafka cluster. The data processing process can be customized according to specific business needs. For example, the exported data can be filtered, converted, aggregated, etc. to meet the requirements of different target systems. This makes the flow of data between different environments more efficient and flexible, reduces the workload of manual data processing, and after the Kafka cluster is logically classified to obtain the logically classified Kafka cluster, the derivative task of the Kafka cluster created based on the derivative environment can improve data management efficiency and meet different business needs; the rationality of the derivative task is evaluated, and the evaluation result can ensure data quality and accuracy, optimize resource utilization, and cost control to improve the stability and reliability of the system, and query the execution resource amount of the task execution system corresponding to the derivative task to understand the execution resource amount, and then determine whether the current system has enough resources to execute the derivative task to ensure the feasibility of the task; during the execution of the derivative task, reading the topic message data of the Kafka cluster in production can obtain the latest and real-time data to ensure the integrity of the data. This is crucial for scenarios that need to reflect business status in a timely manner or perform real-time analysis.Wherein, the topic message data refers to the message data under a specific topic (Topic) in the Kafka message system, and the desensitized data is stored by the desensitized data, and the security of the data can be ensured by obtaining the desensitized data; the desensitized data is transmitted to the data test environment, and the test data can be obtained. The desensitized data can be used to simulate the real business load, and the performance of the software system is evaluated, for example, the response time, throughput and resource utilization when processing a large amount of data can be measured, which helps to determine whether the system can meet the performance requirements of the actual business, and whether it is necessary to perform performance optimization, and the test data is tested for data integrity, and the test results can be obtained. Data errors, verification processing logic, improve system quality and stability, loopholes can be found, and system integration is guaranteed, and accurate data basis, optimization business process, data accuracy reliability, support enterprise efficient operation can also be provided for decision-making. The present invention can improve the accuracy of real-time data testing. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0019] Figure 1 is a schematic diagram of an application environment of a real-time data testing method in one embodiment of the present invention;

[0020] Figure 2 is a flow chart of a real-time data testing method in one embodiment of the present invention;

[0021] Figure 3 is a structural schematic diagram of a real-time data testing device in one embodiment of the present invention;

[0022] Figure 4 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0023] Figure 5 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] The real-time data testing method provided by the embodiment of the present invention can be applied in Figure 1 In an application environment, the client communicates with the server through a network. The server can select the Kafka cluster to be derived from the preconfigured data integration platform, configure the derivative environment for the Kafka cluster, and after logically classifying the Kafka cluster to obtain the logically classified Kafka cluster, create the derivative task of the Kafka cluster based on the derivative environment; perform a rationality evaluation on the derivative task to obtain an evaluation result, and when the evaluation result meets the preset result, query the execution resource amount of the task execution system corresponding to the derivative task, and when the execution resource amount meets the demand for executing the derivative task, use the task execution system to start executing the derivative task; during the execution of the derivative task, read the topic message data of the Kafka cluster in production, perform data desensitization on the topic message data, and store the desensitized data after data desensitization; transmit the desensitized data to the data test environment to obtain test data, perform data integrity test on the test data to obtain test results, and complete the derivative task when the test result is complete.

[0026] The client may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices. The server may be implemented by an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0027] See also Figure 2 As shown, Figure 2 A flow chart of a real-time data testing method provided by an embodiment of the present invention includes the following steps:

[0028] S1. Select a Kafka cluster to be derived from a preconfigured data integration platform, configure a derivative environment for the Kafka cluster, and logically classify the Kafka cluster to obtain a logically classified Kafka cluster. Based on the derivative environment, create a derivative task for the logically classified Kafka cluster.

[0029] The embodiment of the present invention allows users to easily and quickly find and select the Kafka cluster to be derived from a large number of data sources by selecting the Kafka cluster to be derived from the preconfigured data integration platform, avoiding the tedious process of switching and searching between different systems and tools, and improving operational efficiency. Among them, the preconfigured data integration platform refers to a software system specially designed to integrate different data sources, realize data flow and processing, and can provide a unified interface and management environment, which is convenient for users to centrally manage various data-related tasks. Different data integration processes can be viewed, configured and monitored on this platform, including derivative operations on Kafka clusters, etc. The Kafka cluster refers to a distributed message system that can process a large number of messages to meet the needs of large-scale data processing. In application scenarios with high real-time requirements, such as financial transactions, Internet of Things data processing, etc., the Kafka cluster can quickly receive, store and transmit messages to ensure timely processing of data.

[0030] Optionally, the Kafka cluster to be derived can be selected by browsing the list of configured Kafka clusters from the interface or API of the preconfigured data integration platform, and the Kafka cluster on which the derivative operation is required can be selected according to a specific identifier (such as cluster name, description, etc.).

[0031] Furthermore, the embodiment of the present invention can customize the data processing flow according to specific business needs by configuring the derivative environment for the Kafka cluster. For example, the exported data can be filtered, converted, aggregated, etc. to meet the requirements of different target systems. This makes the flow of data between different environments more efficient and flexible, and reduces the workload of manual data processing. Among them, the derivative environment refers to a specific technical environment for performing derivative task-related operations. In practical applications, the derivative environment may include the following aspects: 1. Software environment, with data processing tools, including software tools for extracting, transforming and loading data, such as ETL (Extract, Transform, Load) tools, data integration platforms, etc. These tools can obtain data from various data sources (such as databases, file systems, network protocols, etc.), and clean, transform and integrate to meet the needs of derivative tasks. For example, in the financial industry, professional financial data processing software may be used to extract transaction data from different financial transaction systems and perform real-time analysis and processing. Data analysis software: including statistical analysis software, data mining tools, machine learning platforms, etc. These software can conduct in-depth analysis of the data obtained by the derivative task to discover patterns, trends and associations in the data and provide support for decision-making. For example, in the insurance industry, data analysis software can be used to conduct risk assessment on customer data to determine insurance premiums and claims strategies. Second, the hardware environment, including servers and storage devices: derivative tasks usually need to process a large amount of data, so high-performance servers and storage devices are required to support data storage, processing and analysis. The configuration of the server (such as CPU, memory, hard disk capacity, etc.) and the type of storage device (such as hard disk array, solid-state drive, cloud storage, etc.) will affect the execution efficiency and data storage capacity of the derivative task. For example, in large financial institutions, high-performance server clusters and large-capacity storage devices may be required to process and store massive amounts of financial transaction data. Network equipment: derivative tasks may involve data transmission and sharing, so stable and high-speed network equipment is required to ensure smooth data transmission. The bandwidth, delay and reliability of network equipment will affect the execution speed of derivative tasks and the real-time nature of data. For example, in real-time trading systems, high-speed network equipment is required to ensure real-time transmission and processing of transaction data to avoid transaction delays and risks.

[0032] As an embodiment of the present invention, configuring a derivative environment for the Kafka cluster includes: determining the derivative requirements of the Kafka cluster, evaluating the derivative requirement resources of the Kafka cluster based on the derivative requirements, configuring the derivative tool of the Kafka cluster based on the derivative requirement resources, and configuring the derivative environment of the Kafka cluster based on the derivative tool.

[0033] Optionally, the derivative requirements of the Kafka cluster can be determined through business scenario analysis. Suppose in an e-commerce company, the business department hopes to extract data on specific topics (such as order topics, user behavior topics) from the Kafka cluster in the production environment for data analysis and machine learning model training. This clarifies the business need to export data from the Kafka cluster; data type and format requirements, determine the type of data to be exported, such as structured data (such as order data in JSON format) or unstructured data (such as text data of user comments), and clarify the format requirements of the data, such as whether specific format conversion is required so that it can be correctly processed in the target system. For example, converting the original data format in Kafka into a format acceptable to the target database; time range and data volume, determining the time range of the exported data, whether to export data from the most recent week, a month, or a specific time period; the derivative demand resources of the Kafka cluster evaluated based on the derivative demand can be evaluated through computing resource evaluation: if the derivative task involves a large amount of data processing and conversion, it is necessary to evaluate the required CPU and memory resources, storage resource evaluation: based on the amount of exported data and storage time, evaluate the required storage resources, network resource evaluation: if the derivative task involves transferring data to a remote location (such as from a production environment to a test environment or between data centers), it is necessary to evaluate whether the network bandwidth can meet the data transmission requirements; the derivative tool for configuring the Kafka cluster based on the derivative demand resources can be achieved by using Kafka Connect.

[0034] Furthermore, the embodiment of the present invention logically classifies the Kafka cluster to obtain a logically classified Kafka cluster, which can improve data management efficiency and meet different business requirements.

[0035] Exemplarily, the logically classifying the Kafka cluster to obtain the logically classified Kafka cluster includes: querying the classification goals and classification requirements of the Kafka cluster, performing data feature analysis on the Kafka cluster based on the classification goals and the classification requirements to obtain data analysis results, constructing a logical classification scheme for the Kafka cluster based on the data analysis results, and logically classifying the Kafka cluster based on the logical classification scheme to obtain the logically classified Kafka cluster.

[0036] Optionally, the querying of the classification target and classification requirement of the Kafka cluster can be obtained by identifying user needs, the data feature analysis of the Kafka cluster based on the classification target and the classification requirement, and the data analysis result can be achieved by analyzing using a deep learning network, and the logical classification scheme of the Kafka cluster based on the data analysis result can be constructed by determining the classification dimension according to the data analysis result, such as, based on data analysis, determining the data subject as the main classification dimension, and dividing the data in the Kafka cluster into categories such as order data, user behavior data, and inventory data, and then defining the classification hierarchy of the Kafka cluster, such as, for user behavior data, it can be divided into subcategories such as browsing behavior, purchasing behavior, and comment behavior. Each subcategory can also be classified more finely according to specific behavioral characteristics, and finally classification rules are formulated, such as, in order data, orders with a status of "paid" are classified as paid orders, and orders with a status of "unpaid" are classified as unpaid orders.

[0037] S2. Evaluate the rationality of the derivative task to obtain an evaluation result. When the evaluation result meets the preset result, query the execution resource amount of the task execution system corresponding to the derivative task. When the execution resource amount meets the demand for executing the derivative task, use the task execution system to execute the derivative task.

[0038] The embodiment of the present invention performs a rationality evaluation on the derivative task to obtain an evaluation result that can ensure data quality and accuracy, optimize resource utilization, and control costs to improve system stability and reliability.

[0039] As an embodiment of the present invention, the rationality evaluation of the derivative task to obtain the evaluation result includes: constructing a business requirement compliance standard, a data quality requirement standard, a resource feasibility standard and a time feasibility standard for the derivative task, determining an evaluation index system for the derivative task based on the business requirement compliance standard, the data quality requirement standard, the resource feasibility standard and the time feasibility standard, collecting evaluation data of the derivative task based on the evaluation index system, and evaluating the rationality of the derivative task based on the evaluation data to obtain an evaluation result.

[0040] Among them, the business demand compliance standard refers to the degree of fit between the guiding data task and the actual business demand, the data quality requirement standard refers to the indicator for measuring the quality level of the derived data, the resource feasibility standard refers to the indicator for evaluating the feasibility of the derivative task in terms of resources, and the time feasibility standard refers to the indicator for measuring the feasibility of the derivative task in terms of time.

[0041] Optionally, the business requirement compliance standard, data quality requirement standard, resource feasibility standard and time feasibility standard for constructing the derivative task can be achieved by determining business goals, analyzing business process definition compliance indicators, formulating data quality requirement standards through completeness, accuracy and consistency, defining resource feasibility standards through calculation resources, network resources and storage resources, and through task execution time, such as estimating the execution time of the derivative task, including the time required for data extraction, processing and transmission, considering the time requirements of the business, and ensuring that the task can be completed within the specified time. For example, for real-time data analysis projects, the derivative task may need to be completed within a few seconds; while for offline analysis projects, the time requirements may be relatively loose to formulate the time feasibility standard. The evaluation indicator system can be formulated by assigning corresponding indicator weights to each indicator. The evaluation data can be achieved by regularly extracting data indicators from the system using data collection tools, writing scripts to analyze task execution time and resource usage from log files, and collecting users' evaluations on business requirement compliance through questionnaires. The rationality of the derivative task based on the evaluation data is evaluated, and the evaluation result can be obtained by calculating the value of each evaluation indicator based on the collected evaluation data.

[0042] It should be noted that when the evaluation result meets the preset result, it means that the calculation index value corresponding to the evaluation result meets the preset index value. The calculation index value can be calculated in the following way: let the total evaluation be S, and the four aspects (business demand compliance, data quality requirements, resource feasibility, and time feasibility) be represented by A, B, C, and D respectively. The importance of each aspect is a, b, c, and d (and a+b+c+d=1). The preset minimum score for each aspect is T A ,T B ,T C ,T D， For business requirement compliance A, if it fully meets the business requirements, it will get 100 points, if it partially meets the business requirements, it will get 50 points, and if it does not meet the business requirements, it will get 0 points. Data quality requirements B, resource feasibility C, and time feasibility D are also set in a simple three-level scoring manner.

[0043] Total evaluation score calculation:

[0044] S=aA+bB+cC+dD.

[0045] Determine whether the preset result is met:

[0046] If A ≥ T A , B ≥ T B , C ≥ T C , D ≥ T D And S reaches a certain total score threshold (for example, set a total score value T total ), the evaluation result meets the preset result.

[0047] For example, let a=0.3, b=0.3, c=0.2, d=0.2, partial compliance with business requirement A is 50 points, full compliance with data quality requirement B is 100 points, full compliance with resource feasibility C is 100 points, and partial compliance with time feasibility D is 50 points.

[0048] The total evaluation score is:

[0049] S=0.3×50+0.3×100+0.2×100+0.2×50=15+30+20+

[0050] 10=75

[0051] Assume that the preset business requirement compliance score is T A =40 points, data quality requires a minimum score of T B =80 points, the lowest score for resource feasibility T C =80 points, the lowest score for time feasibility is T D =40 points, total score threshold T total =70 points.

[0052] It should be noted that the preset indicator value needs to be set in combination with the actual application.

[0053] Furthermore, the embodiment of the present invention can understand the execution resource amount by querying the execution resource amount of the task execution system corresponding to the derivative task, and then determine whether the current system has sufficient resources to execute the derivative task to ensure the feasibility of the task.

[0054] Optionally, the execution resource amount may be queried through a system monitoring tool.

[0055] S3. During the execution of the derivative task, the topic message data of the Kafka cluster during production is read, the topic message data is desensitized, and the desensitized data is stored.

[0056] In the embodiment of the present invention, by reading the topic message data of the Kafka cluster in production during the execution of the derivative task, the latest and real-time data can be obtained to ensure the integrity of the data. This is crucial for scenarios that require timely reflection of business status or real-time analysis. Among them, the topic message data refers to the message data under a specific topic in the Kafka message system.

[0057] The embodiment of the present invention desensitizes the subject message data to obtain desensitized data, which can reduce data security risks and reduce the need for high-cost security measures.

[0058] As an embodiment of the present invention, the data desensitizing the subject message data and storing the desensitized data after the data desensitization includes: analyzing the data sensitivity of the subject message data, constructing a desensitization strategy for the subject message data based on the data sensitivity, and configuring a desensitization tool for the subject message data; based on the desensitization strategy and the desensitization tool, desensitizing the subject message data to obtain desensitized data, and storing the desensitized data after the data desensitization.

[0059] Optionally, the analysis of the data sensitivity of the topic message data can be achieved by clarifying which types of data may be regarded as sensitive data in the Topic and evaluating the data sensitivity level. The desensitization strategy for the topic message data based on the data sensitivity can be constructed by selecting an appropriate desensitization method according to the data sensitivity level and usage scenario. The desensitization tool can be replaced, encrypted, obfuscated, truncation and other desensitization tools.

[0060] The desensitized data may be stored by utilizing a storage tool preconfigured by the data integration platform.

[0061] S4. The desensitized data is transmitted to a data testing environment to obtain test data, and a data integrity test is performed on the test data to obtain a test result. When the test result is complete, the derivative task is completed.

[0062] The embodiment of the present invention obtains test data by transmitting the desensitized data to a data testing environment. The desensitized data can be used to simulate real business loads to evaluate the performance of the software system. For example, indicators such as response time, throughput, and resource utilization when processing large amounts of data can be measured. This helps to determine whether the system can meet the performance requirements of actual business and whether performance optimization is needed.

[0063] As an embodiment of the present invention, the desensitized data is transmitted to a data testing environment to obtain test data, including: analyzing the test requirements of the data testing environment corresponding to the desensitized data, performing data preprocessing on the desensitized data based on the test requirements to obtain preprocessed data, configuring transmission parameters of the preprocessed data, and performing a connection test on the transmission connection of the preprocessed data. When the connection test is normal, the desensitized data is transmitted to the data testing environment using the transmission parameters to obtain test data.

[0064] Optionally, the analysis of the test requirements of the data testing environment corresponding to the desensitized data can be achieved by determining the test objectives. For example, in an e-commerce application, the test objectives may include functional testing (such as shopping cart functions, order processing, etc.), performance testing (such as response time, throughput), security testing (such as user data protection, prevention of hacker attacks), etc., and then identifying the test scenarios. For example, in e-commerce applications, there may be normal shopping processes, high-traffic scenarios during promotional activities, return and refund processes, etc., and analyzing which desensitized data is needed in each scenario to simulate the real situation. Finally, according to the resources and performance requirements of the test environment, the amount of data that needs to be transmitted is determined to achieve the data preprocessing of the desensitized data to obtain the pre-processed data. The data processing can be achieved by cleaning and formatting the desensitized data. The transmission parameters can be obtained by determining the data transmission method, such as file transfer, database replication, network transmission, etc., and configuring the corresponding parameters. The connection test can be performed by using a test tool or writing a script to simulate the data transmission process and check whether the connection is successfully tested. The desensitized data is transmitted to the data test environment using the transmission parameters. The corresponding tool or command can be selected by the transmission method to execute the data transmission. For example, for file transfer, FTP client software or command line tools can be used to start transferring files. For database replication, database management tools or scripts can be used to start the replication process.

[0065] Furthermore, the embodiments of the present invention perform data integrity testing on the test data to obtain test results that can reveal data errors, verify processing logic, improve system quality and stability, discover vulnerabilities, ensure system integration, and provide an accurate data basis for decision-making, optimize business processes, ensure data accuracy and reliability, and support efficient enterprise operations.

[0066] As an embodiment of the present invention, performing a data integrity test on the test data to obtain a test result includes: querying a test range of the test data, determining a test data source of the test data based on the test range, performing data sampling on the test data source to obtain sampled test data, and performing a data integrity test on the sampled test data to obtain a test result.

[0067] The test scope refers to the specific fields, data types or business process parts covered when performing data consistency testing on test data.

[0068] Optionally, the test scope of querying the test data can be determined by clarifying the test objectives. For example, in the test of an e-commerce platform, the test objective may be to verify the consistency of user order data between different systems. Then the test scope may include order generation system, inventory management system, payment system and other order-related links. For a financial transaction system, if the test objective is to ensure the accuracy and consistency of transaction data, the test scope may cover transaction matching system, settlement system, account management system, etc., and determine the data type implementation (determine the data type to be tested according to the test objective, for example, in an e-commerce platform, it may be order number, product information, customer information, payment amount and other data; in a financial transaction system, it may be transaction serial number, transaction type, transaction amount, transaction time and other data). The process of determining the test data source of the test data based on the test scope is: determine the data source that may contain relevant test data according to the test scope. For example, in an e-commerce platform, the database of the order generation system, the database of the inventory management system and the log files of the payment system may all be test data sources. In a financial transaction system, transaction matching The system database, the report files of the settlement system, the database of the account management system, etc. are all potential test data sources. The determined data source is evaluated to determine the reliability and accuracy of its data. This can be done by checking the update frequency of the data source, data quality monitoring records, accuracy of historical data, etc. For example, if a data source often has data errors or is not updated in a timely manner, it may need to be used with caution or additional verification measures may be taken during testing. The data sampling of the test data source to obtain the sampled test data may be performed through random sampling, stratified sampling, and other methods. For example, in an e-commerce platform, if you want to test the consistency of order data of different types of goods, you can use stratified sampling and sample according to the product category; if you are only conducting general testing, you can use random sampling. In financial transaction systems, stratified sampling can be performed based on factors such as transaction time and transaction type to ensure that the sampled data can represent different transaction situations.

[0069] Furthermore, the embodiment of the present invention can use the accurate data obtained by the derivative task to conduct in-depth analysis when the test result is complete, and provide a basis for enterprises to make strategic decisions. For example, in marketing, by analyzing customer behavior data, target customer groups can be determined, marketing strategies can be optimized, and market share can be increased.

[0070] It should be noted that when the test result is complete, it means that the data of the Kafka cluster to be derived is consistent during production and testing.

[0071] It should be understood that the size of the serial numbers of the steps in the above embodiment does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention. At the same time, it can improve the accuracy of real-time data testing to meet user needs.

[0072] In one embodiment, a real-time data testing device is provided, which corresponds one-to-one to the real-time data testing method in the above embodiment. Figure 3 As shown, the real-time data testing device includes a derivative task creation module 201, a derivative task execution module 202, a data storage module 203 and a data testing module 204. The detailed description of each functional module is as follows:

[0073] The derivative task creation module 201 is used to select a Kafka cluster to be derived from a preconfigured data integration platform, configure a derivative environment for the Kafka cluster, and logically classify the Kafka cluster to obtain a logically classified Kafka cluster, and create a derivative task for the logically classified Kafka cluster based on the derivative environment;

[0074] The derivative task execution module 202 is used to evaluate the rationality of the derivative task and obtain an evaluation result. When the evaluation result meets the preset result, query the execution resource amount of the task execution system corresponding to the derivative task. When the execution resource amount meets the demand for executing the derivative task, use the task execution system to execute the derivative task.

[0075] The data storage module 203 is used to read the topic message data of the Kafka cluster during production when the derivative task is executed, desensitize the topic message data, and store the desensitized data after desensitization;

[0076] The data testing module 204 is used to transmit the desensitized data to a data testing environment to obtain test data, perform a data integrity test on the test data to obtain a test result, and when the test result is complete, the derivative task is completed.

[0077] In one embodiment, when executing the configuration of the derivative environment for the Kafka cluster, the derivative task creation module 201 is used to: determine the derivative requirements of the Kafka cluster, evaluate the derivative requirement resources of the Kafka cluster based on the derivative requirements, configure the derivative tool of the Kafka cluster based on the derivative requirement resources, and configure the derivative environment of the Kafka cluster based on the derivative tool.

[0078] In one embodiment, when the derivative task creation module 201 performs logical classification on the Kafka cluster to obtain a logically classified Kafka cluster, it is used to: query the classification target and classification requirement of the Kafka cluster, perform data feature analysis on the Kafka cluster based on the classification target and the classification requirement to obtain data analysis results, construct a logical classification scheme for the Kafka cluster based on the data analysis results, and logically classify the Kafka cluster based on the logical classification scheme to obtain a logically classified Kafka cluster.

[0079] In one embodiment, when the derivative task execution module 202 performs a rationality assessment on the derivative task and obtains an assessment result, it is used to: construct a business requirement compliance standard, a data quality requirement standard, a resource feasibility standard and a time feasibility standard for the derivative task, determine an evaluation index system for the derivative task based on the business requirement compliance standard, the data quality requirement standard, the resource feasibility standard and the time feasibility standard, collect evaluation data of the derivative task based on the evaluation index system, and perform a rationality assessment on the derivative task based on the evaluation data to obtain an evaluation result.

[0080] In one embodiment, when the data storage module 203 performs data desensitization on the subject message data and stores the desensitized data after data desensitization, it is used to: analyze the data sensitivity of the subject message data, construct a desensitization strategy for the subject message data based on the data sensitivity, and configure a desensitization tool for the subject message data; based on the desensitization strategy and the desensitization tool, perform data desensitization on the subject message data, and store the desensitized data after data desensitization.

[0081] In one embodiment, when the data testing module 204 transmits the desensitized data to the data testing environment to obtain test data, it is used to: analyze the test requirements of the data testing environment corresponding to the desensitized data, perform data preprocessing on the desensitized data based on the test requirements to obtain preprocessed data, configure transmission parameters of the preprocessed data, and perform a connection test on the transmission connection of the preprocessed data; when the connection test is normal, use the transmission parameters to transmit the desensitized data to the data testing environment to obtain test data.

[0082] In one embodiment, when performing a data integrity test on the test data to obtain a test result, the data testing module 204 is used to: query the test scope of the test data, determine the test data source of the test data based on the test scope, perform data sampling on the test data source to obtain sampled test data, perform data integrity test on the sampled test data to obtain a test result.

[0083] The specific definition of the real-time data testing device can be found in the definition of the real-time data testing method above, which will not be repeated here. Each module in the above-mentioned real-time data testing device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0084] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a real-time data testing method service side.

[0085] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, it realizes the functions or steps of the client side of a real-time data testing method

[0086] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0087] Selecting a Kafka cluster to be derived from a preconfigured data integration platform, configuring a derivative environment for the Kafka cluster, and after logically classifying the Kafka cluster to obtain the logically classified Kafka cluster, creating a derivative task for the Kafka cluster based on the derivative environment;

[0088] Performing a rationality evaluation on the derivative task to obtain an evaluation result, and when the evaluation result meets a preset result, querying the execution resource amount of the task execution system corresponding to the derivative task, and when the execution resource amount meets the demand for executing the derivative task, using the task execution system to start executing the derivative task;

[0089] During the execution of the derivative task, the topic message data of the Kafka cluster during production is read, the topic message data is desensitized, and the desensitized data is stored;

[0090] The desensitized data is transmitted to a data testing environment to obtain test data, and a data integrity test is performed on the test data to obtain a test result. When the test result is complete, the derivative task is completed.

[0091] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0092] Selecting a Kafka cluster to be derived from a preconfigured data integration platform, configuring a derivative environment for the Kafka cluster, and after logically classifying the Kafka cluster to obtain the logically classified Kafka cluster, creating a derivative task for the Kafka cluster based on the derivative environment;

[0093] Performing a rationality evaluation on the derivative task to obtain an evaluation result, and when the evaluation result meets a preset result, querying the execution resource amount of the task execution system corresponding to the derivative task, and when the execution resource amount meets the demand for executing the derivative task, using the task execution system to start executing the derivative task;

[0094] During the execution of the derivative task, the topic message data of the Kafka cluster during production is read, the topic message data is desensitized, and the desensitized data is stored;

[0095] The desensitized data is transmitted to a data testing environment to obtain test data, a data integrity test is performed on the test data to obtain a test result, and when the test result is complete, the derivative task is completed.

[0096] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0097] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0098] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0099] The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some of the technical features therein by equivalents; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should be included in the protection scope of the present invention. It should be noted that if software tools or components other than those of the company appear in the embodiments of this application, they are only used for illustration and do not represent actual use.

Claims

1. A real-time data testing method, characterized in that: include: Selecting a Kafka cluster to be derived from a preconfigured data integration platform, configuring a derivative environment for the Kafka cluster, and logically classifying the Kafka cluster to obtain a logically classified Kafka cluster, and creating a derivative task for the logically classified Kafka cluster based on the derivative environment; Performing a rationality evaluation on the derivative task to obtain an evaluation result, and when the evaluation result meets a preset result, querying the execution resource amount of the task execution system corresponding to the derivative task, and when the execution resource amount meets the demand for executing the derivative task, using the task execution system to execute the derivative task; When the derivative task is executed, the topic message data of the Kafka cluster in production is read, the topic message data is desensitized, and the desensitized data after the data desensitization is stored; The desensitized data is transmitted to a data testing environment to obtain test data, a data integrity test is performed on the test data to obtain a test result, and when the test result is complete, the derivative task is completed.

2. The real-time data testing method according to claim 1, characterized in that: Configuring a derivative environment for the Kafka cluster includes: Determining the derivative requirements of the Kafka cluster; Based on the derivative demand, evaluating the derivative demand resources of the Kafka cluster; A derivative tool for configuring the Kafka cluster based on the derivative demand resources; Based on the derivative tool, configure the derivative environment of the Kafka cluster.

3. The real-time data testing method according to claim 1, characterized in that: The logically classifying the Kafka cluster to obtain the logically classified Kafka cluster includes: Query the classification target and classification requirements of the Kafka cluster; Based on the classification target and the classification requirement, perform data feature analysis on the Kafka cluster to obtain a data analysis result; Based on the data analysis results, construct a logical classification scheme for the Kafka cluster; Based on the logical classification scheme, the Kafka cluster is logically classified to obtain a logically classified Kafka cluster.

4. The real-time data testing method according to claim 1, characterized in that: The performing of a rationality evaluation on the derivative task to obtain an evaluation result includes: Establishing the business requirement compliance standard, data quality requirement standard, resource feasibility standard and time feasibility standard of the derivative task; Determining an evaluation index system for the derivative task based on the business requirement compliance standard, the data quality requirement standard, the resource feasibility standard, and the time feasibility standard; Based on the evaluation indicator system, collecting evaluation data of the derivative task; The rationality of the derivative task is evaluated based on the evaluation data to obtain an evaluation result.

5. The real-time data testing method according to claim 1, characterized in that: The desensitizing the subject message data and storing the desensitized data after the desensitization includes: Analyzing data sensitivity of the subject message data; Based on the data sensitivity, construct a desensitization strategy for the subject message data, and configure a desensitization tool for the subject message data; Based on the desensitization strategy and the desensitization tool, the subject message data is desensitized to obtain desensitized data, and the desensitized data is stored.

6. The real-time data testing method according to claim 1, characterized in that: The transmitting the desensitized data to the data testing environment to obtain the test data includes: Analyze the test requirements of the data testing environment corresponding to the desensitized data; Based on the test requirements, preprocess the desensitized data to obtain preprocessed data; configuring transmission parameters of the preprocessed data, and performing a connection test on a transmission connection of the preprocessed data; When the connection test is normal, the desensitized data is transmitted to the data test environment using the transmission parameters to obtain test data.

7. The real-time data testing method according to claim 1, characterized in that: The performing of a data integrity test on the test data to obtain a test result includes: Query the test scope of the test data; Based on the test data source, data sampling is performed on the test data source to obtain sampled test data; Perform a data integrity test on the sampled test data to obtain a test result.

8. A real-time data testing device, characterized in that: include: A derivative task creation module is used to select a Kafka cluster to be derived from a preconfigured data integration platform, configure a derivative environment for the Kafka cluster, and logically classify the Kafka cluster to obtain a logically classified Kafka cluster, and create a derivative task for the logically classified Kafka cluster based on the derivative environment; A derivative task execution module, used to evaluate the rationality of the derivative task to obtain an evaluation result, and when the evaluation result meets the preset result, query the execution resource amount of the task execution system corresponding to the derivative task, and when the execution resource amount meets the demand for executing the derivative task, use the task execution system to execute the derivative task; A data storage module, used for reading the topic message data of the Kafka cluster during production when the derivative task is executed, desensitizing the topic message data, and storing the desensitized data after desensitization; The data testing module is used to transfer the desensitized data to the data testing environment, obtain test data, perform data integrity test on the test data, obtain test results, and complete the derivative task when the test results are complete.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the real-time data testing method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the real-time data testing method according to any one of claims 1 to 7 are implemented.

Images