A real-time data testing method, device, computer equipment and storage medium

Abstract

The present application relates to the field of data testing and the field of financial technology, and discloses a real-time data testing method, device, equipment and medium, comprising: configuring a derivative environment for a Kafka cluster, creating a derivative task of the Kafka cluster based on the derivative environment after logically classifying the Kafka cluster; reasonably evaluating the derivative task, querying an execution resource amount of a task execution system corresponding to the derivative task when the evaluation result meets a preset result, and starting to execute the derivative task when the execution resource amount meets a demand amount of the derivative task; in the derivative task execution, reading topic message data of the Kafka cluster, performing data desensitization, data storage on the topic message data, obtaining desensitized data, transmitting the desensitized data to a data testing environment, obtaining testing data, performing data integrity testing on the testing data, obtaining a testing result, and completing the derivative task when the testing result is complete. The present application can improve the accuracy of real-time data testing.

Description

Technical Field

[0001] This invention relates to the fields of data testing and financial technology, and particularly to a real-time data testing method, apparatus, computer equipment, and storage medium. Background Technology

[0002] With the rapid development of big data and artificial intelligence technologies, the amount of data in various industries is growing rapidly over time. Research has found that the amount of data has roughly doubled every three years in recent years. For example, the financial and insurance industries have accumulated a large amount of customer and related business information as their businesses develop. If this data can be fully utilized, it will provide a significant advantage for the company's development, giving rise to real-time data application scenarios.

[0003] Currently, real-time scenarios in the industry primarily revolve around building enterprise ecosystems around the streaming capabilities of Kafka. Traditional real-time data processing development needs, such as real-time metric development, require large amounts of diverse data. This typically necessitates data generation in a test environment. However, if the data generated in the test environment differs significantly from the production environment, the accuracy of the metric development cannot be guaranteed. Furthermore, manual data generation in the test environment often faces challenges due to numerous upstream and downstream data stakeholders, long data chains, and involvement across multiple teams and systems, making data generation extremely difficult, requiring significant manpower, and resulting in low data accuracy. Summary of the Invention

[0004] This invention provides a real-time data testing method, apparatus, computer equipment, and storage medium to improve the accuracy of real-time data testing.

[0005] Firstly, a real-time data testing method is provided, including:

[0006] Select the Kafka cluster to be derived from the pre-configured data integration platform, configure the derivation environment for the Kafka cluster, and logically classify the Kafka cluster to obtain logically classified Kafka clusters. Based on the derivation environment, create the derivation task for the logically classified Kafka cluster.

[0007] The rationality of the derivative task is evaluated to obtain the evaluation result. When the evaluation result meets the preset result, the execution resource quantity of the task execution system corresponding to the derivative task is queried. When the execution resource quantity meets the requirement for executing the derivative task, the derivative task is executed using the task execution system.

[0008] When the derivative task is executed, the topic message data of the Kafka cluster during production is read, the topic message data is anonymized, and the anonymized data is stored.

[0009] The de-identified data is transmitted to a data testing environment to obtain test data. Data integrity testing is performed on the test data to obtain test results. If the test results are complete, the derivative task is completed.

[0010] Secondly, a real-time data testing device is provided, comprising:

[0011] The derivative task creation module is used to select the Kafka cluster to be derived from the pre-configured data integration platform, configure the derivative environment for the Kafka cluster, perform logical classification on the Kafka cluster to obtain logically classified Kafka clusters, and create derivative tasks for the logically classified Kafka clusters based on the derivative environment.

[0012] The derivative task execution module is used to evaluate the rationality of the derivative task and obtain the evaluation result. When the evaluation result meets the preset result, it queries the execution resource quantity of the task execution system corresponding to the derivative task. When the execution resource quantity meets the requirement for executing the derivative task, it uses the task execution system to execute the derivative task.

[0013] The data storage module is used to read the topic message data of the Kafka cluster during production when the derivative task is executed, perform data anonymization on the topic message data, and store the anonymized data.

[0014] The data testing module is used to transmit the de-identified data to the data testing environment to obtain test data, perform data integrity testing on the test data, obtain test results, and complete the derivative task when the test results are complete.

[0015] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described objection discourse mining method.

[0016] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described real-time data testing method.

[0017] Compared to the problems described in the background art, this invention allows users to easily and quickly find and select the Kafka cluster to be derived from numerous data sources by selecting it from a pre-configured data integration platform. This avoids the cumbersome process of switching between different systems and tools, thus improving operational efficiency. The pre-configured data integration platform is a software system specifically designed to integrate different data sources and realize data flow and processing. It provides a unified interface and management environment, facilitating centralized management of various data-related tasks. Different data integration processes, including derivative operations on the Kafka cluster, can be viewed, configured, and monitored on this platform. The Kafka cluster is a distributed messaging system capable of handling large volumes of messages, meeting the needs of large-scale data processing. In applications with high real-time requirements, such as financial transactions and IoT data processing, the Kafka cluster can quickly receive, store, and transmit messages, ensuring timely data processing. Configuring a derivative environment for the Kafka cluster allows for customized data processing workflows based on specific business needs. For example, exported data can be filtered, transformed, and aggregated to meet the requirements of different target systems. This makes data flow between different environments more efficient and flexible, reduces the workload of manual data processing, and improves data management efficiency and meets different business needs by creating derivative tasks based on the derivative environment after logically classifying the Kafka cluster into logically categorized Kafka clusters. The feasibility evaluation of the derivative tasks ensures data quality and accuracy, optimizes resource utilization, controls costs, improves system stability and reliability, and allows for querying the execution resource volume of the system corresponding to the derivative task, thus determining whether the current system has sufficient resources to execute the derivative task and ensuring its feasibility. During the execution of the derivative task, reading the topic message data of the Kafka cluster in production provides the latest, real-time data, ensuring data integrity. This is crucial for scenarios that require timely reflection of business status or real-time analysis.The topic message data refers to message data under a specific topic in the Kafka messaging system. By storing the anonymized data, data security is ensured. The anonymized data is then transmitted to a data testing environment to obtain test data. This anonymized data can simulate real business loads to evaluate the performance of the software system. For example, it can measure metrics such as response time, throughput, and resource utilization when processing large amounts of data. This helps determine whether the system can meet the performance requirements of actual business operations and whether performance optimization is needed. Data integrity testing is performed on the test data. The test results can reveal data errors, verify processing logic, improve system quality and stability, discover vulnerabilities, ensure system integration, provide an accurate data foundation for decision-making, optimize business processes, ensure data accuracy and reliability, and support efficient enterprise operations. This invention can improve the accuracy of real-time data testing. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of an application environment for a real-time data testing method according to an embodiment of the present invention;

[0020] Figure 2 This is a flowchart illustrating a real-time data testing method according to an embodiment of the present invention;

[0021] Figure 3 This is a schematic diagram of a real-time data testing device according to an embodiment of the present invention;

[0022] Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0023] Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] The real-time data testing method provided in this embodiment of the invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. The server can select a Kafka cluster to be derived from a pre-configured data integration platform, configure a derivation environment for the Kafka cluster, logically classify the Kafka cluster to obtain logically classified Kafka clusters, and create a derivation task for the Kafka cluster based on the derivation environment. The derivation task is evaluated for reasonableness, and when the evaluation result meets a preset requirement, the execution resource quantity of the corresponding task execution system is queried. When the execution resource quantity meets the requirements for executing the derivation task, the task execution system is used to start executing the derivation task. During the execution of the derivation task, the topic message data of the Kafka cluster during production is read, the topic message data is anonymized, and the anonymized data is stored. The anonymized data is transmitted to a data testing environment to obtain test data, and the test data is subjected to data integrity testing to obtain test results. When the test results are complete, the derivation task is completed.

[0026] The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will now be described in detail through specific embodiments.

[0027] Please see Figure 2 As shown, Figure 2 A flowchart illustrating a real-time data testing method provided in an embodiment of the present invention includes the following steps:

[0028] S1. Select the Kafka cluster to be derived from the pre-configured data integration platform, configure the derivation environment for the Kafka cluster, and logically classify the Kafka cluster to obtain logically classified Kafka clusters. Based on the derivation environment, create the derivation task for the logically classified Kafka cluster.

[0029] This invention, through the selection of the Kafka cluster to be differentiated from a pre-configured data integration platform, allows users to easily and quickly find and select the Kafka cluster from numerous data sources, avoiding the cumbersome process of switching between different systems and tools, thus improving operational efficiency. The pre-configured data integration platform refers to a software system specifically designed to integrate different data sources and realize data flow and processing. It provides a unified interface and management environment, facilitating centralized management of various data-related tasks. Different data integration processes, including derivative operations on the Kafka cluster, can be viewed, configured, and monitored on this platform. The Kafka cluster is a distributed messaging system capable of handling large volumes of messages, meeting the needs of large-scale data processing. In applications with high real-time requirements, such as financial transactions and IoT data processing, Kafka clusters can quickly receive, store, and transmit messages, ensuring timely data processing.

[0030] Optionally, the Kafka cluster to be derived can be selected by browsing the configured Kafka cluster list through the interface or API of the pre-configured data integration platform, and selecting the Kafka cluster to be derived based on specific identifiers (such as cluster name, description, etc.).

[0031] Furthermore, in this embodiment of the invention, configuring the data processing environment for the Kafka cluster allows for customization of data processing workflows based on specific business needs. For example, exported data can be filtered, transformed, and aggregated to meet the requirements of different target systems. This makes data flow between different environments more efficient and flexible, reducing the workload of manual data processing. The data processing environment refers to the specific technical environment for performing data processing tasks. In practical applications, the data processing environment may include the following aspects: 1. Software environment, including data processing tools, such as ETL (Extract, Transform, Load) tools and data integration platforms. These tools can acquire data from various data sources (such as databases, file systems, network protocols, etc.) and perform cleaning, transformation, and integration to meet the needs of data processing tasks. For example, in the financial industry, specialized financial data processing software may be used to extract transaction data from different financial trading systems and perform real-time analysis and processing. Data analysis software includes statistical analysis software, data mining tools, and machine learning platforms. These software programs can perform in-depth analysis of the data obtained from derivative tasks to discover patterns, trends, and correlations within the data, providing support for decision-making. For example, in the insurance industry, data analysis software can be used to assess customer data for risk, determine insurance rates, and establish claims strategies. Secondly, the hardware environment is crucial, including servers and storage devices. Derivative tasks typically process large amounts of data, thus requiring high-performance servers and storage devices to support data storage, processing, and analysis. The server configuration (such as CPU, memory, and hard drive capacity) and the type of storage device (such as hard drive arrays, solid-state drives, and cloud storage) affect the execution efficiency and data storage capacity of derivative tasks. For example, large financial institutions may need to use high-performance server clusters and large-capacity storage devices to process and store massive amounts of financial transaction data. Finally, network equipment is essential. Derivative tasks may involve data transmission and sharing, thus requiring stable, high-speed network equipment to ensure smooth data transmission. The bandwidth, latency, and reliability of network equipment affect the execution speed and real-time performance of derivative tasks. For example, in real-time trading systems, high-speed network equipment is needed to ensure the real-time transmission and processing of transaction data to avoid transaction delays and risks.

[0032] As an embodiment of the present invention, configuring the derivative environment for the Kafka cluster includes: determining the derivative requirements of the Kafka cluster, evaluating the derivative resource requirements of the Kafka cluster based on the derivative requirements, configuring the derivative tools of the Kafka cluster based on the derivative resource requirements, and configuring the derivative environment of the Kafka cluster based on the derivative tools.

[0033] Optionally, the determination of the data export requirements of the Kafka cluster can be achieved through business scenario analysis. For example, in an e-commerce company, the business department may want to extract data on specific topics (such as order topics or user behavior topics) from the Kafka cluster in the production environment for data analysis and training of machine learning models. This clarifies the business requirement to export data from the Kafka cluster. Data type and format requirements are also considered. This involves determining the data type to be exported, such as structured data (e.g., JSON-formatted order data) or unstructured data (e.g., text data of user reviews), and clarifying the data format requirements, such as whether specific format conversions are needed to ensure correct processing in the target system. For example, the raw data format in Kafka is converted to a format acceptable to the target database; the time range and data volume determine the time range for exporting data—whether it's exporting data from the most recent week, a month, or a specific time period; the resource assessment of the Kafka cluster's derivative requirements based on these requirements can be done through computational resource assessment: if the derivative task involves a large amount of data processing and transformation, the required CPU and memory resources need to be assessed; storage resource assessment: based on the amount of exported data and storage time, the required storage resources need to be assessed; network resource assessment: if the derivative task involves data transfer to remote locations (such as transfer from a production environment to a test environment or between data centers), it needs to be assessed whether the network bandwidth can meet the data transfer requirements; the derivative tool for configuring the Kafka cluster based on these resource requirements can be implemented using Kafka Connect.

[0034] Furthermore, by performing logical classification on the Kafka cluster, the embodiments of the present invention obtain logically classified Kafka clusters, which can improve data management efficiency and meet different business needs.

[0035] For example, the step of logically classifying the Kafka cluster to obtain a logically classified Kafka cluster includes: querying the classification target and classification requirements of the Kafka cluster; performing data feature analysis on the Kafka cluster based on the classification target and classification requirements to obtain data analysis results; constructing a logical classification scheme for the Kafka cluster based on the data analysis results; and performing logical classification on the Kafka cluster based on the logical classification scheme to obtain a logically classified Kafka cluster.

[0036] Optionally, the classification target and classification requirements of the Kafka cluster can be obtained by identifying user needs. The data feature analysis of the Kafka cluster based on the classification target and classification requirements, and the resulting data analysis, can be achieved using a deep learning network. The logical classification scheme for the Kafka cluster based on the data analysis results can be constructed by determining the classification dimensions according to the data analysis results. For example, based on data analysis, the main classification dimension can be determined, and the data in the Kafka cluster can be divided into categories such as order data, user behavior data, and inventory data. Then, the classification hierarchy of the Kafka cluster can be defined. For example, for user behavior data, it can be divided into subcategories such as browsing behavior, purchasing behavior, and commenting behavior. Each subcategory can be further subdivided according to specific behavioral characteristics. Finally, classification rules are formulated. For example, in order data, orders with a status of "paid" are classified as paid orders, and orders with a status of "unpaid" are classified as unpaid orders.

[0037] S2. Perform a rationality assessment on the derivative task to obtain an assessment result. When the assessment result meets the preset result, query the execution resource quantity of the task execution system corresponding to the derivative task. When the execution resource quantity meets the requirement for executing the derivative task, use the task execution system to execute the derivative task.

[0038] The embodiments of the present invention, by conducting a rationality assessment of the derivative task, obtain assessment results that can ensure data quality and accuracy, optimize resource utilization, control costs, and improve system stability and reliability.

[0039] As an embodiment of the present invention, the step of conducting a rationality assessment of the derivative task and obtaining an assessment result includes: constructing business requirement compliance standards, data quality requirement standards, resource feasibility standards, and time feasibility standards for the derivative task; determining an assessment index system for the derivative task based on the business requirement compliance standards, the data quality requirement standards, the resource feasibility standards, and the time feasibility standards; collecting assessment data for the derivative task based on the assessment index system; and conducting a rationality assessment of the derivative task based on the assessment data to obtain an assessment result.

[0040] Among them, the business requirement compliance standard refers to the degree of fit between the derivative task and the actual business requirements; the data quality requirement standard refers to the indicator that measures the quality level of the exported data; the resource feasibility standard refers to the indicator that assesses the feasibility of the derivative task in terms of resources; and the time feasibility standard refers to the indicator that measures the feasibility of the derivative task in terms of time.

[0041] Optionally, the business requirement compliance standards, data quality requirement standards, resource feasibility standards, and time feasibility standards for constructing the derivative task can be defined by determining business objectives, analyzing business processes to define compliance indicators, formulating data quality requirement standards based on completeness, accuracy, and consistency, defining resource feasibility standards based on computing resources, network resources, and storage resources, and ensuring that the task can be completed within a specified time by considering the task execution time, such as estimating the execution time of the derivative task, including the time required for data extraction, processing, and transmission, and taking into account the business's time requirements. For example, for real-time data analysis projects, the derivative task may need to be completed within a few seconds; while for offline analysis projects, the time requirements may be relatively lenient, thus establishing time feasibility standards. The evaluation indicator system can be established by assigning corresponding indicator weights to each indicator. The evaluation data can be obtained by periodically extracting data indicators from the system using data acquisition tools, writing scripts to analyze task execution time and resource usage from log files, and collecting user evaluations of business requirement compliance through questionnaires. The rationality evaluation of the derivative task based on the evaluation data, and the evaluation result, can be obtained by calculating the value of each evaluation indicator based on the collected evaluation data.

[0042] It should be noted that when the evaluation result meets the preset result, it means that the calculated indicator value corresponding to the evaluation result meets the preset indicator value. The calculated indicator value can be calculated in the following way: Let the total evaluation score be S, and the four aspects (business requirement compliance, data quality requirements, resource feasibility, and time feasibility) be represented by A, B, C, and D respectively. The importance ratio of each aspect is a, b, c, and d respectively (and a+b+c+d=1). The preset minimum score for each aspect is T. A ,T B ,T C ,T D， For business requirement compliance (A), 100 points are awarded for full compliance, 50 points for partial compliance, and 0 points for non-compliance. Data quality requirements (B), resource feasibility (C), and time feasibility (D) are similarly scored using a simple three-tier system.

[0043] Total assessment score calculation:

[0044] S = aA + bB + cC + dD.

[0045] Determine if the preset result is met:

[0046] If A≥T A B≥T B C≥T C D≥T D And S reaches a certain total score threshold (for example, setting a total score value T). total If the result meets the preset expectations, then the evaluation result will be satisfactory.

[0047] For example, if a = 0.3, b = 0.3, c = 0.2, and d = 0.2, then the business requirement compliance (A) is partially met and scores 50 points; the data quality requirement (B) is fully met and scores 100 points; the resource feasibility (C) is fully met and scores 100 points; and the time feasibility (D) is partially met and scores 50 points.

[0048] Total evaluation score:

[0049] S=0.3×50+0.3×100+0.2×100+0.2×50=15+30+20+

[0050] 10 = 75

[0051] Assuming the minimum score for meeting the pre-defined business requirements is T. A =40 points, minimum score for data quality requirement T B =80 points, minimum score for resource feasibility T C =80 points, minimum time feasibility score T D =40 points, total score threshold T total =70 points.

[0052] It should be noted that the preset index values ​​need to be set according to the actual application.

[0053] Furthermore, in this embodiment of the invention, by querying the execution resource quantity of the task execution system corresponding to the derivative task, the execution resource quantity can be understood, and then it can be determined whether the current system has sufficient resources to execute the derivative task, so as to ensure the feasibility of the task.

[0054] Optionally, the amount of execution resources can be queried through system monitoring tools.

[0055] S3. During the execution of the derivative task, read the topic message data of the Kafka cluster during production, perform data anonymization on the topic message data, and store the anonymized data.

[0056] This invention, through the process of reading the production-time topic message data of the Kafka cluster during the execution of the derivative task, can obtain the latest, real-time data, ensuring data integrity. This is crucial for scenarios requiring timely reflection of business status or real-time analysis. The topic message data refers to message data under a specific topic in the Kafka messaging system.

[0057] The embodiments of the present invention reduce data security risks and the need for costly security measures by performing data anonymization on the topic message data to obtain anonymized data.

[0058] As an embodiment of the present invention, the step of data anonymization of the topic message data and storage of the anonymized data includes: analyzing the data sensitivity of the topic message data, constructing an anonymization strategy for the topic message data based on the data sensitivity, configuring an anonymization tool for the topic message data, performing data anonymization on the topic message data based on the anonymization strategy and the anonymization tool to obtain anonymized data, and storing the anonymized data.

[0059] Optionally, the analysis of the data sensitivity of the topic message data can be achieved by identifying which types of data may be considered sensitive data in the Topic and assessing the data sensitivity level. Based on the data sensitivity, the construction of the desensitization strategy for the topic message data can be achieved by selecting appropriate desensitization methods according to the data sensitivity level and usage scenario. The desensitization tools can be desensitization tools such as replacement, encryption, obfuscation, and truncation.

[0060] The de-identified data can be stored using the storage tools pre-configured on the data integration platform.

[0061] S4. Transmit the desensitized data to the data testing environment to obtain test data. Perform a data integrity test on the test data to obtain the test results. If the test results are complete, the derivative task is completed.

[0062] In this embodiment of the invention, the de-identified data is transmitted to a data testing environment to obtain test data. The de-identified data can be used to simulate real business loads and evaluate the performance of the software system. For example, it can measure indicators such as response time, throughput, and resource utilization when processing large amounts of data. This helps to determine whether the system can meet the performance requirements of actual business and whether performance optimization is needed.

[0063] As an embodiment of the present invention, the step of transmitting the de-identified data to a data testing environment to obtain test data includes: analyzing the testing requirements of the data testing environment corresponding to the de-identified data; performing data preprocessing on the de-identified data based on the testing requirements to obtain preprocessed data; configuring the transmission parameters of the preprocessed data; performing a connection test on the transmission connection of the preprocessed data; and when the connection test is normal, transmitting the de-identified data to the data testing environment using the transmission parameters to obtain test data.

[0064] Optionally, the analysis of the anonymized data and the corresponding testing environment's testing requirements can be achieved by determining the testing objectives. For example, in an e-commerce application, testing objectives might include functional testing (such as shopping cart functionality, order processing, etc.), performance testing (such as response time, throughput), and security testing (such as user data protection, prevention of hacker attacks, etc.). Then, testing scenarios can be identified. In an e-commerce application, these might include normal shopping processes, high-traffic scenarios during promotional activities, and return and refund processes. The analysis can then determine which anonymized data is needed to simulate real-world situations in each scenario. Finally, based on the resource and performance requirements of the testing environment, the amount of data to be transmitted can be determined. The anonymized data is then preprocessed to obtain preprocessed data. Data processing can be achieved by cleaning and format conversion of the de-identified data. The transmission parameters can be obtained by determining the data transmission method, such as file transfer, database copying, network transmission, etc., and configuring the corresponding parameters. The connection test can be performed by using testing tools or writing scripts to simulate the data transmission process and check whether the connection is successful. Using the transmission parameters, the de-identified data is transmitted to the data testing environment by selecting the appropriate tools or commands to execute the data transmission. For example, for file transfer, FTP client software or command-line tools can be used to start the file transfer. For database copying, database management tools or scripts can be used to start the copying process.

[0065] Furthermore, by performing data integrity testing on the test data, the embodiments of the present invention can identify data errors, verify processing logic, improve system quality and stability, discover vulnerabilities, ensure system integration, provide an accurate data foundation for decision-making, optimize business processes, ensure data accuracy and reliability, and support efficient enterprise operation.

[0066] As an embodiment of the present invention, the step of performing data integrity testing on the test data to obtain test results includes: querying the test range of the test data, determining the test data source of the test data based on the test range, sampling the test data source to obtain sampled test data, and performing data integrity testing on the sampled test data to obtain test results.

[0067] The test scope refers to the specific areas, data types, or business process portions covered when conducting data consistency testing on the test data.

[0068] Optionally, the scope of the test data query can be defined by specifying the test objective. For example, in testing an e-commerce platform, the test objective might be to verify the consistency of user order data across different systems. The test scope could then include all order-related aspects such as the order generation system, inventory management system, and payment system. For a financial transaction system, if the test objective is to ensure the accuracy and consistency of transaction data, the test scope might cover the transaction matching system, settlement system, and account management system. It can also be determined by defining the data type (based on the test objective, the data types to be tested are determined; for example, in an e-commerce platform, this might be order number, product information, customer information, payment amount, etc.; in a financial transaction system, it might be transaction serial number, transaction type, transaction amount, transaction time, etc.). The process of determining the test data source based on the test scope is as follows: Based on the test scope, determine the data sources that may contain relevant test data. For example, in an e-commerce platform, the database of the order generation system, the database of the inventory management system, and the log files of the payment system could all be test data sources; in a financial transaction system, the transaction matching system... The system's database, settlement system's report files, and account management system's database are all potential test data sources. Evaluating a chosen data source to determine its reliability and accuracy can be done by examining its update frequency, data quality monitoring records, and the accuracy of historical data. For example, if a data source frequently shows data errors or is not updated in a timely manner, it may need to be used cautiously or additional verification measures may be required during testing. Data sampling of the test data source can be performed using methods such as random sampling and stratified sampling. For instance, in an e-commerce platform, to test the consistency of order data for different types of goods, stratified sampling can be used, sampling according to product category; for general testing, random sampling can be used. In financial transaction systems, stratified sampling can be performed based on factors such as transaction time and transaction type to ensure that the sampled data represents different transaction situations.

[0069] Furthermore, in this embodiment of the invention, when the test result is complete, completing the derivative task allows for in-depth analysis using the accurate data obtained from the derivative task, providing a basis for enterprises to formulate strategic decisions. For example, in marketing, by analyzing customer behavior data, target customer groups can be identified, marketing strategies optimized, and market share increased.

[0070] It should be noted that when the test result is complete, it means that the data of the Kafka cluster to be differentiated is consistent between production and testing.

[0071] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention. At the same time, it can improve the accuracy of real-time data testing to meet user needs.

[0072] In one embodiment, a real-time data testing apparatus is provided, which corresponds one-to-one with the real-time data testing methods described in the above embodiments. For example... Figure 3 As shown, the real-time data testing device includes a derivative task creation module 201, a derivative task execution module 202, a data storage module 203, and a data testing module 204. Detailed descriptions of each functional module are as follows:

[0073] The derivative task creation module 201 is used to select the Kafka cluster to be derived from the pre-configured data integration platform, configure the derivative environment for the Kafka cluster, perform logical classification on the Kafka cluster to obtain logically classified Kafka clusters, and create the derivative task for the logically classified Kafka cluster based on the derivative environment.

[0074] The derivative task execution module 202 is used to evaluate the rationality of the derivative task and obtain the evaluation result. When the evaluation result meets the preset result, it queries the execution resource quantity of the task execution system corresponding to the derivative task. When the execution resource quantity meets the requirement for executing the derivative task, it uses the task execution system to execute the derivative task.

[0075] The data storage module 203 is used to read the topic message data of the Kafka cluster during production when the derivative task is executed, perform data anonymization on the topic message data, and store the anonymized data.

[0076] The data testing module 204 is used to transmit the de-identified data to the data testing environment to obtain test data, perform data integrity testing on the test data, obtain test results, and complete the derivative task when the test results are complete.

[0077] In one embodiment, when the derivative task creation module 201 executes the configuration of the derivative environment for the Kafka cluster, it is configured to: determine the derivative requirements of the Kafka cluster; evaluate the derivative requirement resources of the Kafka cluster based on the derivative requirements; configure the derivative tools of the Kafka cluster based on the derivative requirement resources; and configure the derivative environment of the Kafka cluster based on the derivative tools.

[0078] In one embodiment, when the derivative task creation module 201 performs logical classification on the Kafka cluster to obtain a logically classified Kafka cluster, it is configured to: query the classification target and classification requirements of the Kafka cluster; perform data feature analysis on the Kafka cluster based on the classification target and classification requirements to obtain data analysis results; construct a logical classification scheme for the Kafka cluster based on the data analysis results; and perform logical classification on the Kafka cluster based on the logical classification scheme to obtain a logically classified Kafka cluster.

[0079] In one embodiment, when the derivative task execution module 202 performs a rationality assessment on the derivative task and obtains an assessment result, it is configured to: construct business requirement compliance standards, data quality requirement standards, resource feasibility standards, and time feasibility standards for the derivative task; determine an assessment index system for the derivative task based on the business requirement compliance standards, the data quality requirement standards, the resource feasibility standards, and the time feasibility standards; collect assessment data for the derivative task based on the assessment index system; and perform a rationality assessment on the derivative task based on the assessment data to obtain an assessment result.

[0080] In one embodiment, when the data storage module 203 performs data anonymization on the topic message data and stores the anonymized data, it is configured to: analyze the data sensitivity of the topic message data; construct an anonymization strategy for the topic message data based on the data sensitivity; configure an anonymization tool for the topic message data; perform data anonymization on the topic message data based on the anonymization strategy and the anonymization tool; and store the anonymized data.

[0081] In one embodiment, when the data testing module 204 transmits the de-identified data to the data testing environment to obtain test data, it is configured to: analyze the testing requirements of the data testing environment corresponding to the de-identified data; based on the testing requirements, perform data preprocessing on the de-identified data to obtain preprocessed data; configure the transmission parameters of the preprocessed data; perform connection testing on the transmission connection of the preprocessed data; and when the connection test is normal, use the transmission parameters to transmit the de-identified data to the data testing environment to obtain test data.

[0082] In one embodiment, when the data testing module 204 performs a data integrity test on the test data and obtains a test result, it is configured to: query the test range of the test data, determine the test data source of the test data based on the test range, sample the test data source to obtain sampled test data, perform a data integrity test on the sampled test data, and obtain a test result.

[0083] Specific limitations regarding the real-time data testing device can be found in the limitations of the real-time data testing method described above, and will not be repeated here. Each module in the aforementioned real-time data testing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0084] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a real-time data testing method on the server side.

[0085] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a real-time data testing method on the client side.

[0086] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0087] Select the Kafka cluster to be derived from the pre-configured data integration platform, configure the derivation environment for the Kafka cluster, perform logical classification on the Kafka cluster to obtain logically classified Kafka clusters, and create the derivation task for the Kafka cluster based on the derivation environment.

[0088] The rationality of the derivative task is evaluated to obtain the evaluation result. When the evaluation result meets the preset result, the execution resource quantity of the task execution system corresponding to the derivative task is queried. When the execution resource quantity meets the requirements for executing the derivative task, the task execution system is used to start executing the derivative task.

[0089] During the execution of the derivative task, the topic message data of the Kafka cluster during production is read, the topic message data is anonymized, and the anonymized data is stored.

[0090] The de-identified data is transmitted to a data testing environment to obtain test data. Data integrity testing is performed on the test data to obtain test results. If the test results are complete, the derivative task is completed.

[0091] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0092] Select the Kafka cluster to be derived from the pre-configured data integration platform, configure the derivation environment for the Kafka cluster, perform logical classification on the Kafka cluster to obtain logically classified Kafka clusters, and create the derivation task for the Kafka cluster based on the derivation environment.

[0093] The rationality of the derivative task is evaluated to obtain the evaluation result. When the evaluation result meets the preset result, the execution resource quantity of the task execution system corresponding to the derivative task is queried. When the execution resource quantity meets the requirements for executing the derivative task, the task execution system is used to start executing the derivative task.

[0094] During the execution of the derivative task, the topic message data of the Kafka cluster during production is read, the topic message data is anonymized, and the anonymized data is stored.

[0095] The de-identified data is transmitted to a data testing environment to obtain test data. Data integrity testing is performed on the test data to obtain test results. If the test results are complete, the derivative task is completed.

[0096] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0097] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0098] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0099] The above-described embodiments are merely illustrative of the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention. It should be noted that if any software tools or components not belonging to this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use.

Claims

1. A real-time data testing method, characterized in that, include: Select the Kafka cluster to be derived from the pre-configured data integration platform, and determine the derivative requirements of the Kafka cluster; Based on the aforementioned derivative requirements, assess the derivative resource requirements of the Kafka cluster; Based on the aforementioned derivative requirements, configure the derivative tools for the Kafka cluster; Based on the aforementioned data derivative tool, configure the data derivative environment of the Kafka cluster and query the classification goals and classification requirements of the Kafka cluster; Based on the classification objectives and classification requirements, data feature analysis is performed on the Kafka cluster to obtain data analysis results; Based on the data analysis results, a logical classification scheme for the Kafka cluster is constructed. Based on the logical classification scheme, the Kafka cluster is logically classified to obtain logically classified Kafka clusters. Based on the derivative environment, a derivative task for the logically classified Kafka cluster is created. The process involves constructing business requirement compliance standards, data quality requirement standards, resource feasibility standards, and time feasibility standards for the derivative task; determining an evaluation index system for the derivative task based on these standards; collecting evaluation data for the derivative task based on the evaluation index system; conducting a rationality assessment of the derivative task based on the evaluation data to obtain an evaluation result; querying the execution resource quantity of the task execution system corresponding to the derivative task when the evaluation result meets a preset result; and executing the derivative task using the task execution system when the execution resource quantity meets the requirements for executing the derivative task. When the derivative task is executed, the topic message data of the Kafka cluster during production is read, the topic message data is anonymized, and the anonymized data is stored. The de-identified data is transmitted to a data testing environment to obtain test data. Data integrity testing is performed on the test data to obtain test results. If the test results are complete, the derivative task is completed.

2. The real-time data testing method as described in claim 1, characterized in that, The step of de-identifying the topic message data and storing the de-identified data includes: Analyze the data sensitivity of the topic message data; Based on the data sensitivity, a desensitization strategy for the topic message data is constructed, and a desensitization tool for the topic message data is configured; Based on the aforementioned desensitization strategy and the aforementioned desensitization tool, the topic message data is desensitized to obtain desensitized data, and the desensitized data is stored.

3. The real-time data testing method as described in claim 1, characterized in that, The step of transmitting the de-identified data to the data testing environment to obtain test data includes: Analyze the testing requirements of the data testing environment corresponding to the anonymized data; Based on the aforementioned testing requirements, the de-identified data is preprocessed to obtain preprocessed data; Configure the transmission parameters of the preprocessed data and perform a connection test on the transmission connection of the preprocessed data; When the connection test is normal, the de-identified data is transmitted to the data test environment using the transmission parameters to obtain test data.

4. The real-time data testing method as described in claim 1, characterized in that, The process of performing data integrity testing on the test data to obtain test results includes: Query the test range of the test data; Based on the test data source, data is sampled from the test data source to obtain sampled test data; The sampled test data was subjected to a data integrity test to obtain the test results.

5. A real-time data testing device, characterized in that, include: The derivative task creation module is used to select the Kafka cluster to be derived from the pre-configured data integration platform and determine the derivative requirements of the Kafka cluster. Based on the aforementioned derivative requirements, assess the derivative resource requirements of the Kafka cluster; Based on the aforementioned derivative requirements, configure the derivative tools for the Kafka cluster; Based on the aforementioned data derivative tool, configure the data derivative environment of the Kafka cluster and query the classification goals and classification requirements of the Kafka cluster; Based on the classification objectives and classification requirements, data feature analysis is performed on the Kafka cluster to obtain data analysis results; Based on the data analysis results, a logical classification scheme for the Kafka cluster is constructed. Based on the logical classification scheme, the Kafka cluster is logically classified to obtain logically classified Kafka clusters. Based on the derivative environment, a derivative task for the logically classified Kafka cluster is created. The derivative task execution module is used to construct the business requirement compliance standard, data quality requirement standard, resource feasibility standard, and time feasibility standard for the derivative task; based on the business requirement compliance standard, data quality requirement standard, resource feasibility standard, and time feasibility standard, determine the evaluation index system for the derivative task; based on the evaluation index system, collect evaluation data for the derivative task; based on the evaluation data, perform a rationality evaluation on the derivative task to obtain an evaluation result; when the evaluation result meets a preset result, query the execution resource quantity of the task execution system corresponding to the derivative task; when the execution resource quantity meets the requirement for executing the derivative task, use the task execution system to execute the derivative task. The data storage module is used to read the topic message data of the Kafka cluster during production when the derivative task is executed, perform data anonymization on the topic message data, and store the anonymized data. The data testing module is used to transmit the de-identified data to the data testing environment to obtain test data, perform data integrity testing on the test data, obtain test results, and complete the derivative task when the test results are complete.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the real-time data testing method as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the real-time data testing method as described in any one of claims 1 to 4.

Images