Data processing method and device, storage medium and computer program product

Abstract

The invention discloses a data processing method and device, a storage medium and a computer program product, and relates to the field of data processing.The data processing method comprises the steps that cached data are obtained from a data cache region, and the cached data are original data from a preset data source; data processing is carried out on the cached data to obtain playback data, and the playback data has a playback time label; and distributing the playback data to a data receiving module of a target test environment according to the playback time label. By adopting the technical scheme, the problem that the test data conforming to the production environment cannot be quickly provided is solved.

Description

Technical Field

[0001] This application relates to the field of data processing, and more specifically, to a data processing method and apparatus, a storage medium, and a computer program product. Background Technology

[0002] In fields such as financial transactions involving high-concurrency real-time data processing, the development and testing chain for data applications is lengthy, with numerous requirements at each stage. When a stage changes, upstream resources often need to send test data frequently and repeatedly for logic and technical verification. Multiple environments, including development, testing, user verification, and pre-production, all require real-time data support for related tasks. However, test data that highly matches the real-time data of the production environment is unlikely to be accurate. 24 / 7 availability. Current technologies that export full or incremental data snapshots from the production database via scheduled tasks and then import them in batches to the test environment cannot provide a continuous data stream within a time window. This makes it impossible to verify business logic that relies on real-time sequences or states, and the data playback accuracy is only at the hourly or daily level, completely failing to meet the verification requirements of millisecond-level business scenarios. When test requirements change or specific scenario data is needed, there is a heavy reliance on database administrators or developers to manually perform data export, de-identification, transformation, and import operations. This process is highly dependent on manual intervention, and environment data preparation and switching typically require hours or even days of configuration time, resulting in low efficiency.

[0003] There is currently no effective solution to the problem that related technologies cannot quickly provide test data that meets production environment requirements.

[0004] Therefore, it is necessary to improve the relevant technology to overcome the aforementioned defects. Summary of the Invention

[0005] This application provides a data processing method and apparatus, a storage medium, and a computer program product to at least solve the problem in the related art of being unable to quickly provide test data that meets the production environment.

[0006] According to one aspect of the embodiments of this application, a data processing method is provided, comprising: obtaining cached data from a data cache area, wherein the cached data is raw data from a preset data source; performing data processing on the cached data to obtain playback data, wherein the playback data has a playback timestamp; and distributing the playback data to a data receiving module of a target test environment according to the playback timestamp.

[0007] In an exemplary embodiment, before retrieving cached data from the data cache area, the method further includes: determining a preset data extraction rule, wherein the data extraction rule includes connection information and data extraction range of the preset data source; performing a data extraction operation on the preset data source according to the preset data extraction rule to obtain heterogeneous data, wherein different data sources in the preset data source have different data formats, and the heterogeneous data includes data from different data formats of the preset data source; and performing a data format unification operation on the heterogeneous data to obtain the original dataset.

[0008] In an exemplary embodiment, data processing of the cached data to obtain replay data includes: determining the data replay conditions for a first data replay task, wherein the data replay conditions include data filtering rules, data sorting rules, and a data replay time period; filtering the cached data according to the data filtering rules to obtain target data; sorting the target data according to the data sorting rules to obtain sorted data; and adding a replay time tag to the sorted data according to the data replay time period to obtain the replay data.

[0009] In an exemplary embodiment, processing the cached data to obtain replay data further includes: obtaining the generation time period corresponding to the cached data and the generation timestamp corresponding to each piece of data in the original data; determining a time conversion rule based on the generation time period and the data replay time period corresponding to the second data replay task; determining a replay time tag corresponding to the generation timestamp in the data replay time period based on the time conversion rule; and adding the replay time tag to the cached data to obtain the replay data.

[0010] In an exemplary embodiment, a data receiving module that distributes the replay data to a target test environment based on the replay time tag includes: determining the target test environment that has a subscription relationship with the replay dataset based on subscription configuration information; and sending the replay data to the data receiving module of the target test environment in the form of a data stream.

[0011] In an exemplary embodiment, sending the playback data to the data receiving module of the target test environment as a data stream includes: determining the release rate mode of the data stream, wherein the release rate mode includes at least one of the following: original speed mode, accelerated playback mode, and decelerated playback mode; and sending the playback data to the data receiving module of the target test environment according to the release rate mode of the data stream.

[0012] In an exemplary embodiment, sending the playback data to the data receiving module of the target test environment in the form of a data stream includes: generating multiple data copies for the playback data when the target test environment includes multiple test environments; allocating a data copy to each test environment and sending each data copy to the data receiving module of the corresponding test environment in the form of a data stream; displaying the sending status and data processing rate of the multiple data copies to the target object on an interactive interface; and controlling the playback process of the multiple data copies according to the control instructions of the target object on the interactive interface, wherein the control instructions include a stop instruction, a start instruction, and a playback rate adjustment instruction.

[0013] According to another aspect of the embodiments of this application, a data processing apparatus is also provided, comprising: an acquisition module, configured to acquire cached data from a data cache area, wherein the cached data is raw data from a preset data source; a processing module, configured to perform data processing on the cached data to obtain playback data, wherein the playback data has a playback timestamp; and a distribution module, configured to distribute the playback data to a data receiving module of a target test environment according to the playback timestamp.

[0014] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer-readable storage medium, and the computer program is configured to execute the above-described data processing method when it is run.

[0015] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the data processing method described above through the computer program.

[0016] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps of the methods described in various embodiments of this application.

[0017] This application extracts data from different data sources into a data cache, processes the cached data to construct replayable data with timestamps, and finally distributes the replay data to the test environment for testing based on the replay timestamps. This solves the problem in related technologies of not being able to quickly provide test data suitable for a production environment. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a hardware structure block diagram of a computer terminal for a data processing method according to an embodiment of this application;

[0021] Figure 2 This is a flowchart of a data processing method according to an embodiment of this application;

[0022] Figure 3 This is an architecture diagram of a mirror data playback system according to an embodiment of this application;

[0023] Figure 4 This is a structural block diagram of a data processing apparatus according to an embodiment of this application. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] The methods and embodiments provided in this application can be executed on a computer terminal or similar computing device. Taking running on a computer terminal as an example, Figure 1 This is a hardware structure block diagram of a computer terminal for a data processing method according to an embodiment of this application. For example... Figure 1As shown, a computer terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a central processing unit (CPU) or a field-programmable gate array (FPGA)) and a memory 104 for storing data are also shown. The computer terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the computer terminal described above. For example, the computer terminal may also include components that are more complex than those described above. Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0027] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the data processing method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to a computer terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0028] The computer terminal uses a wireless network provided by a communications provider. In one example, transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0029] This embodiment provides a data processing method applied to the aforementioned computer terminal. Figure 2 This is a flowchart of a data processing method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:

[0030] Step S202: Obtain cached data from the data cache area, wherein the cached data is original data from a preset data source;

[0031] Step S204: Process the cached data to obtain playback data, wherein the playback data has a playback time tag;

[0032] Step S206: Distribute the playback data to the data receiving module of the target test environment according to the playback time tag.

[0033] Optionally, in the above steps, this method decouples the temporal dependencies between data extraction and data distribution by establishing a data cache as an intermediary, thereby achieving asynchronous and controllable data replay. The key lies in adding a "replay time stamp" to the processed data. This transforms the data from a mere copy of the original production data into one capable of simulating specific time-series behaviors in a test environment. For example, in end-of-day settlement tests in the financial sector, real transaction data generated in the production environment at 3 PM can be assigned a new "replay time stamp" (e.g., simulated as settlement data at 5 PM) and then distributed to the pre-production environment's settlement system. This allows the test environment to realistically simulate specific time-point business scenarios at any time (e.g., at night), solving the problem of strong binding between test data and real business time, and achieving the effect of data replay.

[0034] Through the above steps, data generated from different data sources is extracted into a data cache. The cached data is then processed to create timestamped, replayable data. Finally, the replay data is distributed to the test environment for testing based on the replay timestamps. This solves the problem in related technologies of not being able to quickly provide production-ready test data.

[0035] In an exemplary embodiment, before retrieving cached data from the data cache area, the method further includes: determining a preset data extraction rule, wherein the data extraction rule includes connection information and data extraction range of the preset data source; performing a data extraction operation on the preset data source according to the preset data extraction rule to obtain heterogeneous data, wherein different data sources in the preset data source have different data formats, and the heterogeneous data includes data from different data formats of the preset data source; and performing a data format unification operation on the heterogeneous data to obtain the original dataset.

[0036] Optionally, in the above embodiments, the preprocessing process of data from heterogeneous sources to a unified cache is refined. Through configurable data extraction rules and a unified storage engine, heterogeneous data with varying formats is standardized, laying a consistent data foundation for subsequent image generation. For example, the data sources of an IoT platform may include: structured device information tables from relational databases, real-time device status streams in JSON format from Kafka, and operation records in XML format from log files. First, these data sources are connected according to rules, and data within a specified range (such as a specific time period or a specific device ID) is extracted. Then, a parsing engine automatically infers and transforms the JSON and XML data into a unified format compatible with the database table structure, ultimately forming a raw dataset containing all necessary fields, which is then stored in the data cache.

[0037] Optionally, in the above embodiments, the data parsing process can also be completed using a heterogeneous data parsing engine that supports both manual specification and automatic inference modes. In automatic inference mode, the engine can intelligently identify and parse the input raw data (such as JSON, XML, and CSV formats), automatically output its structural definition, and convert it into a unified intermediate format within the system. This automatic inference process includes the following three levels:

[0038] Automatic Formatting Layer: This layer performs preliminary identification of input data based on multi-strategy sampling. For example, when processing a large data file, the engine can prioritize scanning the file header features (such as CSV files often being presented as plain text with rows and columns, JSON files starting with "{" or "[", and XML files starting with "").<?xml> The engine uses header sampling (or root element) and combines it with random sampling (such as extracting data from the middle and end of the file) for cross-validation. By combining header, random, and anomaly detection sampling strategies, the engine can efficiently and accurately determine the data format based on reading only a small amount of data (e.g., 1KB), avoiding the performance loss of full scanning.

[0039] The core layer for structure inference: Based on the sniffing results, this layer performs deep analysis on the sampled data to generate structured definitions. This layer employs differentiated logic for different formats; examples of these formats are provided below:

[0040] CSV format inference: First, a header check is performed by comparing the data type characteristics of the first and second rows. For example, if the first row consists entirely of strings such as "Customer ID" and "Name," while the second row contains mixed types (numbers and strings) such as "12345" and "Zhang San," then the first row is considered the header. Next, column type detection is performed: for example, if a column contains consecutive sampled values ​​"true," "false," and "true" using regular expressions, it can be inferred to be a Boolean type; if the sampled values ​​are all in the format "2023-01-01," it is inferred to be a date type. The inference follows a preset type priority (Boolean → Integer → Long Integer → Floating-point → Date / Timestamp → String), and all sampled data in the column must satisfy a higher-level type; otherwise, it is downgraded to a string type.

[0041] JSON format inference: Recursive traversal parsing is used. For example, when faced with market data fragments {"code":"600000","price":12.5} and {"code":"600001","price":15} from a real-time valuation system, the engine recognizes "code" as a string type and "price" as a floating-point type. If subsequent data appears such as {"code":123,"price":"N / A"}, the "code" field may be inferred as a union type (string or integer), and the "price" field may be downgraded to a string type due to type conflict. Fields with missing records are marked as nullable types.

[0042] XML format inference: The logic is similar, but attributes need to be processed. For example, ... <order id="1001"> <amount> 500< / amount> < / order> Convert it into an internal structure model like {"order":{"@id":1001, "amount":500}}, and then perform type inference on the "@id" attribute and the content of the "amount" element.

[0043] Unified Model Output Layer: Receives the inference results and outputs standardized metadata descriptions. For example, regardless of whether the input data is a CSV transaction log, a JSON market snapshot, or XML configuration information, this layer ultimately describes it as a series of "field name-type-value" triples, or structured metadata with explicit schemas and nullability markers, for use in downstream transformation and processing flows.

[0044] In an exemplary embodiment, data processing of the cached data to obtain replay data includes: determining the data replay conditions for a first data replay task, wherein the data replay conditions include data filtering rules, data sorting rules, and a data replay time period; filtering the cached data according to the data filtering rules to obtain target data; sorting the target data according to the data sorting rules to obtain sorted data; and adding a replay time tag to the sorted data according to the data replay time period to obtain the replay data.

[0045] When managing the aforementioned cached data, it can be grouped, categorized, and tagged according to the actual needs of the requester. The requester can view and retrieve the data they need, and filter the data based on its characteristics, such as extracting and replaying data based on field range conditions or time range conditions.

[0046] Optionally, in the above embodiments, taking a full-link load test of user orders on an e-commerce platform as an example, the data filtering rule can be defined as "only selecting business data of order and payment types"; the data sorting rule ensures that the business data is sorted according to the order time and payment time; the data replay time period can be set to simulate the traffic peak from 0:00 to 1:00 AM during a promotional holiday. Through the above embodiments, order data that meets the conditions is filtered from the cache, strictly sorted according to business logic, and each piece of data is labeled with a replay time tag simulating the peak time period, thereby generating a highly simulated load test data stream.

[0047] In an exemplary embodiment, processing the cached data to obtain replay data further includes: obtaining the generation time period corresponding to the cached data and the generation timestamp corresponding to each piece of data in the original data; determining a time conversion rule based on the generation time period and the data replay time period corresponding to the second data replay task; determining a replay time tag corresponding to the generation timestamp in the data replay time period based on the time conversion rule; and adding the replay time tag to the cached data to obtain the replay data.

[0048] Optionally, the above embodiments provide a refined time stamp generation strategy, particularly suitable for scenarios with extremely high requirements for the true temporal sequence of data. It doesn't simply assign a new time period to the data, but rather establishes a mapping relationship from the data generation time period to the data playback time period. For example, in securities transaction playback, the original data contains a series of orders and transaction records with millisecond-accurate generation timestamps, generated between 9:30:00 AM and 9:30:05 AM. The test aims to quickly replay all transactions occurring within these 5 seconds within 1 second to test the system's peak processing capacity. This method calculates a "time conversion rule" (e.g., 5x acceleration) based on these two time periods, and then maps the original timestamp 9:30:01.200 to the playback timestamp 00:00:00.240 according to this rule. This ensures that even with accelerated playback, the inherent and crucial millisecond-level temporal sequence relationship between different orders is completely preserved.

[0049] In an exemplary embodiment, a data receiving module that distributes the replay data to a target test environment based on the replay time tag includes: determining the target test environment that has a subscription relationship with the replay dataset based on subscription configuration information; and sending the replay data to the data receiving module of the target test environment in the form of a data stream.

[0050] Optionally, the above embodiments clearly define the target orientation and transmission mode of data distribution. Dynamically determining the distribution target through subscription configuration information reflects the platformization and flexibility of the system. Using a data stream for transmission ensures real-time data performance and low latency, which is crucial for achieving millisecond-level playback. For example, in a microservice architecture, a mirrored data stream of a "user profile update" can be simultaneously subscribed to by: development environment A (for debugging new profile algorithms), testing environment B (for verifying the compatibility of the profile service), and pre-production environment C (for user acceptance testing). Based on the subscription configuration, the system automatically pushes the same profile update record from the cache in parallel as a real-time data stream to designated Kafka topics in these three environments, achieving one-time mirroring for multiple reuse.

[0051] Optionally, in the above embodiments, the same data image may be needed in multiple environments such as development, performance testing, and user acceptance. Only one configuration is required; by controlling and selecting the corresponding distribution rules in the subscription configuration information, the data image can be distributed to the specified environments.

[0052] In an exemplary embodiment, sending the playback data to the data receiving module of the target test environment as a data stream includes: determining the release rate mode of the data stream, wherein the release rate mode includes at least one of the following: original speed mode, accelerated playback mode, and decelerated playback mode; and sending the playback data to the data receiving module of the target test environment according to the release rate mode of the data stream.

[0053] Optionally, in the above embodiments, the original speed-up mode is used for functional verification, realistically reproducing the production traffic pattern to ensure the correctness of business logic. The accelerated playback mode (e.g., 10x speed) is used for stress testing and performance bottleneck detection, injecting traffic far exceeding normal levels into the system in a short period. The decelerated playback mode can be used for debugging and problem localization, allowing developers to slowly and clearly observe the flow and state changes of data in the processing chain, facilitating the location of hidden bugs.

[0054] Optionally, in the above embodiments, for example, if the original data is actual transaction data, and it is now necessary to replay these actual transactions at double speed, the speed can be controlled to 2. Transactions that originally occurred within 10 minutes can be replayed in just 5 minutes. The purpose is to enable scenario expansion and derivation, such as for testing in more stressful transaction scenarios.

[0055] Optionally, in the above embodiments, the data playback process can be interrupted and then resumed. This method can simulate the availability of the system under data interruption scenarios, as well as the scenario of data interruption and recovery due to an anomaly.

[0056] Optionally, in the above embodiments, the following mechanism can also be set during playback:

[0057] Macro-rate control based on time mapping: The system scales the original time interval between data records proportionally according to a preset playback rate coefficient. For example, when the playback rate is set to 2x, the playback transmission interval between adjacent data records will be shortened to half of the original interval, thereby achieving accelerated playback of the overall data stream.

[0058] Micro-level traffic shaping based on token bucket: To ensure the stability of the network and data consumers, the system introduces a token bucket algorithm at the micro-level to smoothly control the data transmission rate. This mechanism effectively avoids traffic bursts that may occur due to direct transmission at scaled intervals, ensuring that the data stream is injected into the downstream system smoothly and in a controllable manner.

[0059] Pressure feedback adaptive flow control: The system monitors the processing status of consumers in each target environment in real time (e.g., through the delay of consumption acknowledgment (ACK)). When the consumption delay of a certain environment exceeds a safety threshold, the system can automatically trigger adaptive control of the data stream for that environment, such as temporarily discarding data marked as "non-core" or pausing distribution to that environment, thereby avoiding data accumulation and ensuring the stability of the overall system. This adjustment does not affect the normal playback of other environments.

[0060] Configurable Infinite Loop Playback Mode: The system supports infinite loop playback of data from a selected time period (such as peak business hours). This mode is particularly suitable for scenarios such as system durability testing and stability verification that require long-term operation, examining the system's performance under long-term operation through continuous playback load.

[0061] In summary, the aforementioned playback control mechanism achieves precise, flexible, and robust control over the data playback process by combining macro-rate scaling with micro-flow smoothing, supplemented by adaptive feedback and multi-mode support.

[0062] In an exemplary embodiment, sending the playback data to the data receiving module of the target test environment in the form of a data stream includes: generating multiple data copies for the playback data when the target test environment includes multiple test environments; allocating a data copy to each test environment and sending each data copy to the data receiving module of the corresponding test environment in the form of a data stream; displaying the sending status and data processing rate of the multiple data copies to the target object on an interactive interface; and controlling the playback process of the multiple data copies according to the control instructions of the target object on the interactive interface, wherein the control instructions include a stop instruction, a start instruction, and a playback rate adjustment instruction.

[0063] Optionally, in the above embodiments, the generated distribution strategy is a mapping rule that specifies which task's data should be distributed to which target. Its function is to allow data from a data source to be distributed to multiple environments. For example, multiple environments such as development, performance testing, and user acceptance may require the same data image. Therefore, the data source only needs to be configured once, controlling and selecting the appropriate distribution strategy to determine which environment and target the image should be distributed to.

[0064] Optionally, the above embodiments cover system scalability, visualization, and controllability in multi-target environment distribution. For example, in the parallel verification of a large-scale system refactoring, it is necessary to mirror the same production transaction data to five different physically isolated test clusters. The system generates a data copy for each cluster and displays the real-time "sending status" (e.g., connected, sending, delayed) and "data processing rate" (e.g., records processed per second) of all copies to the operations engineer through an interactive interface. If the engineer finds that the rate of one cluster is decreasing due to excessive load, he can directly issue a "replay rate adjustment command" to that cluster on the interface to temporarily adjust it from accelerated mode to the original speed without affecting the testing progress of other clusters. This fine-grained control capability greatly improves the management efficiency and success rate of complex testing activities.

[0065] To better understand the process of the above data processing method, the following description, in conjunction with optional embodiments, will illustrate the data processing method. In one optional embodiment, it can be achieved through methods such as... Figure 3 The mirror data playback system shown implements the above data processing method. The system is divided into a mirror extraction module and a mirror generation module, which handshake through a data buffer. The mirror extraction module on the left is initially overseen by an extraction controller. This controller maintains connection information for different data sources such as Oracle, TDSql, and Hive. It also packages data ranges, extraction order, special transformation rules, and scheduling, monitoring, and start / stop instructions into control commands and sends them to the extraction executor. Upon receiving the commands, the extraction executor pulls the initially stored heterogeneous data from the original data area. Data requiring secondary logical correction is first written to a temporary transformation area, then formatted and sent to the data buffer, forming a sequential data stream that can be resumed at any time. The image generation module on the right is managed by the generation controller, which is responsible for defining Kafka topics, replay rates, message formats, and distribution strategies. It drives the image generation processor through replay control commands. The processor subscribes to the content of the data buffer in a streaming manner, aligns the timing according to the Lamport logical clock, adjusts the flow according to the token bucket or Burst integer algorithm, accurately labels each piece of data with a replay time tag, encapsulates it into an image data stream, and finally pushes it to the corresponding Topic at a preset original speed, acceleration, or deceleration mode. Messages in a Topic can be subscribed to by multiple physically isolated test environments simultaneously. The system will copy an independent data copy for each environment and display the sending status and processing rate of each copy in real time on the interactive interface. Users can start, stop, or dynamically adjust the replay speed with one click, achieving millisecond-level cross-cluster environment switching and full-process automation.

[0066] In an optional embodiment, to achieve global causal consistency of multi-source heterogeneous data during playback, this embodiment constructs a timing control engine based on the Lamport logical clock and watermark mechanism to solve the data out-of-order problem caused by unreliable physical clocks (such as NTP deviation and network latency). This engine ensures that the output data stream strictly follows the logical chronological order of events (i.e., if event A occurs before event B, the playback order must remain A→B), thereby supporting accurate business logic reproduction and timing analysis. The core architecture and workflow of the timing control engine are as follows:

[0067] Timing Marking and Sequence Generation: At the data access layer, the system assigns a globally unique, monotonically increasing sequence number (Sequence ID) to each arriving data record. This sequence number is generated based on a Hybrid Logical Clock (HLC) mechanism and consists of:

[0068] Physical time component: The local physical time when the data enters the system is used as the coarse-grained reference.

[0069] Logical counting component: Based on Lamport logical clock rules, fine-grained adjustments are made to distinguish events from different sources within the same physical moment and to maintain causal dependencies.

[0070] Through this mechanism, even if the data comes from different physical sources and the original timestamps are biased, the system can still build a unified and causally consistent logical timeline for all events.

[0071] Reordering Buffer Layer: After data is sequenced, it is not immediately sent out. Instead, it enters a reordering buffer based on a priority queue (usually implemented using a min-heap). The purpose of this buffer is to temporarily store the data that has arrived but may be out of order, sort it according to its Sequence ID, and wait for "late data" that may arrive.

[0072] Water level triggering and ordered delivery: The system uses a water level mechanism to determine whether all data has arrived at a given moment. The water level is a logical time threshold that advances over time, representing the maximum time of a currently confirmed complete event. Specifically, the system tracks the latest event time from each data source and uses the minimum time among all sources as the global safety water level. When the logical time of data in the buffer is not greater than the current water level, it can be considered that all data before that moment has arrived, allowing it to be output downstream in sequence. The water level can be advanced periodically or based on events according to a strategy, thereby controlling the balance between the timeliness and completeness of data output.

[0073] Late data processing mechanism: For data that arrives after the water level has advanced (i.e., its logical time is less than the current water level), the system classifies it as a late event. To avoid affecting the already output ordered stream, this type of data is not directly inserted into the main data stream, but is instead imported into the side output stream, triggering monitoring alarms to notify operations and maintenance personnel. Furthermore, a configurable compensation processing strategy is provided. For example, during gaps where business logic allows and the main playback stream is idle, the system can automatically re-inject this late data into the processing flow with a lower priority, or record it for subsequent analysis, thereby maximizing the final integrity of the data while ensuring the real-time performance of the playback.

[0074] Through the control process that combines timing marking, buffer sorting, water level control and late processing, this system can achieve high-precision and causal consistency playback of multi-source data in a distributed asynchronous acquisition environment, providing reliable time dimension guarantee for testing, analysis and monitoring scenarios.

[0075] The aforementioned system architecture, through the collaborative work of the image extraction and image generation modules, achieves fully automated processing of the entire process from extraction, transformation, generation to distribution of multi-source heterogeneous data. It supports real-time, ordered replication of data images across media, templates, and clusters, meeting the high throughput, high timeliness, and strict causal ordering requirements of fields such as financial transactions and the Internet of Things for real-time data verification. Through configurable, template-based, process-oriented, and automated design, this technology platform effectively improves the efficiency and accuracy of data provision and reduces latency during environment switching and data synchronization.

[0076] In one optional embodiment, a real-time valuation system in the financial sector is used as the application scenario. In this scenario, the production environment continuously receives real-time market data (such as stock and bond prices) and transaction flow data. The development (DEV) environment and user acceptance testing (UAT) environment need to perform fault reproduction, algorithm verification, and system stress testing based on real historical data (especially abnormal data) generated by the production environment. Because the production and testing environments are in physically isolated clusters, and the data storage formats are heterogeneous (the production end uses a proprietary binary protocol, while the testing end requires a common format such as JSON), traditional data synchronization methods cannot meet the millisecond-level, order-preserving data playback requirements.

[0077] In this embodiment, the mirror data playback system is executed in the following four core stages:

[0078] 1. Mirror extraction stage.

[0079] Data Source Access and Identification: The extraction executor accesses the persistent storage layer of the production environment. In this embodiment, the raw data area contains two types of heterogeneous data sources: 1) market snapshots (Level-2 data) stored in a Redis cluster, in binary structure format; 2) transaction logs stored in an Oracle database, in structured table format. The extraction executor automatically identifies and accesses these data sources through its embedded heterogeneous data parsing engine, for example, identifying Redis key rules (such as MarketData:Stock:600000) and incremental logs from the Oracle database.

[0080] Data Extraction and Cleaning: The extraction controller issues instructions, setting the time window for the data to be extracted (e.g., 10:00:00 to 10:05:00). The extraction executor reads two types of data in parallel: Oracle data via JDBC interface and Redis data via RDB parsing tools. After reading, the system marks each data entry with a unified logical clock timestamp based on the Lamport logical clock, ensuring that data from different sources have comparable global time-series identifiers.

[0081] 2. Data conversion and temporary storage stage.

[0082] Format standardization and data anonymization: The extracted data is sent to a data transformation processor for standardization. This includes: 1) Protocol conversion: Converting binary market data in Redis to standard JSON format (e.g., {"code":"600000","price":12.5,"timestamp":1620000000}); 2) Logic modification and data anonymization: Using a custom transformation unit to perform hash anonymization on sensitive fields in the transaction log (e.g., "customer account"); and performing offset transformation on the time field to ensure the replay data conforms to the test environment's context on that day. The processed data is written to a temporary transformation area, which can be stored using high-performance storage (e.g., RocksDB or Oracle). All data is globally sorted and stored according to its "replay timestamp," where the replay timestamp = original timestamp + time offset × time compression factor.

[0083] 3. Image generation and control stage.

[0084] Traffic Shaping and Replay Generation: The image generation processor reads data from the temporary conversion area and generates an image stream according to the replay task requirements. For example, to meet stress testing needs, the testing team requires data to be replayed at twice the production environment rate. The generation controller issues the rate parameter (Rate=2.0), and the traffic shaping module in the image generation processor then activates burst mode, dynamically compressing the sending interval based on the difference in the data's timestamps to accelerate replay.

[0085] Resume transmission guarantee: During playback, if an interruption occurs due to network jitter or other anomalies, the generation controller records the offset of the currently transmitted data and uses a CRC32 checksum to ensure the integrity of the data at the breakpoint. Once the anomaly is resolved, the system automatically resumes transmission from the breakpoint, ensuring the continuity and integrity of the data stream.

[0086] 4. Cross-cluster replication and distribution phase.

[0087] Multi-target distribution: The image replication processor distributes the generated image data stream to multiple isolated test environments simultaneously according to pre-configured rules. In this embodiment: 1) Data from the development environment is pushed to a designated topic (such as Dev-Replay-Stream) in the Kafka cluster for developers to consume and debug in real time; 2) Data from the UAT environment is directly written to the HBase database for batch job verification.

[0088] Cross-cluster transmission: The system automatically establishes encrypted transmission tunnels between production and test clusters through configurable image distribution rules, enabling millisecond-level low-latency data delivery across physically isolated environments.

[0089] The mirror data playback system used in this embodiment has a high degree of configurability and management capabilities. Users can dynamically adjust playback behavior without modifying the code, such as changing the playback speed from 2x to 1x, or switching the output data format from JSON to XML. The changes only require modifying the corresponding configuration parameters in the generation controller interface. This greatly improves the flexibility of test scenario construction and operational efficiency.

[0090] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0091] This embodiment also provides a data processing apparatus for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, hardware implementations, or a combination of software and hardware, are also possible and contemplated.

[0092] Figure 4 This is a structural block diagram of a data processing apparatus according to an embodiment of this application. The apparatus includes:

[0093] The acquisition module 42 is used to acquire cached data from the data cache area, wherein the cached data is original data from a preset data source;

[0094] Processing module 44 is used to process the cached data to obtain playback data, wherein the playback data has a playback time tag;

[0095] The distribution module 46 is used to distribute the playback data to the data receiving module of the target test environment according to the playback time tag.

[0096] The aforementioned device extracts data from different data sources into a data cache, processes the cached data to construct replayable data with timestamps, and finally distributes the replay data to the test environment for testing based on the replay timestamps. This solves the problem in related technologies of not being able to quickly provide test data that meets production environment requirements.

[0097] In an exemplary embodiment, the acquisition module 42 is configured to determine a preset data extraction rule, wherein the data extraction rule includes connection information and data extraction range of the preset data source; perform data extraction operation on the preset data source according to the preset data extraction rule to obtain heterogeneous data, wherein different data sources in the preset data source have different data formats, and the heterogeneous data includes data from different data formats of the preset data source; and perform a data format unification operation on the heterogeneous data to obtain the original dataset.

[0098] In an exemplary embodiment, the processing module 44 is configured to determine the data playback conditions for the first data playback task, wherein the data playback conditions include data filtering rules, data sorting rules, and data playback time period; filter the cached data according to the data filtering rules to obtain target data; sort the target data according to the data sorting rules to obtain sorted data; and add playback time tags to the sorted data according to the data playback time period to obtain the playback data.

[0099] In an exemplary embodiment, the processing module 44 is configured to obtain the generation time period corresponding to the cached data and the generation timestamp corresponding to each piece of data in the original data; determine a time conversion rule based on the generation time period and the data playback time period corresponding to the second data playback task; determine the playback time tag corresponding to the generation timestamp in the data playback time period based on the time conversion rule; and add the playback time tag to the cached data to obtain the playback data.

[0100] In an exemplary embodiment, the distribution module 46 is configured to determine the target test environment that has a subscription relationship with the replay dataset based on the subscription configuration information; and send the replay data to the data receiving module of the target test environment in the form of a data stream.

[0101] In an exemplary embodiment, the distribution module 46 is configured to determine the data stream's publishing rate mode, wherein the publishing rate mode includes at least one of the following: original speed mode, accelerated playback mode, and decelerated playback mode; and to send the playback data to the data receiving module of the target test environment according to the data stream's publishing rate mode.

[0102] In an exemplary embodiment, the distribution module 46 is configured to: generate multiple data copies for the playback data when the target test environment includes multiple test environments; allocate a data copy to each test environment; send each data copy to the data receiving module of the corresponding test environment in the form of a data stream; display the sending status and data processing rate of the multiple data copies to the target object on an interactive interface; and control the playback process of the multiple data copies according to the control instructions of the target object on the interactive interface, wherein the control instructions include a stop instruction, a start instruction, and a playback rate adjustment instruction.

[0103] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when run.

[0104] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0105] S1, retrieve cached data from the data cache area, wherein the cached data is original data from a preset data source;

[0106] S2, perform data processing on the cached data to obtain replay data, wherein the replay data has a replay time tag;

[0107] S3, Distribute the playback data to the data receiving module of the target test environment according to the playback time tag.

[0108] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0109] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0110] Embodiments of this application also provide an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0111] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0112] S1, retrieve cached data from the data cache area, wherein the cached data is original data from a preset data source;

[0113] S2, perform data processing on the cached data to obtain replay data, wherein the replay data has a replay time tag;

[0114] S3, Distribute the playback data to the data receiving module of the target test environment according to the playback time tag.

[0115] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0116] Embodiments of this application also provide a computer program product, including a non-volatile computer-readable storage medium storing the computer program product, wherein the computer program, when executed by a processor, implements the steps of the methods described in various embodiments of this application.

[0117] Optionally, in this embodiment, the computer program described above can be configured to perform the following steps when executed by the processor:

[0118] S1, retrieve cached data from the data cache area, wherein the cached data is original data from a preset data source;

[0119] S2, perform data processing on the cached data to obtain replay data, wherein the replay data has a replay time tag;

[0120] S3, Distribute the playback data to the data receiving module of the target test environment according to the playback time tag.

[0121] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0122] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0123] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, include: Retrieve cached data from the data cache area, wherein the cached data is original data from a preset data source; The cached data is processed to obtain replay data, wherein the replay data has a replay time tag; The playback data is distributed to the data receiving module of the target test environment according to the playback time stamp.

2. The data processing method according to claim 1, characterized in that, Before retrieving cached data from the data cache, the method further includes: Determine preset data extraction rules, wherein the data extraction rules include the connection information of the preset data source and the data extraction range; Data extraction is performed on the preset data source according to the preset data extraction rules to obtain heterogeneous data, wherein different data sources in the preset data source have different data formats, and the heterogeneous data includes data from different data formats from the preset data source; The heterogeneous data is subjected to a data format unification operation to obtain the original dataset.

3. The data processing method according to claim 1, characterized in that, The cached data is processed to obtain playback data, including: Determine the data playback conditions for the first data playback task, wherein the data playback conditions include data filtering rules, data sorting rules, and data playback time period; The cached data is filtered according to the data filtering rules to obtain the target data; The target data is sorted according to the data sorting rules to obtain sorted data; The replay time is added to the sorted data according to the replay time period to obtain the replay data.

4. The data processing method according to claim 1, characterized in that, The process of processing the cached data to obtain playback data also includes: Obtain the generation time period corresponding to the cached data and the generation timestamp corresponding to each piece of data in the original data; The time conversion rule is determined based on the generation time period and the data playback time period corresponding to the second data playback task. The playback time label corresponding to the generated timestamp in the data playback time period is determined according to the time conversion rule; Add the playback timestamp to the cached data to obtain the playback data.

5. The data processing method according to claim 1, characterized in that, The data receiving module that distributes the playback data to the target test environment according to the playback time stamp includes: The target test environment that has a subscription relationship with the replay dataset is determined based on the subscription configuration information; The playback data is sent as a data stream to the data receiving module of the target test environment.

6. The data processing method according to claim 5, characterized in that, Sending the playback data to the data receiving module of the target test environment as a data stream includes: Determine the data stream's publishing rate mode, wherein the publishing rate mode includes at least one of the following: original speed mode, accelerated playback mode, and decelerated playback mode; The playback data is sent to the data receiving module of the target test environment according to the data stream publishing rate mode.

7. The data processing method according to claim 5, characterized in that, Sending the playback data to the data receiving module of the target test environment as a data stream includes: In cases where the target test environment includes multiple test environments, multiple data copies are generated for the playback data; Allocate a data copy for each test environment, and send each data copy to the data receiving module of the corresponding test environment in the form of a data stream; The interactive interface displays the sending status and data processing rate of the multiple data copies to the target object. The playback process of the multiple data copies is controlled according to the control instructions of the target object on the interactive interface, wherein the control instructions include stop instructions, start instructions, and playback rate adjustment instructions.

8. A data processing apparatus, characterized in that, include: The acquisition module is used to acquire cached data from the data cache area, wherein the cached data is raw data from a preset data source; The processing module is used to process the cached data to obtain replay data, wherein the replay data has a replay time tag; The distribution module is used to distribute the playback data to the data receiving module of the target test environment according to the playback time tag.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method of any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images