Data processing method and device, equipment, storage medium and program product

Abstract

The application discloses a data processing method and device, equipment, a storage medium and a program product, and belongs to the technical field of data processing. The method obtains a structured data set corresponding to a real transaction message and field mapping information, the data set includes transaction fields, corresponding field characteristics and field values, and the field mapping information is used for establishing a bidirectional mapping relationship between message domain identifiers and transaction fields; the data set is horizontally separated according to the field characteristics and service semantics, and a plurality of data blocks with non-intersecting fields are obtained through division; the data blocks are vertically layered and split according to the occurrence frequency of field combinations in the data blocks, and a plurality of high-frequency and low-frequency data subblocks are obtained; a data synthesizer dedicated to each data subblock is called to generate a simulation data subblock respectively, all simulation data subblocks are fused based on the field mapping information, and a simulation transaction message is restored. In this way, the scene coverage rate and simulation accuracy of the simulation transaction data are improved.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, and in particular relates to a data processing method, apparatus, device, storage medium and program product. Background Technology

[0002] As the core hub of financial transactions, the transaction system collaborates closely with fund clearing institutions, transaction initiators, and service providers to complete the entire service process, including payment, settlement, and risk control. For the dual purpose of conducting transaction system stability testing and supporting the training of large-scale financial models in the payment sector, the industry requires massive amounts of real transaction data as its foundational material. However, real transaction data carries sensitive personal information and is subject to strict legal restrictions, making it difficult to circulate and share across institutions and directly usable for model training and system testing. Therefore, technical solutions for constructing high-fidelity simulated transaction data have emerged.

[0003] In related technologies, mainstream simulation message generation schemes fall into two categories: one relies on fixed templates to generate simulated transaction data in batches, and the other relies on manually configuring transaction parameters based on service scenarios. However, both of these schemes have significant shortcomings. The generated simulated transaction data has limited scenario coverage and cannot fully cover various extreme long-tail scenarios such as overseas transactions, large-scale marketing promotions, and error timeouts, significantly reducing the accuracy of transaction system detection and testing. At the same time, the fidelity of simulated transaction data is insufficient, making it impossible to output high-quality training datasets that cannot meet the training requirements of large-scale models in the financial sector. Summary of the Invention

[0004] This application provides a data processing method, apparatus, device, storage medium, and program product that can solve the problems of incomplete scenario coverage and low simulation accuracy in existing simulated transaction data.

[0005] In a first aspect, embodiments of this application provide a data processing method, the method comprising: Obtain the structured dataset and field mapping information of the real transaction message. The structured dataset includes multiple transaction fields obtained after disassembling the real transaction message, the field features corresponding to each transaction field, and the field value of each transaction field. The field mapping information is used to record the bidirectional mapping relationship between the message field identifier of the real transaction message and the transaction fields of the structured dataset. Based on the field features and service semantic information corresponding to each transaction field, the structured dataset is horizontally segmented to obtain M data blocks. Each data block includes at least two transaction fields, and the transaction fields in different data blocks do not overlap. Based on the frequency of occurrence of the field values ​​of each transaction field in the structured dataset, each data block is vertically segmented into frequency bands to obtain N data sub-blocks corresponding to each data block; The simulation data sub-blocks for each data sub-block are generated by a data synthesizer corresponding to each data sub-block. Based on the field mapping information, the simulated data sub-blocks of all data sub-blocks are merged to obtain the simulated transaction message corresponding to the real transaction message.

[0006] Secondly, embodiments of this application provide a data processing apparatus, the apparatus comprising: The acquisition module is used to acquire the structured dataset and field mapping information of the real transaction message. The structured dataset includes multiple transaction fields obtained after the real transaction message is decomposed, the field features corresponding to each transaction field, and the field value of each transaction field. The field mapping information is used to record the bidirectional mapping relationship between the message field identifier of the real transaction message and the transaction fields of the structured dataset. The processing module is used to perform horizontal field partitioning on the structured dataset based on the field features and service semantic information corresponding to each transaction field, resulting in M ​​data blocks. Each data block includes at least two transaction fields, and the transaction fields in different data blocks do not overlap. The splitting module is used to split each data block vertically into frequency bands based on the frequency of occurrence of the field values ​​of each transaction field in the structured dataset, so as to obtain N data sub-blocks corresponding to each data block; The generation module is used to generate simulation data sub-blocks for each data sub-block through a data synthesizer corresponding to each data sub-block; The fusion module is used to fuse the simulated data sub-blocks of all data sub-blocks according to the field mapping information to obtain the simulated transaction message corresponding to the real transaction message.

[0007] Thirdly, embodiments of this application provide a computer device, which includes: a processor and a memory storing computer program instructions; the processor executes the computer program instructions to implement the data processing method as shown in the first aspect.

[0008] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the data processing method as shown in any one of the first to fifth aspects.

[0009] In one aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the data processing method as shown in any one of the first to fifth aspects.

[0010] The data processing method, apparatus, device, storage medium, and program product of this application embodiment acquire a structured dataset and field mapping information of a real transaction message. The structured dataset includes multiple transaction fields obtained after disassembling the real transaction message, field features corresponding to each transaction field, and field values ​​of each transaction field. The field mapping information is used to record the bidirectional mapping relationship between the message field identifier of the real transaction message and the transaction fields of the structured dataset. Based on the field features and service semantic information corresponding to each transaction field, the structured dataset is horizontally segmented to obtain M data blocks. Each data block includes at least two transaction fields, and the transaction fields in different data blocks do not overlap. According to the frequency of occurrence of the field values ​​of each transaction field in the structured dataset, each data block is vertically segmented by frequency band to obtain N data sub-blocks corresponding to each data block. A simulated data sub-block for each data sub-block is generated through a data synthesizer corresponding to each data sub-block. According to the field mapping information, the simulated data sub-blocks of all data sub-blocks are fused to obtain the simulated transaction message corresponding to the real transaction message. In this way, a structured dataset including complete transaction fields, field features, and field values ​​can be obtained by parsing real transaction messages. Horizontal field segmentation is then performed based on field features and service semantic information, grouping transaction fields that match service semantics and mathematical correlations into independent data blocks. This avoids the distortion of field relationships caused by mixed modeling of transaction fields from different service dimensions. Furthermore, by vertically segmenting each data block according to the frequency of field combinations, high-frequency regular transaction samples and low-frequency long-tail transaction samples are accurately distinguished, achieving differentiated, fine-grained data partitioning. Subsequently, a dedicated data synthesizer for each data sub-block independently generates corresponding simulation data sub-blocks, enabling targeted fitting of data distribution characteristics for regular and scarce long-tail transaction scenarios. This effectively compensates for the shortcomings of traditional simulation models in covering niche and low-frequency transaction scenarios. Finally, multiple simulation data sub-blocks are fused and restored using field mapping information, fully preserving the field association rules and message structure features of real transaction messages. This significantly improves the consistency of distribution, accuracy of field associations, and scenario adaptability between simulated and real transaction data, effectively solving the technical problems of incomplete scenario coverage and low simulation accuracy in existing simulated transaction data. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A schematic diagram of a data processing system provided in an embodiment of this application; Figure 2A two-dimensional effect data diagram of a data processing method provided in an embodiment of this application; Figure 3 A flowchart illustrating a data processing method provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a data processing apparatus provided in one embodiment of this application; Figure 5 This is a schematic diagram of the structure of a computer device provided in one embodiment of this application. Detailed Implementation

[0013] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are intended only to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0014] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0015] The acquisition, storage, use, and processing of data (including but not limited to features and information mentioned in this document) in the technical solution of this application all comply with the relevant provisions of national laws and regulations.

[0016] In the financial sector, with the rapid development of artificial intelligence technology, financial institutions are increasingly demanding high-quality training data. Transaction data, as a primary resource for training financial AI models, is subject to strict legal and compliance constraints in its acquisition and use. For example, in service scenarios such as building pilot-scale financial AI bases and implementing large-scale payment models, massive amounts of real transaction data face the dual challenges of privacy compliance and high-quality simulation modeling. Real transaction data is subject to relevant laws, such as the Personal Information Protection Law, and cross-institutional sharing and circulation present compliance barriers. Traditional transaction synthesis schemes rely solely on fixed statistical rules for modeling, making it difficult to uncover implicit field relationships with strong semantic connections but sparse numerical values ​​in payment scenarios, such as the inherent coupling relationship between merchant categories, acquiring institutions, and transaction channels. This results in poor generalization ability of simulation data in online gray-scale testing and system stress simulation scenarios, and insufficient coverage of long-tail extreme transaction scenarios such as holiday promotions, niche merchants, and abnormal error reports.

[0017] In related technologies, transaction data can be generated in the following ways: one method is to use templates to configure probe messages, or to manually configure simulated transaction messages to mimic production scenarios. Another method is to directly feed the produced transaction data into a neural network for training, and then use the trained model to generate simulated transaction messages.

[0018] However, the aforementioned prior art has the following problems, as detailed below.

[0019] For the first method, using template configuration often has limited coverage, only covering most normal production scenarios. It is difficult to cover abnormal or a few scenarios. As for manually simulating production scenario configuration, firstly, it consumes a lot of manpower, and secondly, manual simulation can only imitate service scenarios and cannot imitate or replicate the distribution of transactions in production.

[0020] For the second approach, although neural networks can learn transaction information from production data, due to the large volume of financial transactions and the complexity of transaction scenarios, it is difficult to learn the complete information of production data through neural networks alone. Often, the correlation between fields is lost and the transaction information of a few scenarios is overwhelmed by the transaction information of a large number of scenarios. In addition, since production transactions are directly fed into neural networks for learning, a single session usually consumes a large amount of processor memory, which places high demands on the hardware of the training environment.

[0021] It is evident that the aforementioned existing technologies have significant drawbacks in addressing the problem of generating simulated transaction messages, including limited scenario coverage, loss of field association information, submersion of long-tail distributions, and high consumption of hardware resources.

[0022] To address the aforementioned shortcomings, this application proposes a data processing method, apparatus, device, storage medium, and program product based on dual-dimensional adaptive semantic segmentation. This method deeply integrates spectral clustering field segmentation technology based on statistical correlation, vertical frequency band segmentation technology based on frequency distribution inflection points, hyperparameter optimization technology based on active fine-tuning of generated results, and semantic verification technology driven by a large language model. This achieves high-fidelity and high-efficiency simulation of real transaction messages, effectively supporting the requirements of training large financial models and improving the accuracy of transaction system detection and testing.

[0023] The following will be combined with the appendix Figures 1 to 5 The data processing methods, apparatus, devices, storage media, and program products described in this application are described in detail. It should be noted that these embodiments are not intended to limit the scope of this application.

[0024] The following is combined with Figure 1 The payment system provided in the embodiments of this application will be described in detail.

[0025] Figure 1 This is a schematic diagram of a data processing system provided in an embodiment of this application.

[0026] like Figure 1 As shown, the data processing system 10 in this embodiment may include a transaction preprocessing module 101, a data segmentation module 102, a data simulation module 103, and a transaction conversion module 104.

[0027] The detection method provided in the embodiments of this application will be described in detail below.

[0028] The transaction preprocessing module 101 is the data entry point for generating simulated transaction messages provided in this application embodiment. It is used for cleaning, format conversion and field mapping template generation of real transaction messages, providing standardized and configurable data input for subsequent data segmentation and model training.

[0029] Specifically, this module can be used to acquire original real transaction messages; clean the original real transaction messages, remove incomplete transaction data, and fill in null values ​​in some fields; decompose the cleaned transaction messages into multiple transaction fields, extract the field features and field values ​​corresponding to each transaction field, and convert each decomposed field value into a CSV format file line by line according to each transaction record to obtain a structured dataset; based on the correspondence between the message field identifiers of the real transaction messages and the transaction fields in the CSV format file, generate a field mapping template to represent the bidirectional mapping relationship for use by the subsequent transaction conversion module. In this embodiment, CSV is a general file format that stores tabular data in plain text form, where each line represents a data record, and each field is separated by a comma or other delimiter.

[0030] The data segmentation module 102 is used for two-dimensional adaptive data segmentation based on field features and field values. It divides the original structured dataset into multiple data blocks and data sub-blocks with strong inter-field correlation and relatively simple field content, providing a data foundation for subsequent block training.

[0031] This module incorporates a horizontal field segmentation engine and a vertical frequency band splitting engine, pre-configured with two segmentation strategies: spectral clustering analysis based on statistical correlation and frequency band partitioning based on frequency distribution inflection points. Its functions include: receiving the structured dataset output from the transaction preprocessing module; triggering the horizontal field segmentation engine to horizontally segment the dataset based on field features, obtaining M data blocks; triggering the vertical frequency band splitting engine to vertically split each data block based on the frequency of field values, obtaining N data sub-blocks corresponding to each data block; and outputting the segmentation results to the data simulation module 103.

[0032] Based on this, the specific implementations of the horizontal field segmentation engine and the vertical frequency band splitting engine will be explained in detail below.

[0033] The horizontal field splitting engine is used to group transaction fields according to statistical correlation, so that the fields within the same data block are strongly correlated, while the fields between different data blocks are weakly correlated.

[0034] Specific functions may include: calculating the statistical correlation between any two transaction fields in the structured dataset based on the data type indicated by the field features of each transaction field, and determining the graph adjacency matrix, degree matrix, and Laplace matrix based on the statistical correlation. Eigenvalue decomposition is performed on the Laplace matrix to obtain an ascending sequence of eigenvalues ​​and the differences between adjacent eigenvalues ​​in the sequence, thereby determining the optimal number of subgroups M to avoid manually setting the number of subgroups and ensure that the grouping results highly match the inherent structure of the data. Based on the number of subgroups M, a spectral clustering algorithm is executed to divide the transaction fields into M initial field groups. M initial field groups are input into the semantic field classification agent. The agent verifies and corrects the M initial field groups based on the service semantic information corresponding to the field names of each transaction field. Specifically, the agent extracts the service semantic information of each transaction field in each initial field group and determines whether all transaction fields in the group belong to the same service category. If there are transaction fields that do not belong to the same service category, they are marked as fields to be corrected, and moved to the matching target field group according to their service semantic information. The above verification and correction steps are repeated until all transaction fields in each initial field group belong to the same service category. The resulting M initial field groups are then output as M data blocks. For each of the M data blocks, the average statistical correlation between each transaction field in the current data block and all other transaction fields in the structured dataset that belong to the current data block is calculated. This average correlation is used as the cross-group average correlation for that transaction field. The transaction field with the highest cross-group average correlation in the current data block is determined as the anchor field of the current data block. This anchor field is used as a condition variable in subsequent synthesis to build cross-group dependency chains, fundamentally ensuring the global consistency of the synthesized data.

[0035] The vertical frequency band splitting engine is used to stratify field values ​​according to their frequency of occurrence within each data block, so that high-frequency, medium-frequency and low-frequency data can be trained independently, thereby solving the problem of long-tail distribution being overwhelmed.

[0036] Specific functionalities may include: For each data block, obtaining the field values ​​of each transaction field contained in that data block within the structured dataset. Counting the frequency of each field value in the structured dataset and sorting the field values ​​in descending order of frequency. Plotting a frequency vs. ranking curve in a log-log coordinate system with the sorting position of each field value as the x-axis and the corresponding frequency as the y-axis. Smoothing the frequency vs. ranking curve using a Savitzky-Golay filter to obtain a smoothed curve. Calculating the curvature of the smoothed curve and selecting local maxima of curvature as inflection points, determining the number of detected inflection points. Adaptively determining the frequency band division strategy based on the number of inflection points: when the number of inflection points is zero, treating the entire data block as a single data sub-block; when there is one inflection point, dividing the data block into two data sub-blocks using the inflection point as the boundary; and when there are two inflection points, dividing the data block into three data sub-blocks using the two inflection points as the boundary. This method is entirely driven by data distribution and does not require a preset frequency threshold.

[0037] The data simulation module 103 is the core of the data synthesis and generation of the data processing system provided in this application embodiment. It can call the data synthesizer corresponding to each pre-trained data sub-block to generate the simulation data sub-blocks of each data sub-block in batches, providing complete simulation data input for the transaction conversion module.

[0038] Specifically, the data simulation module 103 can be used to receive M data blocks and N data sub-blocks corresponding to each data block output by the data segmentation module 102; for each data sub-block, load the data synthesizer corresponding to the data sub-block from the model storage area; apply the data synthesizer to the data sub-block to generate the corresponding simulation data sub-block; summarize the simulation data sub-blocks corresponding to all data sub-blocks and output them to the transaction conversion module 104.

[0039] In this system, there is a one-to-one correspondence between the data synthesizer and the data sub-blocks; that is, each data sub-block has its own independently trained, independently tuned, and dedicated data synthesizer. The network parameters and hyperparameter configurations of the data synthesizers corresponding to different data sub-blocks are independent of each other and do not affect each other. By calling the data synthesizer corresponding to each data sub-block to generate simulation data, the local statistical distribution characteristics and field relationships of each data sub-block can be accurately restored. At the same time, through the subsequent fusion processing of the transaction conversion module 104, the overall structure of the original structured dataset is restored, and finally, a simulated transaction message consistent with the format of the real transaction message is output.

[0040] The transaction conversion module 104 is the output assembly module of the data processing system provided in this application embodiment, which is responsible for merging the simulation data sub-blocks generated by each data synthesizer into a complete simulation transaction message.

[0041] Specific functions may include: obtaining the simulation data sub-block corresponding to each data sub-block; using the anchor field corresponding to each data block determined by the horizontal field separation engine as the association primary key, vertically merging the simulation data sub-blocks of all data sub-blocks belonging to the same data block to obtain the simulation data block corresponding to each data block; using the anchor field as the cross-block association primary key, horizontally splicing M simulation data blocks to obtain a complete simulation structured dataset, the field structure of which is the same as the original structured dataset; and converting the simulation structured dataset line by line into a simulation transaction message consistent with the format of the real transaction message according to each transaction record, based on the bidirectional mapping relationship between the message field identifier and the transaction field recorded in the field mapping template.

[0042] In addition, in some embodiments of this application, in order to train the data synthesizer provided by the data training module 103 corresponding to each data sub-block, the data processing system 10 in this embodiment of the application may also include a data training module 150 and an active fine-tuning module 160.

[0043] The data training module 150 uses a Generative Adversarial Network (CTGAN) from the synthetic data domain to independently model and train each data sub-block after it has been divided by the adaptive partitioning network.

[0044] Specifically, the data training module 150 may perform the following training steps for each data sub-block: obtain the training sample corresponding to the data sub-block, that is, the real data sub-block corresponding to the data sub-block in the structured dataset; input the training sample into the CTGAN corresponding to the data sub-block for training, the CTGAN includes a generator and a discriminator; during the training process, the name semantics and data type of the transaction field are introduced into the training process as key condition information, so that the generator can learn the deep data distribution characteristics of specific fields under different service categories; record the loss value change curve of the generator and the loss value change curve of the discriminator.

[0045] The active fine-tuning module 160 is the tuning module of the data processing system provided in this application embodiment. It undertakes the functions of evaluating the training quality and adaptively optimizing the hyperparameters of the CTGAN model corresponding to each data sub-block, and provides the data training module with closed-loop model tuning capability.

[0046] The specific functions include: after training reaches a preset number of rounds, generating test synthetic data using the current CTGAN generator, and calculating the distribution coefficient (Column Shapes Score) and correlation coefficient (Column Pair Trends Score) between the test synthetic data and the training samples; determining whether the distribution coefficient is greater than or equal to a second threshold and whether the correlation coefficient is greater than or equal to a third threshold; if so, the current training result is deemed satisfactory, training is stopped, and the currently obtained CTGAN is used as the output of the data synthesizer corresponding to that data sub-block; if not, i.e., the distribution coefficient is less than the second threshold or the correlation coefficient is less than the third threshold, the oscillation index and convergence slope in the generator's loss value change curve are extracted. Based on the oscillation index and convergence slope, the training hyperparameters of the CTGAN corresponding to that data sub-block are adjusted according to a preset mapping rule, and the CTGAN is retrained using the adjusted hyperparameters. If the generative adversarial network for the same data sub-block fails to meet the target after multiple consecutive training sessions (i.e., the cumulative number of retraining sessions for the same data sub-block is greater than or equal to the fourth threshold), then the morphological features, distribution coefficient, and correlation coefficient of the generator's loss value change curve are input into the semantic analysis big language model to obtain feedback information output by the big language model. The feedback information includes at least one of the following: analysis of the reasons for training failure and a generative adversarial network parameter adjustment scheme. The feedback information is used to trigger manual intervention to adjust the generative adversarial network.

[0047] Therefore, the data processing system provided in this application deeply integrates spectral clustering field segmentation based on statistical correlation, vertical frequency band segmentation based on frequency distribution inflection points, hyperparameter optimization based on active fine-tuning of generated results, and semantic verification driven by a large language model. Compared with the prior art, it achieves the following technical effects: First, it improves the fidelity of synthesized simulated transaction messages. That is, through horizontal field segmentation, transaction fields with strong inter-field correlation and similar data types are grouped together for independent training, avoiding the loss of complex data information; through vertical frequency band segmentation, high-frequency, medium-frequency, and low-frequency data are processed in layers, solving the problem of long-tail data being overwhelmed. Figure 2The experimental results show that, based on 50,000 real training data points, the frequency distribution of values ​​in the synthetic data of this embodiment across eight key transaction fields—Biztp, ChannelIssrId, MrchntTpId, PyeeAcctIssrId, PyeeIssrId, PyerAcctId, SysRtncd, and TrxTrmTp—is essentially consistent with the real production data. Secondly, it reduces training resource consumption by dividing the complete structured dataset into multiple data blocks and sub-blocks. Each sub-block contains significantly less data than the original dataset, resulting in a substantial reduction in GPU memory usage during independent training of each sub-block. This avoids the problem of excessive GPU memory consumption caused by feeding the entire dataset into the neural network at once. Furthermore, the active fine-tuning module automatically adjusts hyperparameters such as learning rate, batch size, and number of training epochs based on loss curve characteristics during training, such as oscillation exponent and convergence slope. This avoids the underfitting problem caused by fixed hyperparameters in traditional methods. When training fails to meet targets multiple times, a large language model is introduced for in-depth analysis, providing an interpretable basis for manual intervention. Then, through the anchor field filtering mechanism, the field with the highest external correlation is selected from each horizontally partitioned data block as the link for cross-group dependency chains. This ensures that the simulation data generated by the independently trained data synthesizers after partitioning can still maintain the integrity of cross-field service logic during fusion.

[0048] It should be noted that the data processing system provided in this application embodiment can be applied to transaction system stability detection and grayscale verification, and to the training of large-scale financial models in the payment vertical field, as shown below.

[0049] For scenarios involving stability testing and canary release verification of trading systems, the trading system, as a financial hub, collaborates closely with fund clearing institutions, transaction initiators, and service providers to complete the entire service process, including payment, settlement, and risk control. Therefore, before system launch, IT innovation transformation, and version iteration, sufficient testing and verification are required to ensure stability. However, performance testing often selects a limited number of transactions for single-transaction stress testing or mixed-transaction stress testing, resulting in insufficient transaction coverage and mixed concurrency coverage, frequently leading to insufficient system load capacity after launch. Furthermore, verification methods such as canary releases rely on real customer transaction traffic, making it impossible to verify the functions of the unlaunched system in advance. Based on this, in this embodiment, the data segmentation module 102 can perform horizontal field segmentation and vertical frequency band splitting of the structured dataset, decomposing the original transaction data into multiple data blocks and data sub-blocks with strong field correlation and relatively simple content; each data sub-block is generated by an independently trained data synthesizer to produce a corresponding simulated data sub-block. The transaction conversion module 104 uses the anchor field as the associated primary key to merge the simulated data sub-blocks into a complete simulated transaction message. Testers can conduct full-link stress testing, abnormal scenario injection testing, and gray-scale environment simulation verification based on the generated simulated transaction messages, and discover system vulnerabilities in advance without relying on real customer transaction traffic.

[0050] In this way, by splitting high-frequency, medium-frequency, and low-frequency transactions vertically, simulated transaction data covering various extreme long-tail scenarios such as overseas transactions, marketing promotions, and error timeouts can be generated. This overcomes the limitation of traditional template solutions in terms of scenario coverage. It does not rely on real customer transaction traffic and can complete functional verification and stress testing in advance before the system goes live. Moreover, through horizontal field separation and anchor field mechanisms, the synthesized data is highly consistent with real production data in terms of field distribution and cross-field relationships, ensuring the validity of test results.

[0051] For the training of large-scale financial models in the payment sector, as these models rapidly penetrate core services, the construction of such models requires massive amounts of high-quality transaction data as training material. However, real transaction data carries sensitive personal information and is strictly restricted by laws and regulations. Processing of personal information should be limited to the minimum scope required to achieve financial service processing, making cross-institutional sharing difficult. High-quality financial datasets remain scarce; internal bank data is fragmented and inconsistent in definition. Traditional synthesis methods rely on fixed templates or manual configuration, resulting in insufficient fidelity in the generated simulation data, failing to meet the requirements for large-scale model training. Therefore, this embodiment uses a transaction preprocessing module 101 to clean, decompose, and convert real transaction messages into a structured dataset, extracting the field features and values ​​of each transaction field. The data synthesizer corresponding to each data sub-block is trained by the data segmentation module 102 and the active fine-tuning module 160; the data simulation module 103 calls each data synthesizer to generate high-fidelity simulated data sub-blocks in batches; the transaction conversion module 104 merges each simulated data sub-block into a complete simulated transaction message. The generated simulated transaction message is highly consistent with the real data in terms of statistical distribution and field correlation, and can be used as high-quality training data for the payment vertical domain large model.

[0052] In this way, the generated simulated transaction messages do not contain real user personal information, and can be circulated and shared across institutions without violating personal information protection laws, for use in large model training. Further reference is available. Figure 2 Experimental verification shows that, based on 50,000 real training data points, the frequency distribution of values ​​in the eight key transaction fields of the transaction data synthesized by the data processing system in this embodiment is basically consistent with that of real production data. Horizontal and vertical segmentation improved the overall score from 68.52% to 82.75% and from 78.73% to 82.56%, respectively, significantly improving the statistical fidelity of the synthesized data. Vertical frequency band splitting effectively generates transaction data for extreme scenarios such as the Spring Festival promotion, compensating for the scarcity of long-tail samples in real data and improving the generalization ability of large models in edge scenarios. Moreover, by decomposing the complete dataset into multiple data sub-blocks for independent training through data segmentation, the processor memory occupied during the training of each sub-block is significantly reduced, avoiding the high hardware resource requirements of a single full training run.

[0053] Based on the data processing system mentioned above, this application embodiment also provides an application for, for example, Figure 1 The data processing method of the data processing system shown can be specifically combined with... Figure 3 Please provide a detailed explanation.

[0054] Figure 3 This is a flowchart of a data processing method provided in an embodiment of this application.

[0055] like Figure 3 As shown, this data processing method can be applied to, for example... Figure 1 The data processing system shown may include the following specific data processing methods: Step 310: Obtain the structured dataset and field mapping information of the real transaction message. The structured dataset includes multiple transaction fields obtained after disassembling the real transaction message, the field features corresponding to each transaction field, and the field value of each transaction field. The field mapping information is used to record the bidirectional mapping relationship between the message field identifier of the real transaction message and the transaction fields of the structured dataset. Step 320: Based on the field features and service semantic information corresponding to each transaction field, perform horizontal field segmentation processing on the structured dataset to obtain M data blocks. Each data block includes at least two transaction fields, and the transaction fields in different data blocks do not overlap. Step 330: Based on the frequency of occurrence of the field values ​​of each transaction field in the structured dataset, perform vertical frequency band segmentation on each data block to obtain N data sub-blocks corresponding to each data block. Step 340: Generate a simulated data sub-block for each data sub-block using a data synthesizer corresponding to each data sub-block. Step 350: Based on the field mapping information, fuse the simulated data sub-blocks of all data sub-blocks to obtain the simulated transaction message corresponding to the real transaction message.

[0056] In this way, a structured dataset including complete transaction fields, field features, and field values ​​can be obtained by parsing real transaction messages. Horizontal field segmentation is then performed based on field features and service semantic information, grouping transaction fields that match service semantics and mathematical correlations into independent data blocks. This avoids the distortion of field relationships caused by mixed modeling of transaction fields from different service dimensions. Furthermore, by vertically segmenting each data block according to the frequency of field combinations, high-frequency regular transaction samples and low-frequency long-tail transaction samples are accurately distinguished, achieving differentiated, fine-grained data partitioning. Subsequently, a dedicated data synthesizer for each data sub-block independently generates corresponding simulation data sub-blocks, enabling targeted fitting of data distribution characteristics for regular and scarce long-tail transaction scenarios. This effectively compensates for the shortcomings of traditional simulation models in covering niche and low-frequency transaction scenarios. Finally, multiple simulation data sub-blocks are fused and restored using field mapping information, fully preserving the field association rules and message structure features of real transaction messages. This significantly improves the consistency of distribution, accuracy of field associations, and scenario adaptability between simulated and real transaction data, effectively solving the technical problems of incomplete scenario coverage and low simulation accuracy in existing simulated transaction data.

[0057] The steps described above are explained in detail below.

[0058] Regarding step 310, in this embodiment of the application, before step 310, the data processing method may further include steps 3001 to 3004, as shown below.

[0059] Step 3001: Perform data cleaning on the real transaction message to obtain the cleaned transaction message.

[0060] The data cleaning process includes at least one of the following: removing incomplete transaction data and supplementing missing field values ​​in actual transaction messages. Specifically, removing incomplete transaction data means identifying and filtering messages that are missing key fields or have non-standard formats to ensure data integrity in subsequent processing. For example, message records lacking merchant IDs or transaction amounts are removed during the cleaning phase. Supplementing missing field values ​​in actual transaction messages means that for fields that are allowed to be empty but need to be used as placeholders in subsequent processing, they are appropriately filled according to service rules or context information, such as using default values, average values, or specific identifiers, to maintain the structural integrity and consistency of the data records.

[0061] It is important to note that data cleaning is fundamental to ensuring the quality of synthesized data. Raw, real transaction messages may contain data quality issues due to various reasons, such as network transmission anomalies or system acquisition biases. Directly using this abnormal data for subsequent segmentation and model training will cause the generated data synthesizer to learn incorrect data patterns, resulting in low-quality simulation data. This data cleaning step ensures the quality and consistency of the input data, providing a reliable data foundation for subsequent field decomposition, feature extraction, and model training.

[0062] Step 3002: Decompose the cleaned transaction message into multiple transaction fields.

[0063] In this step, each cleaned transaction message is broken down into multiple independent transaction fields according to its message structure. For example, a typical payment transaction message "20260704|Merchant ID M001|Payment Channel A|Amount 99.00|Debit Card|Return Code 00" is broken down into the following fields: Transaction Date: "20260704", Merchant ID: "Merchant ID M001", Payment Channel: "Payment Channel A", Transaction Amount: "Amount 99.00", Bank Card Type: "Debit Card", and Return Code: "Return Code 00".

[0064] In this step, the transaction message is a data collection organized in a specific format, including multiple field fields, each carrying different service information. This step decomposes the aggregated transaction message into independent transaction fields and their corresponding field values ​​according to field boundaries, transforming the unstructured message data into data elements organized by field. This allows subsequent steps to perform independent feature extraction and data processing on each transaction field. This decomposition process preserves the correspondence between fields and field values, providing structured data material for subsequent CSV format conversion.

[0065] Step 3003: Extract the field features and field values ​​corresponding to each transaction field, and convert the field values ​​of each transaction field into a CSV format file line by line according to each transaction record to obtain a structured dataset.

[0066] In this step, for each transaction field obtained in step 3002, the corresponding field features are extracted. The field features include at least the data type, such as numeric or categorical, and may further include metadata information such as field length, precision, and value range.

[0067] For example, the transaction amount field has the characteristics of "numeric type, maximum length 15 digits, retain 2 decimal places"; the payment channel field has the characteristics of "category type, enumerated values ​​include payment channel A, payment channel B, payment channel C, etc." The field values ​​of each transaction field obtained in step 3002 are written to a CSV file line by line for each transaction record. Specifically, each transaction message is treated as a row in the CSV file, the values ​​of each field in the transaction message are treated as the columns of that row, and the field names are used as the header (first row) of the CSV file, as shown in Table 1 below: Table 1 The CSV format used in this embodiment is a tabular data storage format. Converting transaction messages into CSV format files allows subsequent data segmentation modules to read and process data in a standardized manner. Simultaneously, extracting field features provides a basis for algorithm selection for subsequent horizontal field segmentation. Numerical fields are calculated using the Pearson correlation coefficient, while categorical fields are calculated using the Cramer coefficient. The decomposition in step 3002 and the CSV conversion in this step together transform the original message into a complete structured dataset that includes both field features (column-level metadata) and field values ​​(row-level data instances), providing complete data input for subsequent horizontal and vertical segmentation.

[0068] Step 3004: Generate field mapping information to represent the bidirectional mapping relationship based on the correspondence between the message field identifier of the real transaction message and the transaction fields in the structured dataset.

[0069] In this embodiment, the bidirectional mapping relationship includes forward mapping and reverse mapping. Forward mapping refers to the mapping from message field identifiers to CSV field names, i.e., which transaction field in the CSV corresponds to a given message field. Reverse mapping refers to the mapping from CSV field names to message field identifiers, i.e., which message field in the CSV corresponds to a given transaction field in the original message. Based on this, the bidirectional mapping relationship can be generated as field mapping information or a field mapping template, saved in structured data form, for use by the subsequent transaction conversion module.

[0070] In this step, the field mapping information serves as the basis for subsequently converting the simulation data back into the transaction message format. After the data synthesizer generates simulation data sub-blocks, these sub-blocks are merged into a complete simulation structured dataset. At this point, the CSV format simulation data needs to be converted back into simulation transaction messages consistent with the original real transaction message format. The bidirectional mapping relationship recorded in the field mapping information provides the basis for the conversion from CSV fields to message fields, enabling the conversion process to accurately restore the complete structure and field order of each transaction message.

[0071] Taking the example message "20260704|Merchant ID M001|Payment Channel A|Amount 99.00|Debit Card|Return Code 00" as an example, the field mapping information records that the first field in the original message corresponds to the "Transaction Date" in the CSV, the second field corresponds to the "Merchant ID", and so on, with the sixth field corresponding to the "Return Code". When the transaction conversion module needs to convert the simulated CSV data into a simulated transaction message, it extracts the field values ​​of each row in the CSV according to this bidirectional mapping relationship and reassembles them according to the field order and delimiters of the original message to restore the transaction message with the same format as the original message.

[0072] This allows for the standardization of real transaction messages during preprocessing, generating standardized structured datasets and bidirectional field mapping information, providing standard and usable input for subsequent full-process processing.

[0073] Regarding step 320, in some embodiments of this application, the specific implementation of the horizontal field separation process includes at least data type as the field feature in the embodiments of this application. Based on this, step 320 may include steps 3201 to 3205.

[0074] Step 3201: Calculate the statistical correlation between any two transaction fields in the structured dataset according to the data type indicated by the field characteristics of each transaction field.

[0075] In this step, the statistical correlation degree in this embodiment refers to a quantitative indicator used to measure the degree of statistical correlation between any two transaction fields in a structured dataset. Its specific value depends on the data type of the field; for numeric fields, the absolute value of the Pearson correlation coefficient is used as the statistical correlation degree; for categorical fields, the Cramer coefficient is used. The range of the statistical correlation degree is [0,1], with a larger value indicating a stronger statistical dependency between the two fields.

[0076] Step 3202: Based on the graph adjacency matrix corresponding to the statistical correlation between any two transaction fields, determine the eigenvalue sequence of the Laplacian matrix corresponding to the graph adjacency matrix; wherein, the eigenvalue sequence includes eigenvalues ​​arranged in ascending order of numerical value, and the eigenvalues ​​are used to characterize the penalty value for each transaction field being divided into different data blocks with other transaction fields in the structured dataset, and the number of eigenvalues ​​in the eigenvalue sequence is the same as the number of transaction fields in the structured dataset.

[0077] In this embodiment, the graph adjacency matrix refers to a square matrix constructed using transaction fields in a structured dataset as nodes and the statistical correlation between any two transaction fields as edge weights. This matrix has P rows and P columns, where P is the total number of transaction fields in the structured dataset. Its off-diagonal elements represent the statistical correlation between corresponding two transaction fields, and its diagonal elements are 0. This matrix is ​​used to organize and store pairwise associations between fields in a unified mathematical form, providing input for subsequent spectral clustering analysis.

[0078] In this embodiment, the Laplacian matrix is ​​a matrix jointly determined by the graph adjacency matrix and the degree matrix, defined as L=DA, where D is the degree matrix and A is the graph adjacency matrix. The diagonal elements of the Laplacian matrix are the diagonal elements of the degree matrix; the off-diagonal elements are the negatives of the corresponding statistical correlations in the graph adjacency matrix, representing the penalty value for partitioning two transaction fields into different data blocks. This matrix is ​​used to transform the positive correlation between fields into a graph cut cost metric required by the spectral clustering algorithm, thus highlighting the inherent group boundaries of the data.

[0079] In this embodiment, the degree matrix refers to the diagonal matrix corresponding to the graph adjacency matrix. Its diagonal elements are the sum of all elements in the corresponding row of the graph adjacency matrix, that is, the sum of the statistical correlation between the i-th transaction field in the structured dataset and all other transaction fields except itself. It is used to characterize the total correlation strength or centrality of the i-th transaction field in the entire field association network. The off-diagonal elements are all 0.

[0080] In this application's embodiments, the penalty value refers to the mathematical cost of dividing two transaction fields into different data blocks. In the Laplace matrix, the penalty value is specifically represented by the values ​​of the off-diagonal elements, i.e., the negative value of the statistical correlation. The greater the statistical correlation between two fields, the larger the absolute value of their corresponding penalty value, indicating a higher cost to forcibly separate them, thus tending to group them into the same data block during spectral clustering.

[0081] In this embodiment, the eigenvalue sequence refers to the sequence obtained by arranging all eigenvalues ​​in ascending order of their numerical values ​​after performing eigenvalue decomposition on the Laplacian matrix. This sequence is denoted as λ1≤λ2≤…≤λ P P represents the total number of transaction fields in the structured dataset. Each eigenvalue in the eigenvalue sequence represents the cost required to segment the transaction field along the corresponding orthogonal dimension. The smaller the eigenvalue, the stronger the statistical correlation and the more pronounced the clustering trend among the fields in that dimension; the larger the eigenvalue, the more discrete the distribution among the fields in that dimension. Significant jumps between adjacent eigenvalues, i.e., eigenvalue gaps, reflect the natural separation boundary of the data's inherent groups.

[0082] Step 3203: Determine the number of groups M based on the feature value sequence.

[0083] Step 3204: According to the number of groups M, the transaction fields in the structured dataset are divided into M initial transaction field groups using the spectral clustering algorithm.

[0084] Step 3205: Based on the service semantic information of each transaction field in each initial transaction field group, verify and correct the M initial transaction field groups, and determine the M verified and corrected initial transaction field groups as M data blocks.

[0085] In this embodiment, the service semantic information refers to information extracted from the field names of transaction fields that characterizes the service category or service meaning of that field at the service level. For example, "transaction amount" and "transaction date" belong to the "basic transaction information" category, "merchant number" and "merchant name" belong to the "merchant information" category, and "return code" and "channel identifier" belong to the "transaction result information" category. This information is used to verify and correct the service rationality of the grouping results based on the pure mathematical clustering results, so that the final data blocks maintain consistency in both statistical association and service semantic dimensions.

[0086] In this embodiment, the initial transaction field group refers to the intermediate grouping result after the transaction fields in the structured dataset are mathematically divided according to the number of groups M using a spectral clustering algorithm, but before service semantic verification and correction. This intermediate result is based solely on statistical correlation and must undergo service semantic verification and correction before it can be used as the final data block output.

[0087] The verification and correction in this embodiment refers to the process of reviewing and adjusting the initial transaction field groups output by the spectral clustering algorithm at the service semantic level. Specifically, this includes: extracting the service semantic information of each transaction field in each initial transaction field group; verifying whether all transaction fields within the group belong to the same service category; if there are transaction fields that do not belong to the same service category, marking them as fields to be corrected, and moving them to other matching field groups based on their service semantic information. This process is used to correct statistically correlated but service-irrelevant misgroupings that may occur in pure mathematical clustering, ensuring that the final grouping results simultaneously satisfy statistical rationality and service semantic consistency.

[0088] Therefore, the complete execution logic of horizontal field separation can be limited. First, the field coupling relationship is quantified by statistical correlation. Then, the number of groups is automatically determined and spectral clustering is completed based on the eigenvalues ​​of the Laplacian matrix. Finally, the grouping results are corrected by combining service semantic verification. There is no need to manually preset the number of groups, which ensures that the fields within the divided data blocks are closely mathematically related and the service semantics are unified. This avoids the division deviation caused by manual grouping and improves the simulation accuracy of subsequent block modeling.

[0089] Furthermore, in this embodiment of the application, a unified similarity measurement framework is provided to address the characteristic of mixed numerical and categorical fields in CSV data. The absolute value of the Pearson correlation coefficient is used for numerical fields, and the Cramer correlation coefficient is used for categorical fields, ensuring that the association strength between fields of different data types is comparable, so as to convert the calculation results of the above two algorithms on CSV data into a graph adjacency matrix. Based on this, step 3201 mentioned above may specifically include: When the transaction field is of numeric data type, the Pearson correlation coefficient is used to calculate the statistical correlation between any two transaction fields in the structured dataset.

[0090] In this step, the Pearson correlation coefficient can be expressed as shown in the following formula (1): in, For any two transaction fields, The mean is taken. As a statistical correlation degree, the value range is [0, 1].

[0091] When the data type of the transaction field is categorical, the Cramer correlation coefficient is used to calculate the statistical correlation between any two transaction fields in the structured dataset.

[0092] In this step, the Cramér's correlation coefficient (V) can be expressed as follows: in, This is the chi-square statistic. v represents the total number of transaction fields in the structured dataset, and v represents the statistical correlation. These represent the number of columns and rows in the structured dataset, respectively, with a value range of [0, 1], and 0 when completely independent.

[0093] Therefore, corresponding correlation coefficients are selected to calculate statistical correlation for the two types of transaction fields, namely numerical and categorical, to match the data characteristics of payment transaction fields, accurately quantify the inherent coupling relationship between different types of fields, ensure that the correlation calculation results are consistent with the actual service distribution, and provide reliable data basis for subsequent spectral clustering and grouping.

[0094] Furthermore, in the embodiments of this application, regarding the determination of the graph adjacency matrix and the Laplace matrix, the above step 3202 may specifically include steps 32021 to 32024, as shown below.

[0095] Step 32021: Based on the statistical correlation between any two transaction fields, construct a graph adjacency matrix. The graph adjacency matrix is ​​a square matrix of P rows and P columns, where P is the total number of transaction fields in the structured dataset. The off-diagonal elements of the graph adjacency matrix represent the statistical correlation between any two transaction fields, and the diagonal elements of the graph adjacency matrix represent the first threshold.

[0096] In this step, if there are P transaction fields in the structured dataset, each transaction field is considered a node. Step 32021 uses the statistical correlation between any two transaction fields calculated in step 3201 as the weight of the edge between nodes, constructing a P-row, P-column graph adjacency matrix A. Specifically, the element A in the i-th row and j-th column of the graph adjacency matrix A... ij This represents the statistical correlation between the i-th and j-th transaction fields. For off-diagonal elements (i≠j), the statistical correlation value calculated in step 3201 is directly entered, with a range of [0,1]. The larger the value, the stronger the statistical correlation between the two fields. For diagonal elements (i=j), this represents the correlation between a node and itself, defined as 0 in standard graph theory, meaning no self-loop is formed, so it is set to the first threshold of 0. This matrix is ​​a symmetric matrix (A). ij =A jiThis fully characterizes the pairwise statistical relationships between all transaction fields, providing a unified mathematical input for subsequent spectral clustering analysis.

[0097] Step 32022: Determine the Laplacian matrix based on the graph adjacency matrix and the degree matrix corresponding to the graph adjacency matrix; wherein, the degree matrix is ​​a diagonal matrix, and the diagonal elements in the diagonal matrix are the sum of the statistical correlation degrees between the first transaction field in the structured dataset and all second transaction fields in the structured dataset except for the first transaction field; the off-diagonal elements in the Laplacian matrix are the penalty values ​​for dividing the first transaction field and the second transaction field into different data blocks, and the diagonal elements in the Laplacian matrix are the diagonal elements in the degree matrix.

[0098] In this step, the Laplacian matrix L is defined as L=DA, where D is the degree matrix and A is the graph adjacency matrix constructed in step 32021. This step transforms the graph adjacency matrix A into the Laplacian matrix L to suit the mathematical requirements of spectral clustering. Specifically, the degree matrix D is a diagonal matrix, and its diagonal elements D ii Let D be the sum of all elements in the i-th row of the graph adjacency matrix A. ii =ΣjA ij Let represent the sum of statistical correlations between the i-th transaction field and all other transaction fields in the structured dataset. Subtracting the graph adjacency matrix A from the degree matrix D yields the Laplace matrix L = DA. The diagonal elements L of the Laplace matrix L are... ii =D ii , represents the diagonal element of the degree matrix; L represents the off-diagonal element of L. ij (i≠j) = -A ij The value is the inverse of the statistical correlation, i.e., the partitioning cost. The larger the absolute value, the higher the partitioning cost required to divide these two fields into different data blocks. This mathematical transformation from positive correlation to negative partitioning cost allows subsequent eigenvalue decomposition to naturally reveal the inherent group boundaries of the data: fields with strong correlations incur larger negative penalties, while fields with weaker correlations incur smaller penalties.

[0099] Step 32023: Perform eigenvalue decomposition on the Laplace matrix to obtain P eigenvalues.

[0100] In this step, eigenvalue decomposition is performed on the P-row, P-column Laplace matrix L, i.e., solving for non-zero solutions satisfying L*v = λ*v, where v is the eigenvector and λ is the corresponding eigenvalue. Since the Laplace matrix L is a real symmetric matrix, all its eigenvalues ​​are real numbers and non-negative. Eigenvalue decomposition maps the graph structure represented by the Laplace matrix L from the original P-dimensional space to P mutually orthogonal dimensional directions, with the energy magnitude in each direction represented by the corresponding eigenvalue. A total of P eigenvalues ​​are obtained after eigenvalue decomposition, the number of which is consistent with the total number of transaction fields in the structured dataset.

[0101] Step 32024: Sort the P feature values ​​in ascending order according to their numerical values ​​to obtain the feature value sequence.

[0102] In this step, the P eigenvalues ​​obtained in step 32023 are sorted in ascending order of value to obtain the eigenvalue sequence λ1≤λ2≤…≤λ P This sorting result forms the data basis for calculating the eigenvalue gap (Eigengap) in the subsequent step 32025. Each eigenvalue in the eigenvalue sequence represents the cost required to segment the transaction fields in the corresponding dimension. The smaller the eigenvalue, the stronger the statistical correlation and the more obvious the clustering trend among the transaction fields in that dimension, and the lower the segmentation cost. The larger the eigenvalue, the weaker the statistical correlation and the more discrete the distribution among the transaction fields in that dimension, and the higher the segmentation cost. When there is a significant jump between adjacent eigenvalues, i.e., where the eigenvalue gap is the largest, it represents the natural boundary from tight clustering to loose independence. The index corresponding to this position is the automatically determined optimal number of groups K.

[0103] Therefore, the construction rules and eigenvalue decomposition process of graph adjacency matrix, degree matrix and Laplacian matrix can be fully defined, the graph transformation method of field association relationship can be standardized, the quantification logic of field grouping penalty value can be unified, the calculation results of feature value sequence can be stable and reproducible, and the objectivity and consistency of field grouping can be improved.

[0104] Furthermore, in the embodiments of this application, regarding the specific method of determining the number of groups based on the feature value sequence, the above step 3203 may specifically include steps 32031 and 32032.

[0105] Step 32031: Based on the feature value sequence, calculate the difference between adjacent feature values ​​in the feature value sequence to obtain multiple feature value gaps.

[0106] In this step, step 32024 has performed eigenvalue decomposition on the Laplace matrix and sorted it in ascending order, resulting in the eigenvalue sequence λ1≤λ2≤…≤λP, where P is the total number of transaction fields in the structured dataset. For the sorted eigenvalue sequence, the difference between two adjacent eigenvalues ​​is calculated sequentially using the following formula (3): Based on this, P-1 eigenvalue gaps can be obtained. The eigenvalue gap quantifies the degree of jump between adjacent sorting positions in the eigenvalue sequence; that is, when the eigenvalue gap is significantly larger at a certain point, it indicates that the eigenvalue has jumped drastically at that position. In spectral clustering theory, the magnitude of the eigenvalues ​​of the Laplacian matrix reflects the cost required to segment the nodes in the graph. Smaller eigenvalues ​​correspond to more closely related nodes in the same dimension, indicating they belong to the same group; larger eigenvalues ​​correspond to more dispersed nodes in the same dimension. Assuming the first K eigenvalues ​​are small, and the (K+1)th eigenvalue suddenly increases significantly, then the Eigengap(K) between the Kth and (K+1)th eigenvalues ​​will be significantly larger. In this case, the sorting index K corresponding to the maximum eigenvalue gap reflects the natural boundary between tightly clustered and loosely independent clusters; that is, in the dimension before the Kth eigenvalue, they should belong to the same group; starting from the (K+1)th eigenvalue, they should belong to different groups.

[0107] Step 32032: Select the sorting index value corresponding to the largest value among the gaps of multiple feature values, and use it as the number of groups M.

[0108] In this step, the largest value among all the eigenvalue gaps calculated in step 32031 is found using the following formula (4): The optimal number of groups M is determined by selecting the sorting index k corresponding to the maximum value of Eigengap(k). This sorting index k is the optimal number of groups M. The basis for this step is that the eigenvalues ​​of the Laplace matrix exhibit spectral gaps when the data has a natural population structure. When there are k tightly clustered groups in the dataset, the eigenvalues ​​of the Laplace matrix exhibit the following pattern: the first k eigenvalues ​​are very small, close to 0, corresponding to k tightly connected internal groups; the (k+1)th eigenvalue suddenly increases significantly. Therefore, the gap Eigengap(k) between the kth and (k+1)th eigenvalues ​​will be significantly larger than the gaps between other adjacent eigenvalues. In this case, the sorting index k of the kth eigenvalue can be used as the number of groups M. By selecting the index corresponding to the maximum value of Eigengap, the optimal number of groups can be determined entirely by the data distribution, completely avoiding the subjectivity and uncertainty of manually pre-setting the number of groups.

[0109] Here, we can illustrate this with the following example. Suppose the structured dataset has 5 transaction fields, and the following sequence of eigenvalues ​​is obtained after eigenvalue decomposition of the Laplacian matrix: λ1=0.01, λ2=0.02, λ3=0.05, λ4=0.98, λ5=1.5 Calculate the difference between adjacent feature values ​​in sequence: Eigengap(1)=λ2-λ1=0.02-0.01=0.01 Eigengap(2)=λ3-λ2=0.05-0.02=0.03 Eigengap(3)=λ4-λ3=0.98-0.05=0.93 (maximum value) Eigengap(4)=λ5-λ4=1.5-0.98=0.52 Eigengap(3) is the maximum value (0.93), and its corresponding sort index value k=3. Therefore, the number of groups is determined to be M=3. This result indicates that the fields in the dimension directions corresponding to the first 3 feature values ​​should be classified into the same group or a few close groups. There is a significant jump between the 3rd and 4th feature values, indicating the inherent natural separation boundary of the data.

[0110] Therefore, the optimal number of groups M can be automatically determined by utilizing the maximum value of the eigenvalue gap. This grouping is driven entirely by the inherent distribution of the data, avoiding the problems of over-division and under-division caused by manually setting the number of groups. This makes the grouping results more objective and accurate, and applicable to different types of transaction datasets.

[0111] Furthermore, in the embodiments of this application, the specific implementation of service semantic verification and correction of the initial transaction field group, the above step 3205 may specifically include steps 32051 to 32053.

[0112] Step 32051: Extract the service semantic information of each transaction field based on the field name of each transaction field in each initial transaction field group.

[0113] In this step, step 3204 uses a spectral clustering algorithm to divide the transaction fields in the structured dataset into M initial transaction field groups according to the number of groups M. Each initial transaction field group includes several transaction fields, each with its original field name, such as "transaction amount," "merchant number," "payment channel," and "return code." Based on this, the M initial transaction field groups are input into the semantic field classification agent. The semantic field classification agent first traverses each initial transaction field group, reads the field name of each transaction field within the group, and extracts the service semantic information to which the field belongs at the service level from the field name.

[0114] Taking payment transaction scenarios as an example, the semantic field classification agent can identify the following based on the field names: "Transaction Amount", "Transaction Date", and "Transaction Type" belong to the "Basic Transaction Information" service category; "Merchant ID", "Merchant Name", and "Merchant Category" belong to the "Merchant Information" service category; "Payment Channel" and "Channel Identifier" belong to the "Channel Information" service category; and "Return Code" and "Return Description" belong to the "Transaction Result Information" service category.

[0115] Since field names are the most direct and stable semantic carriers of transaction fields, in this embodiment, a semantic field classification agent can be used to extract service semantic information from field names to obtain the meaning of each transaction field at the service level and the service scope to which it belongs, providing a semantic basis for subsequent service rationality verification and correction of pure mathematical clustering results.

[0116] Step 32052: Based on the service semantic information of each transaction field, verify whether all transaction fields in each initial transaction field group belong to the same service, and obtain the verification result.

[0117] In this step, for each initial transaction field group, the semantic field classification agent compares and verifies the service semantic information of all transaction fields within the group to determine whether all transaction fields in the group belong to the same service category. If the service semantic information of all transaction fields in the group is consistent, such as all belonging to the service category of "basic transaction information," the verification passes, and the grouping result of the initial transaction field group is reasonable at the service semantic level, requiring no adjustment. If there are transaction fields in the group with inconsistent service semantic information, for example, most fields in an initial transaction field group belong to the "basic transaction information" service category, but a "merchant number" field belonging to the "merchant information" service category is mixed in, the verification fails, and there are fields in the group that need to be corrected.

[0118] In this embodiment, the spectral clustering algorithm groups data entirely based on statistical correlation, reflecting which fields are statistically frequently observed together or highly correlated. However, statistical correlation is not always equivalent to service correlation, even though they belong to completely different service dimensions in service logic. For example, "transaction amount" and "promotional identifier" may show statistical correlation within a specific time period, but the former belongs to transaction information, while the latter belongs to marketing activity information. This step introduces service semantic verification, placing the results of purely mathematical clustering within a service logic framework for review, ensuring that the grouping results are not only statistically reasonable but also semantically reasonable.

[0119] Step 32053: If the verification result indicates that there are transaction fields to be corrected that do not belong to the same service in each initial transaction field group, determine the target initial field group that matches the service semantic information of the transaction fields to be corrected in the M initial transaction field groups according to the service semantic information of the transaction fields to be corrected, and move the transaction fields to be corrected from the current initial field group to the target initial field group.

[0120] In this step, when the verification in step 32052 finds one or more transaction fields within an initial transaction field group that do not belong to the dominant service category of that group, the semantic field classification agent marks these fields as transaction fields to be corrected. Subsequently, the semantic field classification agent can search through all M initial transaction field groups based on the service semantic information of the transaction field to be corrected, to find a target initial field group that matches the service semantic information of the transaction field to be corrected. For example, suppose the "Merchant ID" field is grouped into the same initial transaction field group as "Transaction Amount" and "Transaction Date" by the spectral clustering algorithm due to statistical correlation, but the service semantic information of "Merchant ID" is of the "Merchant Information" category, while the service semantic information of other fields in the group is of the "Basic Transaction Information" category. The semantic field classification agent marks it as a field to be corrected, and based on the "merchant ID" service semantics of "merchant information", it finds the target initial field group with "merchant information" as the dominant service category in all M initial transaction field groups. This group already includes fields such as "merchant name" and "merchant category". Then, the "merchant ID" field is moved from the current "transaction basic information" group to the "merchant information" group.

[0121] In this way, this step achieves service semantic correction of the purely mathematical clustering results by extracting semantically mismatched fields from their initial field groups and placing them into field groups that match their service semantics. This operation is equivalent to a targeted fine-tuning of the spectral clustering output results. It does not overturn the overall grouping framework of spectral clustering, but rather makes local corrections based on it, so that the final data blocks achieve a reasonable state in both statistical correlation and service semantic consistency, thereby making the separation results more reasonable and in line with the actual service scenario.

[0122] It should be noted that after this closed-loop processing is completed, the semantic field classification agent outputs the corrected M initial transaction field groups as the final M data blocks. In the specific implementation, steps 32052 and 32053 can be repeated multiple times until there are no more transaction fields to be corrected in any of the initial transaction field groups, ensuring that the correction converges. This iterative correction logic ensures that the separation result not only captures the statistical correlation information between field contents but also supplements the service semantic information, making the final grouping result consistent in both statistical regularity and service common sense dimensions.

[0123] Therefore, by extracting the semantics of field services to verify the initial field group and migrate cross-service dimension fields, the semantic mismatch defects caused by relying solely on mathematical correlation are corrected, ensuring that fields within the same data block belong to the same payment service dimension, and eliminating the correlation distortion problem caused by mixed modeling of cross-semantic fields.

[0124] Regarding step 330, in this embodiment of the application, step 330 may specifically include steps 3301 to 3305, as shown below.

[0125] Step 3301: Based on the frequency of occurrence of the field values ​​of each transaction field in the structured dataset within each data block, the frequency of occurrence of each field value in the structured dataset is counted, and each field value is sorted in descending order of frequency to obtain a frequency sorting sequence.

[0126] Step 3302: Construct a frequency and ranking curve for field values ​​based on the frequency sorting sequence.

[0127] Step 3303: Calculate the curvature of the field value frequency and ranking curve based on the field value frequency and ranking curve. The curvature is the absolute value of the second derivative of the field value frequency and ranking curve.

[0128] In this step, the curvature of the smoothed curve is calculated. The curvature is taken as the absolute value of the second derivative of the curve. The discrete approximation formula (5) is used to quantify the degree of curvature of the curve and to locate the boundary position of the frequency interval switching.

[0129] in, Let be the curve curvature corresponding to the i-th sampling point, and r be the horizontal axis variable, representing the sorting position of the field value. This is the field frequency function after being smoothed by the Savitzky-Golay filter. The frequency value of the field after smoothing for the i-th sampling point. The smoothed frequency value of the sampling point adjacent to the left of the i-th sampling point. The smoothed frequency value is the value of the sampling point adjacent to the right of the i-th sampling point.

[0130] Step 3304: Select the local maximum point of curvature as the inflection point in the field value frequency and ranking curve; wherein, the inflection point includes at least one of the following: the boundary point between the high frequency band and the mid frequency band of curvature, and the boundary point between the mid frequency band and the low frequency band.

[0131] Among them, the curvature K calculated in step 3303 can be selected. i The local maximum point is used as the dividing point, i.e., the inflection point.

[0132] Step 3305: Determine the N data sub-blocks corresponding to each data block based on the number of inflection points.

[0133] In this step, the number of sub-blocks N can be adaptively determined based on the number of identified inflection points. That is, if there are no inflection points, the entire data block is treated as a single data sub-block; if there is a single inflection point, the data is divided into two parts by the inflection point; if there are two inflection points, the data is divided into three parts by the two inflection points. This automatically distinguishes high-frequency, mid-frequency, and low-frequency long-tail samples, thus completing the vertical hierarchical splitting.

[0134] Furthermore, in the embodiments of this application, step 3302 mentioned above may specifically include step 33021 and step 33022.

[0135] Step 33021: In a log-log coordinate system, generate an initial field value frequency and ranking curve with the sorting position of each field value as the horizontal axis and the occurrence frequency corresponding to each field value as the vertical axis.

[0136] In this step, the initial field value frequency versus ranking curve is plotted on a logarithmic coordinate system with the ranking position as the horizontal axis and the field value frequency as the vertical axis. This amplifies the distribution differences of sparse low-frequency samples and facilitates the identification of the boundary features between high-frequency, medium-frequency, and low-frequency intervals.

[0137] Step 33022: Smooth the initial field value frequency and ranking curve to obtain the field value frequency and ranking curve.

[0138] In this step, the Savitzky-Golay filter can be used to smooth the initial field value frequency and ranking curve according to the following formula (6) to obtain the field value frequency and ranking curve.

[0139] in, The convolution coefficients are... This is the length of the filtering window. The frequency value of the field after smoothing for the i-th sampling point. This is the original frequency of the sampling point with index i+j before the curve is filtered and smoothed, eliminating the interference of random noise on the curve shape and ensuring the accuracy of the curvature calculation results.

[0140] Furthermore, in the embodiments of this application, step 3305 mentioned above may specifically include: When the number of inflection points is zero, each data block is treated as a whole as a data sub-block; When there is only one inflection point, each data block is divided into two data sub-blocks using the inflection point as the dividing point. When there are two inflection points, each data block is divided into three data sub-blocks using the two inflection points as the dividing points.

[0141] Therefore, by statistically analyzing the frequency of field values ​​within a data block and generating an ordered frequency ranking sequence, a frequency ranking curve is constructed in a logarithmic coordinate system and smoothed and denoised using a Savitzky-Golay filter. Then, the absolute value of the second derivative of the curve is used as the curvature to extract local maxima of curvature as inflection points to distinguish between high-frequency, mid-frequency, and low-frequency samples. Finally, the data block is adaptively divided into corresponding numbers of data sub-blocks based on the number of inflection points. The entire process is driven by the distribution of the data itself, without the need for manually preset frequency thresholds. This can accurately distinguish between regular high-frequency transaction samples and scarce long-tail transaction samples, avoiding the problem of a large number of high-frequency transactions overwhelming a small number of long-tail transaction data, causing the model to lose the characteristics of long-tail scenarios. Independently splitting data sub-blocks at different frequency levels facilitates subsequent targeted modeling, effectively improving the coverage of simulation transaction data for various long-tail business scenarios and the overall simulation accuracy.

[0142] Regarding step 340, in this embodiment of the application, before step 340, the data processing method may further include steps 4101 to 4106, as shown below.

[0143] Step 4101: Obtain the training samples corresponding to the sample data sub-blocks. The training samples include multiple sample transaction fields corresponding to the sample data sub-blocks, sample field features corresponding to each sample transaction field, and sample field values ​​for each sample transaction field.

[0144] In this step, for the current data sub-block to be trained, the corresponding real data sub-block is extracted from the structured dataset as training samples. The training samples include multiple sample transaction fields corresponding to the data sub-block, such as transaction amount, merchant number, etc., sample field features corresponding to each sample transaction field, such as data type (numerical or categorical), and sample field values ​​for each sample transaction field, such as the specific transaction amount value, merchant number value, etc.

[0145] Here, this step ensures that a one-to-one mapping relationship is established between each data synthesizer and its corresponding data sub-block. The training samples of each data sub-block are different in terms of field composition and data distribution, so that the data synthesizers trained subsequently can accurately learn the local statistical features and field relationships of their respective data sub-blocks, realizing a divide-and-conquer training strategy.

[0146] Step 4102: Input the training samples into the sample data synthesizer corresponding to the sample data sub-blocks and perform iterative training. The sample data synthesizer includes a generator and a discriminator.

[0147] In this step, the training samples obtained in step 4101 are input into the CTGAN corresponding to the data sub-block for iterative training. The sample data synthesizer includes a generator and a discriminator. During model training, the CTGAN incorporates the semantics of transaction field names and their data types as key conditional information into the training process. Specifically, the generator not only receives random noise vectors as input but also concatenates encoded field names such as "transaction amount" and "merchant number" as conditional vectors into the input layer. This conditional mechanism enables the generator to learn the deep data distribution characteristics of specific fields under different business categories. During the adversarial game, the discriminator, in distinguishing the authenticity of data, is forced to focus on the intrinsic relationship between the conditional fields and the data content.

[0148] In this way, through this conditional training method, the network can accurately capture the marginal distribution and joint dependencies of each field in the transaction data, effectively avoiding the field misalignment or logical breakage problems commonly found in traditional GANs when generating tabular data. At the same time, each data sub-block has an independent data synthesizer, and the network parameters of the data synthesizers corresponding to different data sub-blocks do not affect each other, allowing each to learn the local patterns of its own data sub-block.

[0149] Step 4103: During the iterative training of the sample data synthesizer, record the loss value change curves corresponding to the generator and the loss value change curves corresponding to the discriminator.

[0150] In this step, during the iterative training of the adversarial network, the change curves of the generator loss and the discriminator loss are continuously recorded. Each loss curve records the loss value of the corresponding model in each iteration throughout the entire training cycle, reflecting the convergence state and training stability of the model.

[0151] Here, the loss value variation curve serves as the data source for calculating the oscillation exponent and convergence slope in subsequent steps. The morphological characteristics of the generator's loss value variation curve can reflect the stability and convergence speed of model training, providing a quantitative basis for the adaptive adjustment of hyperparameters.

[0152] Step 4104: After the number of training iterations reaches a preset number of rounds, the generator of the current generative adversarial network is used to generate sample simulation data sub-blocks, and the distribution coefficient and correlation coefficient between the sample simulation data sub-blocks and the training samples are calculated.

[0153] In this step, after a preset number of training iterations, a sample simulation data sub-block is generated using the generator of the current adversarial network. Subsequently, the distribution coefficient (Column Shapes Score) and correlation coefficient (Column Pair Trends Score) between this sample simulation data sub-block and the training samples obtained in step 4101 are calculated. The distribution coefficient measures the similarity between the sample simulation data sub-block and the training samples in the statistical distribution of a single field, while the correlation coefficient measures the similarity between the sample simulation data sub-block and the training samples in the pairwise correlation between fields. Both coefficients range from 0 to 1; the closer the value is to 1, the more consistent the statistical characteristics of the simulation data are with the real data.

[0154] Here, the distribution coefficient and correlation coefficient are quantitative indicators for evaluating the training quality of the data synthesizer. By statistically comparing the simulated data produced by the current generator with the real training data, we can objectively determine whether the generator has fully learned the essential laws of the real data, rather than simply memorizing the training samples.

[0155] Step 4105: If the distribution coefficient is less than the second threshold or the correlation coefficient is less than the third threshold, extract the oscillation index and convergence slope from the generator's loss value change curve.

[0156] In this step, it is determined whether the distribution coefficient calculated in step 4104 is greater than or equal to the second threshold, and whether the correlation coefficient is greater than or equal to the third threshold. If both are met, the current training result is deemed satisfactory, training is stopped, and step 4106 is executed to stop training, using the converged current generative adversarial network as the data synthesizer corresponding to the data sub-block. If the distribution coefficient is less than the second threshold or the correlation coefficient is less than the third threshold, i.e., either indicator is not met, the current training result is deemed unsatisfactory, and the oscillation index and convergence slope are extracted from the generator loss value change curve recorded in step 4103. The oscillation index is used to quantify the volatility of the generator loss value curve, and the convergence slope is used to quantify the rate of descent of the generator loss value curve.

[0157] In this way, the distribution coefficient and correlation coefficient measure simulation quality from two dimensions: single-field distribution and inter-field correlation, respectively. These two factors are independent and complementary. Failure of either index to meet the standard indicates that the generator has not yet fully learned the patterns of real data in that dimension, triggering the hyperparameter adjustments described below. Using OR logic instead of AND logic ensures that quality defects in any dimension can be detected and corrected promptly.

[0158] Step 4106: Based on the oscillation index and convergence slope, adjust the training hyperparameters of the generative adversarial network according to the preset mapping rules, and retrain the generative adversarial network using the adjusted training hyperparameters until the distribution coefficient is greater than or equal to the second threshold and the correlation coefficient is greater than or equal to the third threshold. Stop training and use the converged current generative adversarial network as the data synthesizer corresponding to the data sub-block.

[0159] In this step, based on the oscillation index and convergence slope extracted in step 4105, the training hyperparameters of the generative adversarial network are adjusted according to a preset mapping rule, and the adversarial network is retrained iteratively using the adjusted training hyperparameters. After retraining, the process returns to step 4104 to recalculate the distribution coefficient and correlation coefficient, and the threshold judgment is performed again. This process is repeated iteratively until the distribution coefficient is greater than or equal to the second threshold and the correlation coefficient is greater than or equal to the third threshold. At this point, training stops, and the converged current generative adversarial network is output as the data synthesizer corresponding to the data sub-block.

[0160] As can be seen, this step achieves independent adaptive optimization of the hyperparameters of the data synthesizer corresponding to each data sub-block. Since the data volume, data distribution, and field composition vary among different data sub-blocks, the applicable hyperparameter configurations also differ. By automatically adjusting the hyperparameters based on quantization rules according to the shape of the loss curve, the problem of underfitting in some models caused by all data blocks sharing a fixed set of hyperparameters, as in traditional solutions, is avoided. At the same time, the rule-based adjustment logic replaces repeated manual trial and error, significantly improving training efficiency and model quality.

[0161] Furthermore, step 4106 mentioned above may specifically include: If the oscillation index is greater than the fifth threshold, halve the learning rate in the training hyperparameters and increase the batch training sample size; if the oscillation index is less than or equal to the fifth threshold, increase the learning rate in the training hyperparameters and decrease the batch training sample size; if the convergence slope is less than the sixth threshold and the loss value has not stopped decreasing, increase the number of training epochs per iteration in the generative adversarial network; if the convergence slope is greater than or equal to the loss value and the loss value has stopped decreasing, reduce the number of training epochs per iteration in the generative adversarial network.

[0162] In this step, based on the preset mapping rules, the hyperparameters of the adversarial network training are adaptively adjusted according to the scenario by combining the two types of indicators, namely the oscillation index and the convergence slope, output by the generator loss value change curve, as shown below.

[0163] When the oscillation index is greater than the fifth threshold, such as 0.35, it indicates that the generator loss curve fluctuates wildly and the model training is unstable. Therefore, the learning rate can be halved, and the batch training sample size can be increased simultaneously to smooth the gradient update amplitude, alleviate the training oscillation problem, and improve the model training stability. When the oscillation index is less than or equal to the fifth threshold, it indicates that the loss curve fluctuates little and the model is generally stable. At this time, the learning rate can be increased and the batch training sample size can be reduced to accelerate the model's learning speed of the data sub-block distribution characteristics and improve training efficiency. When the convergence slope is less than the sixth threshold, such as 0.005, and the loss value continues to decrease, it indicates that the model converges slowly and has not yet fully learned the data distribution. Therefore, the number of training cycles per iteration can be increased to extend the training time and allow the model to fully fit the field distribution and correlation features of the current data sub-block. When the convergence slope is greater than or equal to the sixth threshold and the loss value no longer decreases, it indicates that the model has converged prematurely. Continuing training will waste computing power or even lead to overfitting. Therefore, the number of training cycles per iteration can be reduced to save processor training resources.

[0164] Therefore, based on the loss curve oscillation index and convergence slope, hyperparameters such as learning rate, batch size, and training period are adaptively adjusted according to different scenarios. The training configuration is precisely optimized for different training states such as model oscillation, slow convergence, and early convergence, which accelerates the convergence speed of the adversarial network, improves the fitting effect of the dedicated synthesizer for each data sub-block, and improves the matching accuracy between simulation data and real transaction distribution.

[0165] In addition, the data processing method may also include step 4107, where, for the generative adversarial network corresponding to the same sample data sub-block, if the cumulative number of retraining times is greater than or equal to the fourth threshold, the morphological features, distribution coefficient, and correlation coefficient of the generator's loss value change curve are input into the large language model to obtain feedback information output by the large language model. The feedback information includes at least one of the following: analysis of training failure reasons and generative adversarial network parameter adjustment scheme. The feedback information is used to trigger manual intervention to adjust the generative adversarial network.

[0166] This step addresses scenarios where multiple adaptive hyperparameter tuning and retraining attempts still fail to meet targets. When the cumulative retraining count for the generative adversarial network (GAN) corresponding to the same data sub-block reaches the fourth threshold, it indicates that automatic hyperparameter fine-tuning using preset mapping rules is no longer sufficient to achieve the desired model performance. At this point, the morphological characteristics of the generator loss curve, the distribution coefficients of the simulated data sub-blocks, and the correlation coefficients between the training samples are input into a large language model. This model, combined with domain-specific training knowledge, performs intelligent inference and outputs feedback information including an analysis of the training failure causes and a proposed adjustment scheme for the GAN parameters. Based on this feedback, prompts can be pushed to guide developers to manually modify network configurations, hyperparameters, and network structure, resolving complex training anomalies that the automatic hyperparameter tuning mechanism cannot handle.

[0167] Therefore, when the simulation performance still fails to meet the target after multiple adaptive parameter tunings of the adversarial network, intelligent analysis can be performed by combining a large language model with training data to automatically locate the root cause of the model training failure and output parameter optimization solutions. This reduces the threshold for manual debugging, quickly guides personnel to complete network optimization, and improves the parameter tuning efficiency of the data synthesizer and the final simulation effect.

[0168] In this embodiment of the application, step 350 may specifically include steps 3501 to 3503.

[0169] Step 3501: Using the anchor field corresponding to each data block as the association primary key, vertically merge the simulation data sub-blocks of all data sub-blocks belonging to the same data block to obtain the simulation data block corresponding to each data block.

[0170] In this step, the anchor field of each data block is pre-determined as the association primary key, and vertical row merging is performed on all simulation data sub-blocks under the same data block. High-frequency, medium-frequency and low-frequency simulation samples within the same data block are integrated into a complete simulation data block, completing the collection of simulation samples within a single data block and providing complete block-level simulation data for cross-block splicing.

[0171] Step 3502: Using the anchor field corresponding to each data block as the cross-block association primary key, the M simulation data blocks are horizontally concatenated to obtain the simulation structured dataset. The field structure of the simulation structured dataset is the same as that of the structured dataset.

[0172] In this step, the anchor field is reused as the unified association primary key across data blocks. All M simulated data blocks are horizontally concatenated to generate a simulated structured dataset with field arrangement and number that is completely consistent with the original structured dataset. This fully restores the correspondence between all fields in the real transaction and ensures that the field association logic is not lost.

[0173] Step 3503: Based on the field mapping information, convert the simulated structured dataset line by line into simulated transaction messages that are consistent with the format of real transaction messages.

[0174] In this step, the field mapping information generated in the previous preprocessing is called to reverse map and encapsulate the simulated structured dataset line by line, and the output format is converted to a simulated transaction message that completely matches the original real transaction message, so as to realize the standardized conversion of simulated structured data into usable business messages.

[0175] Therefore, by relying on the anchor field, the simulation data sub-blocks are vertically merged and multiple data blocks are horizontally spliced ​​to restore the simulation structured dataset with the same structure as the original structured dataset, and the original relationship between each field is completely preserved. Then, the field mapping information of the message field identifier and the transaction field is reverse-encapsulated to generate a standard simulation transaction message. This ensures that the sample matching is accurate and error-free when multiple sub-blocks and multiple data blocks are merged, and can directly output simulation transaction messages that can be used in business systems without additional secondary format conversion. It completely restores the structure and field mapping rules of the real transaction message and improves the usability of simulation data.

[0176] Furthermore, the embodiments of this application also include steps 3504 to 3506, determining the anchor field involved above.

[0177] Step 3504: For any current data block among the M data blocks, traverse each current transaction field within the current data block.

[0178] In this step, for any current data block among the M horizontally divided data blocks, each current transaction field included in the current data block is traversed, and each transaction field is taken as a candidate field to be evaluated in turn, completely covering all fields in the block without omitting any candidate objects, thus providing all calculation samples for the subsequent unified calculation of cross-group correlation indicators of each field.

[0179] Step 3505: Calculate the average statistical correlation between the current transaction field in the current data block and the transaction fields in the structured dataset other than the current data block, and use it as the cross-group average correlation corresponding to the current transaction field.

[0180] In this step, for a single current transaction field obtained in step 3504, all external transaction fields belonging to other data blocks in the structured dataset are obtained, and the statistical correlation between the current transaction field and each external cross-block transaction field is calculated in turn. The arithmetic mean of all statistical correlations is calculated, and the calculated mean is defined as the cross-group average correlation corresponding to the current transaction field. This indicator can quantify the overall linkage strength between the field and other global dimension fields, and accurately characterize the cross-group dependency ability of the field.

[0181] For example, horizontal partitioning results in two data blocks: Data block A (high-frequency merchant sample block): Transaction date, merchant ID, merchant type (fields within the block); Data block B (Payment transaction sample block): Payment channel, transaction amount, response code (cross-block external field).

[0182] Calculate the average cross-group correlation for the Merchant ID and Merchant Type fields in data block A. Calculate the pairwise statistical correlation between Merchant ID and the payment channel, transaction amount, and response code in data block B. Assuming the correlations for the three groups are 0.85, 0.78, and 0.82 respectively, the average correlation is (0.85 + 0.78 + 0.82) ÷ 3 = 0.817, meaning the average cross-group correlation for Merchant ID is 0.817. Calculate the pairwise statistical correlation between Merchant Type and the payment channel, transaction amount, and response code in data block B. The correlations for the three groups are 0.61, 0.43, and 0.50, respectively. The average correlation is (0.61 + 0.43 + 0.50) ÷ 3 = 0.513, meaning the average cross-group correlation for Merchant Type is 0.513.

[0183] Step 3506: Determine the transaction field with the highest average cross-group correlation within the current data block as the anchor field of the current data block.

[0184] In this step, the average cross-group correlation degree of each candidate transaction field in the current data block is compared, and the transaction field with the highest average cross-group correlation degree is selected and designated as the anchor field of the current data block. This anchor field has the strongest global cross-block linkage characteristic and can be used as a unified association primary key to build a complete field dependency chain across data blocks, thereby ensuring the global field association consistency of the simulated structured dataset after splicing and merging multiple simulation data blocks from the bottom layer.

[0185] For example, comparing the average cross-group correlation of all fields within data block A: Merchant ID = 0.817, Merchant Type = 0.513, Transaction Date = 0.402. Among them, Merchant ID has the highest average cross-group correlation value. Therefore, Merchant ID is determined as the anchor field of data block A.

[0186] Therefore, by calculating the average cross-group correlation degree between each transaction field within a data block and external cross-block fields, the field with the highest cross-group correlation degree is selected as the anchor field. The anchor field has global linkage characteristics. When using it as the association primary key to complete the splicing of simulation data sub-blocks and simulation data blocks, a cross-data block field dependency chain can be stably built, avoiding the confusion of field correspondence after the fusion of multiple simulation data blocks, completely preserving the global field association logic of real transactions, ensuring the consistency of association between the fused simulation structured dataset and the original data, and improving the data fidelity of the final simulation transaction message.

[0187] For ease of explanation, the above will be explained below. Figure 1 The interaction of data processing flows between various modules in the data processing system is explained.

[0188] First, the training phase of the data synthesizer.

[0189] Before the formal execution of simulated transaction message generation, the training of the data synthesizer corresponding to each data sub-block is completed through the collaborative work of the data training module and the active fine-tuning module.

[0190] Specifically, the data training module acquires the structured dataset output by the transaction preprocessing module and performs independent training on each data sub-block in the structured dataset: for each data sub-block, the corresponding real data sub-block is obtained as a training sample and input into the CTGAN generative adversarial network corresponding to that data sub-block for training. During training, the active fine-tuning module evaluates the training results in real time, calculating the distribution coefficient and correlation coefficient between the test synthetic data produced by the generator and the training samples; if the evaluation results are not up to standard, the active fine-tuning module adjusts the training hyperparameters of the CTGAN corresponding to that data sub-block according to the oscillation exponent and convergence slope of the generator loss curve, according to the preset mapping rules, including the learning rate, batch size, number of training epochs, etc., and triggers retraining. If the target is not met after multiple training iterations, the active fine-tuning module inputs the loss curve morphology, distribution coefficient, and correlation coefficient of the training process into the semantic analysis model. The model then outputs a cause analysis and adjustment plan for manual intervention. This process continues until the CTGAN training for all data sub-blocks reaches the target. Once the CTGAN training has reached the target, each CTGAN is saved as a data synthesizer for each data sub-block and can be used in subsequent data simulation stages.

[0191] Secondly, the transaction preprocessing stage.

[0192] The transaction preprocessing module loads the original real transaction messages and performs data cleaning, which involves removing incomplete transaction data and filling in null values ​​in some fields. The cleaned transaction messages are then broken down into multiple transaction fields, and the field features and values ​​for each field are extracted. The decomposed field values ​​are then converted line by line into a CSV file for each transaction record, resulting in a structured dataset. Simultaneously, based on the correspondence between the message field identifiers of the real transaction messages and the transaction fields in the CSV file, a field mapping template representing the bidirectional mapping relationship is generated. The transaction preprocessing module outputs the structured dataset to both the data segmentation module and the transaction transformation module, and outputs the field mapping template to the transaction transformation module.

[0193] The data segmentation module receives the structured dataset output by the transaction preprocessing module and sequentially performs horizontal field segmentation and vertical frequency band splitting. Specifically, the data segmentation module first triggers the horizontal field segmentation engine, which calculates the statistical correlation between any two transaction fields based on the data type indicated by the field features of each transaction field, constructs a graph adjacency matrix, and calculates the eigenvalue sequence of the Laplacian matrix; it calculates the difference between adjacent eigenvalues ​​and selects the index corresponding to the largest eigenvalue gap as the optimal number of groups M; based on M, it executes a spectral clustering algorithm to divide the transaction fields into M initial field groups; subsequently, it calls the semantic field classification agent to verify and correct the M initial field groups according to the service semantic information corresponding to the field names of each transaction field, and outputs M data blocks.

[0194] Suppose that the original structured dataset includes transaction fields (1), (2), (3), (4)... After horizontal field separation, M data blocks are output, such as data block A, data block B, etc., where data block A includes transaction fields (1) and (2); and data block B includes transaction fields (3) and (4).

[0195] For each data block, the data segmentation module triggers the vertical frequency band splitting engine. Specifically, it obtains the field values ​​of each transaction field in the structured dataset for that data block, counts the frequency of each field value, and sorts them in descending order; it plots a frequency versus ranking curve in a logarithmic coordinate system, smooths it using a Savitzky-Golay filter, and calculates the curvature; it selects local maxima of curvature as inflection points, and adaptively determines the frequency band splitting strategy based on the number of inflection points. For example, data block A is split vertically into data block A1 and data block A2; data block B is split vertically into data block B1 and data block B2.

[0196] The data segmentation module outputs the segmentation results, including M data blocks and N corresponding sub-blocks, to the data simulation module and the transaction conversion module. Simultaneously, the data segmentation module also filters anchor fields for each data block: it calculates the average statistical correlation between each transaction field in the current data block and all other transaction fields in the structured dataset that belong to the current data block, identifies the transaction field with the highest average cross-group correlation as the anchor field for that data block, and outputs the anchor field information to the transaction conversion module.

[0197] The data simulation module receives the data sub-blocks output by the data segmentation module and calls the pre-trained data synthesizers corresponding to each data sub-block. Specifically, for each data sub-block, the data simulation module loads the data synthesizer corresponding to that data sub-block from the saved data synthesizers, i.e., the CTGAN generator that has been independently trained and achieved the target during the training phase; it then uses this data synthesizer to generate simulation data for that data sub-block and outputs the corresponding simulation data sub-block. The data synthesizer corresponding to data block A1 generates simulation data sub-block A1', the data synthesizer corresponding to data block A2 generates simulation data sub-block A2', the data synthesizer corresponding to data block B1 generates simulation data sub-block B1', and the data synthesizer corresponding to data block B2 generates simulation data sub-block B2'.

[0198] The transaction conversion module receives the simulated data sub-blocks output by the data simulation module, the anchor field information output by the data segmentation module, and the field mapping template output by the transaction preprocessing module. Specifically, the transaction conversion module first uses the anchor field corresponding to each data block as the association primary key to vertically merge the simulated data sub-blocks of all data sub-blocks belonging to the same data block, obtaining the simulated data block corresponding to each data block. Data block A1' and data block A2' are vertically merged using the anchor field as the association primary key to obtain the simulated data block A' corresponding to data block A; data block B1' and data block B2' are vertically merged to obtain the simulated data block B' corresponding to data block B.

[0199] Subsequently, the transaction conversion module uses the anchor field as the cross-block association primary key to horizontally concatenate the M simulation data blocks, obtaining a complete simulation structured dataset. The field structure of this simulation structured dataset is the same as that of the original structured dataset. Simulation data block A' and simulation data block B' are then horizontally concatenated using the anchor field as the association primary key to restore the complete simulation structured dataset.

[0200] Finally, the transaction conversion module converts the simulated structured dataset line by line into simulated transaction messages that are consistent with the format of real transaction messages, according to the bidirectional mapping relationship between the message field identifiers and transaction fields recorded in the field mapping template, as the final output.

[0201] Based on the same inventive concept Figure 4 This is a schematic diagram of the structure of a data processing apparatus provided in one embodiment of this application.

[0202] In some embodiments of this application, Figure 4 The data processing device shown can be installed on a computer device.

[0203] like Figure 4 As shown, the data processing device 40 may specifically include: The acquisition module 401 is used to acquire the structured dataset and field mapping information of the real transaction message. The structured dataset includes multiple transaction fields obtained after the real transaction message is decomposed, the field features corresponding to each transaction field, and the field value of each transaction field. The field mapping information is used to record the bidirectional mapping relationship between the message field identifier of the real transaction message and the transaction fields of the structured dataset. The processing module 402 is used to perform horizontal field segmentation processing on the structured dataset based on the field features and service semantic information corresponding to each transaction field, to obtain M data blocks. Each data block includes at least two transaction fields, and the transaction fields in different data blocks do not overlap. The splitting module 403 is used to perform vertical frequency band splitting on each data block according to the frequency of occurrence of the field values ​​of each transaction field in the structured dataset, so as to obtain N data sub-blocks corresponding to each data block; The generation module 404 is used to generate the simulation data sub-blocks for each data sub-block through the data synthesizer corresponding to each data sub-block; The fusion module 405 is used to fuse the simulated data sub-blocks of all data sub-blocks according to the field mapping information to obtain the simulated transaction message corresponding to the real transaction message.

[0204] In this way, a structured dataset including complete transaction fields, field features, and field values ​​can be obtained by parsing real transaction messages. Horizontal field segmentation is then performed based on field features and service semantic information, grouping transaction fields that match service semantics and mathematical correlations into independent data blocks. This avoids the distortion of field relationships caused by mixed modeling of transaction fields from different service dimensions. Furthermore, by vertically segmenting each data block according to the frequency of field combinations, high-frequency regular transaction samples and low-frequency long-tail transaction samples are accurately distinguished, achieving differentiated, fine-grained data partitioning. Subsequently, a dedicated data synthesizer for each data sub-block independently generates corresponding simulation data sub-blocks, enabling targeted fitting of data distribution characteristics for regular and scarce long-tail transaction scenarios. This effectively compensates for the shortcomings of traditional simulation models in covering niche and low-frequency transaction scenarios. Finally, multiple simulation data sub-blocks are fused and restored using field mapping information, fully preserving the field association rules and message structure features of real transaction messages. This significantly improves the consistency of distribution, accuracy of field associations, and scenario adaptability between simulated and real transaction data, effectively solving the technical problems of incomplete scenario coverage and low simulation accuracy in existing simulated transaction data.

[0205] The detection device 40 in the embodiments of this application will be described in detail below.

[0206] In one or more optional embodiments, the data processing device 40 may further include a calculation module, which is used to calculate the statistical correlation between any two transaction fields in the structured dataset according to the data type indicated by the field feature of each transaction field, provided that the field features include at least a data type. The data processing device 40 may further include a determining module, used to determine the eigenvalue sequence of the Laplacian matrix corresponding to the graph adjacency matrix based on the graph adjacency matrix corresponding to the statistical correlation between any two transaction fields; wherein, the eigenvalue sequence includes eigenvalues ​​arranged in ascending order of numerical value, and the eigenvalues ​​are used to characterize the penalty value for each transaction field being partitioned into different data blocks with other transaction fields in the structured dataset, and the number of eigenvalues ​​in the eigenvalue sequence is the same as the number of transaction fields in the structured dataset; The determination module can also be used to determine the number of groups M based on the feature value sequence; The data processing device 40 may further include a partitioning module for dividing the transaction fields in the structured dataset into M initial transaction field groups according to the number of groups M, using a spectral clustering algorithm. The data processing device 40 may further include a verification and correction module, which verifies and corrects the M initial transaction field groups based on the service semantic information of each transaction field in each initial transaction field group, and determines the M initial transaction field groups after verification and correction as M data blocks.

[0207] In one or more optional embodiments, the data processing device 40 may further include a calculation module for calculating the statistical correlation between any two transaction fields in the structured dataset using the Pearson correlation coefficient when the transaction field is of numerical data type. When the data type of the transaction field is categorical, the Cramer correlation coefficient is used to calculate the statistical correlation between any two transaction fields in the structured dataset.

[0208] In one or more optional embodiments, the data processing device 40 may further include a construction module for constructing a graph adjacency matrix based on the statistical correlation between any two transaction fields. The graph adjacency matrix is ​​a square matrix of P rows and P columns, where P is the total number of transaction fields in the structured dataset. The off-diagonal elements of the graph adjacency matrix are the statistical correlation between any two transaction fields, and the diagonal elements of the graph adjacency matrix are a first threshold. The data processing device 40 may further include a determining module for determining a Laplacian matrix based on the graph adjacency matrix and the degree matrix corresponding to the graph adjacency matrix; wherein the degree matrix is ​​a diagonal matrix, and the diagonal elements of the diagonal matrix are the sum of the statistical correlation degrees between the first transaction field in the structured dataset and all second transaction fields in the structured dataset other than the first transaction field; the off-diagonal elements of the Laplacian matrix are the penalty values ​​for dividing the first transaction field and the second transaction field into different data blocks, and the diagonal elements of the Laplacian matrix are the diagonal elements of the degree matrix; The data processing device 40 may also include a decomposition module for performing eigenvalue decomposition on the Laplace matrix to obtain P eigenvalues. The data processing device 40 may further include an arrangement module for arranging the P feature values ​​in ascending order according to their numerical values ​​to obtain a feature value sequence.

[0209] In one or more optional embodiments, the data processing device 40 may further include a calculation module for calculating the difference between adjacent feature values ​​in the feature value sequence based on the feature value sequence, thereby obtaining multiple feature value gaps. The data processing device 40 may further include a determining module for selecting the sorting index value corresponding to the largest value among multiple feature value gaps, as the grouping number M.

[0210] In one or more optional embodiments, the data processing device 40 may further include an extraction module for extracting service semantic information of each transaction field based on the field name of each transaction field in each initial transaction field group; The data processing device 40 may also include a verification module, which is used to verify whether all transaction fields in each initial transaction field group belong to the same service based on the service semantic information of each transaction field, and obtain the verification result. The processing module 402 can also be used to, when the verification result indicates that there is a transaction field to be corrected that does not belong to the same service in each initial transaction field group, determine the target initial field group that matches the service semantic information of the transaction field to be corrected in the M initial transaction field groups according to the service semantic information of the transaction field to be corrected, and move the transaction field to be corrected from the current initial field group to the target initial field group.

[0211] In one or more optional embodiments, the data processing device 40 may further include a statistics module, which is used to count the frequency of occurrence of each field value in the structured dataset based on the frequency of occurrence of each field value in each transaction field in the structured dataset. The data processing device 40 may also include a sorting module for sorting each field value in descending order of frequency of occurrence to obtain a frequency sorting sequence. The data processing device 40 may also include a construction module for constructing a field value frequency versus ranking curve based on the frequency sorting sequence; The data processing device 40 may further include a calculation module for calculating the curvature of the field value frequency and ranking curve based on the field value frequency and ranking curve, wherein the curvature is the absolute value of the second derivative of the field value frequency and ranking curve. The data processing device 40 may further include a determining module for selecting local maxima of curvature as inflection points in the field value frequency and ranking curve; wherein, the inflection point includes at least one of the following: the boundary point between the high-frequency and mid-frequency bands of curvature, and the boundary point between the mid-frequency and low-frequency bands; The determination module is also used to determine the N data sub-blocks corresponding to each data block based on the number of inflection points.

[0212] In one or more optional embodiments, the generation module 404 can also be used to generate an initial field value frequency and ranking curve in a double logarithmic coordinate system, with the sorting position of each field value as the horizontal axis and the occurrence frequency corresponding to each field value as the vertical axis. The processing module 402 can also be used to smooth the initial field value frequency and ranking curve to obtain the field value frequency and ranking curve.

[0213] In one or more alternative embodiments, the data processing device 40 may further include a determining module for treating each data block as a whole as a data sub-block when the number of inflection points is zero. When there is only one inflection point, each data block is divided into two data sub-blocks using the inflection point as the dividing point. When there are two inflection points, each data block is divided into three data sub-blocks using the two inflection points as the dividing points.

[0214] In one or more optional embodiments, the data processing device 40 may further include a merging module, which is used to vertically merge the simulation data sub-blocks of all data sub-blocks belonging to the same data block using the anchor field corresponding to each data block as the association primary key, so as to obtain the simulation data block corresponding to each data block. The data processing device 40 may also include a splicing module, which is used to horizontally splice M simulation data blocks with the anchor field corresponding to each data block as the cross-block association primary key to obtain a simulation structured dataset. The field structure of the simulation structured dataset is the same as that of the structured dataset. The data processing device 40 may also include a conversion module for converting the simulated structured dataset line by line into simulated transaction messages that are consistent with the format of real transaction messages, based on field mapping information.

[0215] In one or more optional embodiments, the data processing device 40 may further include a traversal module for traversing each current transaction field within any current data block among the M data blocks; The data processing device 40 may further include a determining module, which is used to calculate the average statistical correlation between the current transaction field in the current data block and the transaction fields in the structured dataset other than the current data block, as the cross-group average correlation corresponding to the current transaction field; The determination module is also used to determine the transaction field with the highest average correlation across groups within the current data block as the anchor field of the current data block.

[0216] In one or more optional embodiments, the acquisition module 401 can also be used to acquire training samples corresponding to sample data sub-blocks, the training samples including multiple sample transaction fields corresponding to sample data sub-blocks, sample field features corresponding to each sample transaction field, and sample field values ​​for each sample transaction field. The data processing device 40 may further include a training module for iteratively training a sample data synthesizer corresponding to a sample data sub-block by inputting training samples into the sample data synthesizer. The sample data synthesizer includes a generator and a discriminator. The data processing device 40 may also include a recording module for recording the loss value change curves corresponding to the generator and the loss value change curves corresponding to the discriminator during the iterative training of the sample data synthesizer. The processing module 402 can also be used to generate sample simulation data sub-blocks using the generator of the current generative adversarial network after the number of training iterations reaches a preset number of rounds, and to calculate the distribution coefficient and correlation coefficient between the sample simulation data sub-blocks and the training samples. The data processing device 40 may also include an extraction module for extracting the oscillation index and convergence slope in the loss value change curve of the generator when the distribution coefficient is less than the second threshold or the correlation coefficient is less than the third threshold. The training module can also be used to adjust the training hyperparameters of the generative adversarial network according to the oscillation index and convergence slope and a preset mapping rule, and then retrain the generative adversarial network using the adjusted training hyperparameters until the distribution coefficient is greater than or equal to the second threshold and the correlation coefficient is greater than or equal to the third threshold, at which point training stops, and the converged current generative adversarial network is used as the data synthesizer corresponding to the data sub-block.

[0217] In one or more optional embodiments, the processing module 402 can also be used to input the morphological features, distribution coefficient and correlation coefficient of the generator's loss value change curve into the large language model when the cumulative number of retraining times for the generative adversarial network corresponding to the same sample data sub-block is greater than or equal to the fourth threshold, and obtain the feedback information output by the large language model. The feedback information includes at least one of the following: training failure cause analysis and generative adversarial network parameter adjustment scheme. The feedback information is used to trigger manual intervention to adjust the generative adversarial network.

[0218] In one or more alternative embodiments, the processing module 402 may also be used to halve the learning rate in the training hyperparameters and increase the batch training sample size when the oscillation index is greater than the fifth threshold. If the oscillation index is less than or equal to the fifth threshold, increase the learning rate in the training hyperparameters and decrease the batch training sample size. If the convergence slope is less than the sixth threshold and the loss value does not stop decreasing, increase the number of training cycles per iteration in the anti-network. If the convergence slope is greater than or equal to the loss value and the loss value has stopped decreasing, reduce the number of training cycles per iteration in the generative adversarial network.

[0219] In one or more optional embodiments, the processing module 402 can also be used to perform data cleaning processing on the real transaction message to obtain the cleaned transaction message. The splitting module 403 can also be used to split the cleaned transaction message into multiple transaction fields; The processing module 402 can also be used to extract the field features and field values ​​corresponding to each transaction field, and convert the field values ​​of each transaction field into a CSV format file line by line according to each transaction record to obtain a structured dataset; The generation module 404 can also be used to generate field mapping information that represents the bidirectional mapping relationship, based on the correspondence between the message field identifier of the real transaction message and the transaction fields in the structured dataset.

[0220] Based on the same inventive concept, this application also provides a computer device. (Specifically combined with...) Figure 5 Please provide a detailed explanation.

[0221] Figure 5 This is a schematic diagram of the structure of a computer device provided in one embodiment of this application.

[0222] like Figure 5 As shown, the computer device may include a processor 501 and a memory 502 storing computer program instructions.

[0223] Specifically, the processor 501 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0224] Specifically, the processor 501 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0225] Memory 502 may include a large-capacity memory for data or instructions. For example, and not limitingly, memory 502 may include a hard disk drive (HDD), a floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 502 may include removable or non-removable (or fixed) media. Where appropriate, memory 502 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 502 is a non-volatile solid-state memory. In a particular embodiment, memory 502 includes read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0226] The processor 501 implements any of the data processing methods described in the above embodiments by reading and executing computer program instructions stored in the memory 502.

[0227] In one example, computer device 50 may also include a communication interface 503 and a bus 510. Wherein, as... Figure 5 As shown, the processor 501, memory 502, and communication interface 503 are connected through bus 510 and complete communication with each other.

[0228] The communication interface 503 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0229] Bus 510 includes hardware, software, or both, that couples components of a flow control device together. For example, and not limited to, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count Bus (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VESA Local Bus, VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 510 may include one or more buses. Although specific buses are described and illustrated in the embodiments of this application, this application considers any suitable bus or interconnection.

[0230] The computer device can execute the content detection method described in the embodiments of this application, thereby achieving a combination of Figure 3 The data processing methods and apparatus described.

[0231] Furthermore, in conjunction with the content detection methods in the above embodiments, this application embodiment can provide a computer-readable storage medium for implementation. This computer-readable storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the content detection methods in the above embodiments.

[0232] Furthermore, in conjunction with the content detection methods in the above embodiments, this application embodiment can provide a computer program product for implementation. This program product is stored in a storage medium and may specifically include a computer program or instructions. When executed by a processor, the computer program or instructions implement any of the content detection methods in the above embodiments. This program product is executed by at least one processor to implement the various processes of the content detection method embodiments described above, and can achieve the same technical effects. To avoid repetition, further details are omitted here.

[0233] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0234] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0235] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0236] The above are merely specific embodiments of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A data processing method, characterized in that, include: Obtain a structured dataset and field mapping information of a real transaction message. The structured dataset includes multiple transaction fields obtained after disassembling the real transaction message, field features corresponding to each transaction field, and field values ​​of each transaction field. The field mapping information is used to record the bidirectional mapping relationship between the message field identifier of the real transaction message and the transaction fields of the structured dataset. Based on the field features and service semantic information corresponding to each transaction field, the structured dataset is horizontally segmented to obtain M data blocks. Each data block includes at least two transaction fields, and the transaction fields in different data blocks do not overlap. Based on the frequency of occurrence of the field values ​​of each transaction field within each data block in the structured dataset, each data block is vertically segmented into frequency bands to obtain N data sub-blocks corresponding to each data block; Simulated data sub-blocks for each of the data sub-blocks are generated using a data synthesizer corresponding to each of the data sub-blocks; Based on the field mapping information, the simulated data sub-blocks of all data sub-blocks are merged to obtain the simulated transaction message corresponding to the real transaction message.

2. The method according to claim 1, characterized in that, The field features include at least a data type; based on the field features and service semantic information corresponding to each transaction field, the structured dataset is horizontally segmented to obtain M data blocks, including: Calculate the statistical correlation between any two transaction fields in the structured dataset based on the data type indicated by the field characteristics of each transaction field. Based on the graph adjacency matrix corresponding to the statistical correlation between any two transaction fields, determine the eigenvalue sequence of the Laplacian matrix corresponding to the graph adjacency matrix; wherein, the eigenvalue sequence includes eigenvalues ​​arranged in ascending order of numerical value, the eigenvalues ​​are used to characterize the penalty value for each transaction field being partitioned into different data blocks with other transaction fields in the structured dataset, and the number of eigenvalues ​​in the eigenvalue sequence is the same as the number of transaction fields in the structured dataset; Based on the feature value sequence, determine the number of groups M; According to the number of groups M, the transaction fields in the structured dataset are divided into M initial transaction field groups using a spectral clustering algorithm; Based on the service semantic information of each transaction field in each initial transaction field group, the M initial transaction field groups are verified and corrected, and the verified and corrected M initial transaction field groups are determined as the M data blocks.

3. The method according to claim 2, characterized in that, The step of calculating the statistical correlation between any two transaction fields in the structured dataset based on the data type indicated by the field characteristics of each transaction field includes: When the data type of the transaction field is numerical, the Pearson correlation coefficient is used to calculate the statistical correlation between any two transaction fields in the structured dataset; When the data type of the transaction field is categorical, the Cramer correlation coefficient is used to calculate the statistical correlation between any two transaction fields in the structured dataset.

4. The method according to claim 2, characterized in that, The step of determining the eigenvalue sequence of the Laplacian matrix corresponding to the graph adjacency matrix based on the graph adjacency matrix corresponding to the statistical correlation between any two transaction fields includes: Based on the statistical correlation between any two transaction fields, a graph adjacency matrix is ​​constructed. The graph adjacency matrix is ​​a square matrix of P rows and P columns, where P is the total number of transaction fields in the structured dataset. The off-diagonal elements of the graph adjacency matrix represent the statistical correlation between any two transaction fields, and the diagonal elements of the graph adjacency matrix represent a first threshold. The Laplacian matrix is ​​determined based on the graph adjacency matrix and the degree matrix corresponding to the graph adjacency matrix; wherein, the degree matrix is ​​a diagonal matrix, and the diagonal elements of the diagonal matrix are the sum of the statistical correlation degrees between the first transaction field in the structured dataset and all second transaction fields in the structured dataset other than the first transaction field; the off-diagonal elements of the Laplacian matrix are the penalty values ​​for dividing the first transaction field and the second transaction field into different data blocks, and the diagonal elements of the Laplacian matrix are the diagonal elements of the degree matrix; The Laplacian matrix is ​​subjected to eigenvalue decomposition to obtain P eigenvalues; The P feature values ​​are sorted in ascending order according to their numerical values ​​to obtain the feature value sequence.

5. The method according to claim 2, characterized in that, Determining the number of groups M based on the feature value sequence includes: Based on the feature value sequence, the difference between adjacent feature values ​​in the feature value sequence is calculated to obtain multiple feature value gaps; The sorting index value corresponding to the largest value among the multiple feature value gaps is selected as the grouping number M.

6. The method according to claim 2, characterized in that, The step of verifying and correcting the M initial transaction field groups based on the service semantic information of each transaction field in each initial transaction field group to obtain the M data blocks includes: Based on the field name of each transaction field in each initial transaction field group, extract the service semantic information of each transaction field; Based on the service semantic information of each transaction field, the verification results are obtained by checking whether all transaction fields in each initial transaction field group belong to the same service. If the verification result indicates that there is a transaction field to be corrected that does not belong to the same service in each of the initial transaction field groups, a target initial field group that matches the service semantic information of the transaction field to be corrected is determined from the M initial transaction field groups, and the transaction field to be corrected is moved from the current initial field group to the target initial field group.

7. The method according to claim 1, characterized in that, The step of vertically splitting each data block into N data sub-blocks based on the frequency of occurrence of the field values ​​of each transaction field within the structured dataset, to obtain N data sub-blocks corresponding to each data block, includes: Based on the frequency of occurrence of the field values ​​of each transaction field in each data block in the structured dataset, the frequency of occurrence of each field value in the structured dataset is calculated, and each field value is sorted in descending order of frequency to obtain a frequency sorting sequence. Based on the frequency sorting sequence, construct a field value frequency versus ranking curve; Based on the frequency and ranking curve of the field value, calculate the curvature of the frequency and ranking curve of the field value, where the curvature is the absolute value of the second derivative of the frequency and ranking curve of the field value. The local maxima of the curvature are selected as inflection points in the field value frequency and ranking curve; wherein, the inflection point includes at least one of the following: the boundary point between the high frequency band and the mid frequency band of the curvature, and the boundary point between the mid frequency band and the low frequency band; Based on the number of inflection points, determine the N data sub-blocks corresponding to each data block.

8. The method according to claim 7, characterized in that, The step of constructing a field value frequency versus ranking curve based on the frequency sorting sequence includes: In a double logarithmic coordinate system, an initial field value frequency and ranking curve is generated with the sorting position of each field value as the horizontal axis and the occurrence frequency corresponding to each field value as the vertical axis. The initial field value frequency and ranking curve are smoothed to obtain the field value frequency and ranking curve.

9. The method according to claim 7, characterized in that, The step of determining the N data sub-blocks corresponding to each data block based on the number of inflection points includes: When the number of inflection points is zero, each data block is treated as a whole as a data sub-block; When there is only one inflection point, each data block is divided into two data sub-blocks using the inflection point as the dividing point. When there are two inflection points, each data block is divided into three data sub-blocks using the two inflection points as the dividing points.

10. The method according to claim 1, characterized in that, The step of fusing the simulated data sub-blocks of all data sub-blocks according to the field mapping information to obtain the simulated transaction message corresponding to the real transaction message includes: Using the anchor field corresponding to each data block as the association primary key, the simulation data sub-blocks of all data sub-blocks belonging to the same data block are vertically merged to obtain the simulation data block corresponding to each data block. Using the anchor field corresponding to each data block as the cross-block association primary key, M simulation data blocks are horizontally concatenated to obtain a simulation structured dataset. The field structure of the simulation structured dataset is the same as that of the structured dataset. Based on the field mapping information, the simulated structured dataset is converted line by line into a simulated transaction message that is consistent with the format of the real transaction message.

11. The method according to claim 10, characterized in that, The method further includes: For any one of the M data blocks, iterate through each current transaction field within the current data block; Calculate the average statistical correlation between the current transaction field in the current data block and the transaction fields in the structured dataset other than the current data block, and use it as the cross-group average correlation corresponding to the current transaction field; The transaction field with the highest average cross-group correlation within the current data block is determined as the anchor field of the current data block.

12. The method according to claim 1, characterized in that, The method further includes: Obtain training samples corresponding to sample data sub-blocks. The training samples include multiple sample transaction fields corresponding to the sample data sub-blocks, sample field features corresponding to each sample transaction field, and sample field values ​​for each sample transaction field. The training samples are input into a sample data synthesizer corresponding to the sample data sub-block for iterative training. The sample data synthesizer includes a generator and a discriminator. During the iterative training of the sample data synthesizer, the loss value change curves corresponding to the generator and the loss value change curves corresponding to the discriminator are recorded. After the training iterations reach a preset number of rounds, the generator of the current generative adversarial network generates sample simulation data sub-blocks, and calculates the distribution coefficient and correlation coefficient between the sample simulation data sub-blocks and the training samples. If the distribution coefficient is less than the second threshold or the correlation coefficient is less than the third threshold, extract the oscillation index and convergence slope from the loss value change curve of the generator. Based on the oscillation index and the convergence slope, the training hyperparameters of the generative adversarial network are adjusted according to a preset mapping rule. The generative adversarial network is then retrained using the adjusted training hyperparameters until the distribution coefficient is greater than or equal to the second threshold and the correlation coefficient is greater than or equal to the third threshold. Training is then stopped, and the converged current generative adversarial network is used as the data synthesizer corresponding to the data sub-block.

13. The method according to claim 12, characterized in that, The method further includes: When the cumulative number of retraining times for the generative adversarial network corresponding to the same sample data sub-block is greater than or equal to the fourth threshold, the morphological features of the generator's loss value change curve, the distribution coefficient, and the correlation coefficient are input into the large language model to obtain the feedback information output by the large language model. The feedback information includes at least one of the following: training failure cause analysis and generative adversarial network parameter adjustment scheme. The feedback information is used to trigger manual intervention to adjust the generative adversarial network.

14. The method according to claim 12, characterized in that, The step of adjusting the training hyperparameters of the generative adversarial network according to the oscillation index and the convergence slope and a preset mapping rule includes: If the oscillation index is greater than the fifth threshold, the learning rate in the training hyperparameters is halved and the batch training sample size is increased. If the oscillation index is less than or equal to the fifth threshold, the learning rate in the training hyperparameters is increased and the batch training sample size is decreased. If the convergence slope is less than the sixth threshold and the loss value does not stop decreasing, increase the number of training cycles per iteration in the anti-network. If the convergence slope is greater than or equal to and the loss value has stopped decreasing, reduce the number of training epochs per iteration in the generative adversarial network.

15. The method according to claim 1, characterized in that, Before obtaining the structured dataset and field mapping information of the real transaction message, the method further includes: The real transaction message is cleaned to obtain the cleaned transaction message. The cleaned transaction message is broken down into multiple transaction fields; Extract the field features and field values ​​corresponding to each of the transaction fields, and convert the field values ​​of each transaction field into a CSV format file line by line according to each transaction record to obtain the structured dataset; Based on the correspondence between the message field identifier of the real transaction message and the transaction fields in the structured dataset, field mapping information is generated to represent the bidirectional mapping relationship.

16. A data processing apparatus, characterized in that, include: The acquisition module is used to acquire a structured dataset and field mapping information of a real transaction message. The structured dataset includes multiple transaction fields obtained after disassembling the real transaction message, field features corresponding to each transaction field, and field values ​​of each transaction field. The field mapping information is used to record the bidirectional mapping relationship between the message field identifier of the real transaction message and the transaction fields of the structured dataset. The processing module is used to perform horizontal field segmentation processing on the structured dataset based on the field features and service semantic information corresponding to each transaction field, to obtain M data blocks. Each data block includes at least two transaction fields, and the transaction fields in different data blocks do not overlap. The splitting module is used to perform vertical frequency band splitting on each data block according to the frequency of occurrence of the field values ​​of each transaction field in the structured dataset, so as to obtain N data sub-blocks corresponding to each data block; A generation module is used to generate a simulation data sub-block for each of the data sub-blocks through a data synthesizer corresponding to each of the data sub-blocks; The fusion module is used to fuse the simulated data sub-blocks of all data sub-blocks according to the field mapping information to obtain the simulated transaction message corresponding to the real transaction message.

17. A computer device, characterized in that, The computer device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the data processing method as described in any one of claims 1-15.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the data processing method as described in any one of claims 1-15.

19. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the data processing method as described in any one of claims 1-15.

Images