Simulation data generation method and device, equipment, storage medium and program product

Abstract

The invention discloses a simulation data generation method and device, equipment, a storage medium and a program product, and relates to the technical field of data processing, and the method comprises the steps: carrying out the iterative training of sample data through a pre-created rule recommendation model, and obtaining an optimal rule recommendation model corresponding to the sample data; the optimal rule recommendation model is dynamically optimized through a feedback optimization algorithm, a target rule recommendation model is obtained, and the target rule recommendation model is used for obtaining recommended data generation rules; when the target rule recommendation model cannot hit the data generation rule for recommendation, generating a recommended data generation rule through a model recommendation supplementary algorithm; and generating simulation data corresponding to the sample data based on the recommended data generation rule. The recommended data generation rule is generated from the sample data through the target rule recommendation model and the model recommendation supplementary algorithm, and the simulation data for testing is generated based on the data generation rule, so that the generation efficiency of the simulation data is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, equipment, storage medium and program product for generating simulation data. Background Art

[0002] With the development of computer network and software technology, especially the access of mobile Internet devices in recent years, higher requirements have been put forward for the testing of software systems. It is not only necessary to meet user needs in terms of functions and performance, but also to ensure compatibility and stability on different devices, platforms and environments. In this case, it is very important to test the system in all aspects and from multiple angles before the application goes online. In order to ensure the reliability of the test conclusion, it is necessary to ensure the diversity of business data and the total amount of data in the testing phase. Therefore, how to simply and quickly identify the data generation rules of test data and generate data is one of the problems to be solved at present. In addition, the simulation data required in the financial testing field is heterogeneous, large in quantity, complex in rules, and has many repeated scenarios. The existing number generation methods and systems are difficult to adapt to the simulation data generation scenarios in the financial testing field. How to meet the needs of efficient preparation of simulation data in the financial testing field has become an urgent problem for enterprises. The pain points of this problem are mainly: for data construction methods for different test scenarios, the current manual number generation, code number generation, and third-party number generation tools require users to configure or even implement the data rules for number generation. When there are many data fields, the workload of rule configuration is large and the efficiency is low.

[0003] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of the present application is to provide a method, device, equipment, storage medium and program product for generating simulation data, aiming to solve the technical problem of improving the efficiency of generating simulation data for testing.

[0005] To achieve the above purpose, the present application proposes a method for generating simulation data, the method comprising:

[0006] Iteratively train the sample data through the pre-created rule recommendation model to obtain the best rule recommendation model corresponding to the sample data;

[0007] Dynamically optimizing the optimal rule recommendation model through a feedback optimization algorithm to obtain a target rule recommendation model, wherein the target rule recommendation model is used to obtain a recommended data generation rule;

[0008] When the target rule recommendation model cannot match the data generation rule for recommendation, generating the recommended data generation rule through the model recommendation supplement algorithm;

[0009] Based on the recommended data generation rules, simulation data corresponding to the sample data is generated.

[0010] In one embodiment, the step of iteratively training the sample data through a pre-created rule recommendation model to obtain the best rule recommendation model corresponding to the sample data includes:

[0011] Obtain sample data from a data source, and perform balanced sampling on the sample data after desensitization;

[0012] Perform feature extraction on the sample data obtained after sampling to obtain data features of the sample data;

[0013] Determine a data label of the sample data, establish a mapping relationship between the data feature and the data label, and obtain a feature vector of the sample data;

[0014] The pre-created rule recommendation model is trained and updated based on the feature vector to obtain the best rule recommendation model corresponding to the sample data.

[0015] In one embodiment, the step of dynamically optimizing the optimal rule recommendation model through a feedback optimization algorithm to obtain a target rule recommendation model includes:

[0016] Obtaining recommended data generation rules based on the optimal rule recommendation model;

[0017] Obtain the data generation rules actually used by users;

[0018] Compare the recommended data generation rules with the actually used data generation rules, and calculate the correlation coefficient of the recommended data generation rules;

[0019] Calculate the feature weights of the data generation rules according to the preset weight division of the correlation coefficient;

[0020] Based on the feature values ​​of the rule similarity, the optimal rule recommendation model is adjusted and optimized to obtain a target rule recommendation model.

[0021] In one embodiment, the step of generating a recommended data generation rule by using a model recommendation supplement algorithm includes:

[0022] The recommended data generation rules are generated by at least one of a historical rule approximate matching algorithm, a constant filling algorithm, an expression language description, and a data canvas dynamic algorithm.

[0023] In one embodiment, the step of generating a recommended data generation rule by using a historical rule approximate matching algorithm includes:

[0024] Get the field name of the sample data;

[0025] Obtain historical data with the most similar field names to the sample data through the string similarity matching algorithm;

[0026] The data generation rule of the historical data is determined according to the most similar historical data, and the data generation rule of the sample data is obtained.

[0027] In one embodiment, the step of generating the recommended data generation rules by the data canvas dynamic algorithm includes:

[0028] Pre-set field rules in the data canvas;

[0029] Encapsulate the field rule as a custom function;

[0030] Describing the custom functions in a unified manner through an expression language;

[0031] The custom function described in the expression language is parsed by the syntax analyzer to obtain the recommended data generation rules.

[0032] In one embodiment, the step of generating simulation data corresponding to the sample data based on the recommended data generation rule includes:

[0033] Set the time and frequency of generating simulation data through the built-in timing unit;

[0034] Based on the recommended data generation rules, simulation data is generated under the scheduling of the data generation task thread;

[0035] The simulation data is sent to a preset sending target for simulation testing.

[0036] In addition, to achieve the above purpose, the present application also proposes a device for generating simulation data, the device for generating simulation data comprising:

[0037] The model training module is used to iteratively train the sample data through the pre-created rule recommendation model to obtain the best rule recommendation model corresponding to the sample data;

[0038] A model optimization module, used to dynamically optimize the optimal rule recommendation model through a feedback optimization algorithm to obtain a target rule recommendation model, wherein the target rule recommendation model is used to obtain a recommended data generation rule;

[0039] A supplementary recommendation module, used for generating a recommended data generation rule by using a model recommendation supplementary algorithm when the target rule recommendation model fails to match the data generation rule for recommendation;

[0040] The data generation module is used to generate simulation data corresponding to the sample data based on the recommended data generation rules.

[0041] In addition, to achieve the above-mentioned purpose, the present application also proposes a device for generating simulation data, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the method for generating simulation data as described above.

[0042] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the method for generating simulation data as described above are implemented.

[0043] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the method for generating simulation data as described above are implemented.

[0044] One or more technical solutions proposed in this application have at least the following technical effects:

[0045] The embodiment of the present application proposes a method, apparatus, device, storage medium and program product for generating simulation data, which iteratively trains sample data through a pre-created rule recommendation model to obtain an optimal rule recommendation model corresponding to the sample data; dynamically optimizes the optimal rule recommendation model through a feedback optimization algorithm to obtain a target rule recommendation model, and the target rule recommendation model is used to obtain a recommended data generation rule; when the target rule recommendation model cannot hit the data generation rule for recommendation, a recommended data generation rule is generated through a model recommendation supplement algorithm; based on the recommended data generation rule, simulation data corresponding to the sample data is generated. The present application generates recommended data generation rules for sample data through at least one of a target rule recommendation model, a historical rule approximate matching algorithm, a constant filling algorithm, an expression language description, and a data canvas dynamic algorithm, generates simulation data for testing based on the data generation rules, and improves the generation efficiency of simulation data. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0048] Figure 1 A schematic diagram of a flow chart provided for the first embodiment of the method for generating simulation data of the present application;

[0049] Figure 2 A schematic diagram of the module structure of a device for generating simulation data according to an embodiment of the present application;

[0050] Figure 3 A schematic diagram of the device structure of the hardware operating environment involved in the method for generating simulation data in an embodiment of the present application.

[0051] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0052] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0053] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0054] The main solution of the embodiment of the present application is: iteratively train the sample data through a pre-created rule recommendation model to obtain the optimal rule recommendation model corresponding to the sample data; dynamically optimize the optimal rule recommendation model through a feedback optimization algorithm to obtain a target rule recommendation model, and the target rule recommendation model is used to obtain a recommended data generation rule; when the target rule recommendation model cannot hit the data generation rule for recommendation, generate a recommended data generation rule through a model recommendation supplementary algorithm; based on the recommended data generation rule, generate simulation data corresponding to the sample data.

[0055] In this embodiment, for the convenience of description, the following description is made by taking the generation device for identifying simulation data as the execution subject.

[0056] In order to ensure the reliability of the test conclusions in the system test, the test phase must ensure the diversity of business data and the total amount of data. Therefore, how to simply and quickly identify the data generation rules of the test data and generate data is one of the problems to be solved. In addition, the simulation data required in the financial testing field is heterogeneous, large in quantity, complex in rules, and has many repeated scenarios. The existing number generation methods and systems are difficult to adapt to the simulation data generation scenarios in the financial testing field. How to meet the needs of efficient preparation of simulation data in the financial testing field has become an urgent problem for enterprises. The pain points of this problem are mainly: for data construction methods in different test scenarios, the current manual number generation, code number generation, and third-party number generation tools all require users to configure or even implement the data rules for number generation. When there are many data fields, the workload of rule configuration is large and the efficiency is low.

[0057] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a simulation data generation device, etc. The following takes the simulation data generation device as an example to illustrate this embodiment and the following embodiments.

[0058] Based on this, the present application embodiment provides a method for generating simulation data, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the method for generating simulation data of the present application.

[0059] In this embodiment, the method for generating simulation data includes steps S10 to S40:

[0060] Step S10, iteratively training the sample data through a pre-created rule recommendation model to obtain an optimal rule recommendation model corresponding to the sample data.

[0061] It should be noted that the sample data refers to the real data in the financial field in the database of the software system. The sample data can be passed in from the system front end, can be manually input, or can be directly configured with the data source connection and imported from the existing data source. For example, the data source can be a relational database such as MySQL, Gauss, or a non-relational database such as Elasticsearch, a message middleware such as Kafka, or an OLTP system such as ClickHouse.

[0062] In addition, it should be noted that the pre-created rule recommendation model is used to obtain recommended data generation rules based on sample data. The rule recommendation model can be a linear model, a decision tree, a random forest, a deep neural network, etc.

[0063] In addition, it should be noted that the optimal rule recommendation model is obtained by iteratively training and updating the rule recommendation model through sample data. During the model training and updating process, the value of the target loss function will gradually decrease. When the value of the target loss function is the smallest, the rule recommendation model obtained is the optimal rule recommendation model.

[0064] Specifically, a rule recommendation model that needs to be trained is determined, sample data is obtained from a data source, the sample data is desensitized and balancedly sampled, feature extraction is performed on the sample data obtained after sampling to obtain data features of the sample data, data labels of the sample data are determined, a mapping relationship is established based on the data features and the data labels to obtain a feature vector of the sample data, and a pre-created rule recommendation model is trained and updated based on the feature vector to obtain the optimal rule recommendation model corresponding to the sample data.

[0065] Step S20, dynamically optimizing the optimal rule recommendation model through a feedback optimization algorithm to obtain a target rule recommendation model, wherein the target rule recommendation model is used to obtain a recommended data generation rule.

[0066] It should be noted that the feedback optimization algorithm is used to optimize the best data rule recommendation model according to the feedback of the data generation rules used by the user, so that the model gradually approaches the actual needs of the user in the subsequent data generation rule recommendations.

[0067] In addition, it should be noted that the target rule recommendation model refers to a model obtained by dynamically optimizing the optimal rule recommendation model, and is used to obtain the recommended data generation rules.

[0068] In addition, it should be noted that the data generation rules refer to a series of standards and specifications that need to be followed in the process of defining and constraining simulation data generation. These rules ensure that the generated data not only conforms to business logic and industry standards, but also can simulate financial transactions or scenarios in the real world.

[0069] Data generation rules can involve various aspects of data, such as: data format, which is used to specify the type (such as numeric type, string type, date type, etc.) and format (such as currency format, date format, etc.) of data; data range, which is used to define the value range of data, such as the balance of an account cannot be negative, or the amount of a transaction must be within a specific range; data relationship, which is used to specify the dependency and consistency between data, such as the recipient account of a transaction must exist in the account list, or the account balances of the two parties to the transaction must meet a certain logical relationship before and after the transaction; business logic, which is used to set rules according to the specific logic of financial business, such as the rate calculation of different types of transactions, special processing procedures at specific time points, etc.; data consistency, which is used to ensure that the generated data is coherent and consistent as a whole, and there will be no logical contradictions or conflicts; security, which follows the relevant rules of data protection and security, such as desensitizing sensitive data to prevent the leakage of personal information or commercial secrets.

[0070] Specifically, based on the optimal rule recommendation model, the recommended data generation rules are obtained, the data generation rules actually used by the user are obtained, the recommended data generation rules are compared with the data generation rules actually used, the correlation coefficient of the recommended data generation rules is calculated, and the feature weights of the data generation rules are calculated according to the preset weight division of the correlation coefficients. Based on the feature values ​​of the rule similarity, the optimal rule recommendation model is adjusted and optimized to obtain the target rule recommendation model.

[0071] Step S30: when the target rule recommendation model cannot match the data generation rule for recommendation, a recommended data generation rule is generated by a model recommendation supplement algorithm.

[0072] It should be noted that the model recommendation supplement algorithm is used to generate data generation rules when the target rule recommendation model cannot hit the data generation rules used for recommendation. The step of generating data generation rules through the model recommendation supplement algorithm includes generating recommended data generation rules through at least one of the historical rule approximate matching algorithm, constant filling algorithm, expression language description, and data canvas dynamic algorithm.

[0073] Specifically, when the target rule recommendation model cannot hit the data generation rule for recommendation, the recommended data generation rule is generated by the model recommendation supplement algorithm, and the model recommendation supplement algorithm includes but is not limited to the historical rule approximate matching algorithm, the constant filling algorithm, the expression language description, and the data canvas dynamic algorithm.

[0074] Step S40: Generate simulation data corresponding to the sample data based on the recommended data generation rule.

[0075] It should be noted that the simulation data refers to a large amount of data that meets specific requirements generated through simulation. These data are used to simulate scenarios in real environments so as to comprehensively test, evaluate or verify the system.

[0076] Specifically, the time and frequency of generating simulation data are set through a built-in timing unit, and based on the recommended data generation rules, simulation data is generated under the scheduling of the data generation task thread, and the simulation data is sent to a preset sending target for simulation testing.

[0077] Through the above scheme, this embodiment can improve the precision and accuracy of the data generation rules recommended by the rule recommendation model by iteratively training the rule recommendation model; by dynamically optimizing the best rule recommendation model, the model can continuously adapt to data changes and new requirements; by generating recommended data generation rules through the target rule recommendation model and the model recommendation supplementary algorithm for sample data, and generating simulation data for testing based on the data generation rules, the generation efficiency of simulation data is improved, and the data generation rules generated based on the sample data can improve the authenticity and credibility of the simulation data.

[0078] Based on the above implementation scheme, in a feasible implementation manner, the iterative training of the sample data through the pre-created rule recommendation model to obtain the best rule recommendation model corresponding to the sample data includes steps S11 to S14:

[0079] Step S11, obtaining sample data from a data source, and performing balanced sampling on the sample data after desensitization.

[0080] It should be noted that the data source refers to a device for obtaining sample data, which can be a system front end for inputting sample data, or a database storing sample data. For example, it can be a relational database such as MySQL, Gauss, or a non-relational database such as Elasticsearch, a message middleware such as Kafka, or an OLTP system such as ClickHouse.

[0081] Specifically, sample data is obtained from the data source, and the sample data is desensitized. Desensitization refers to processing sensitive information in the sample data to protect the privacy and security of the data, while ensuring that the processed data can still be used in specific scenarios (such as testing, analysis, etc.). Specifically, the desensitization process involves deforming or replacing sensitive data so that it maintains a certain data structure and characteristics without leaking the original sensitive information. Then the desensitized sample data is evenly sampled; some frequently used data in the field of financial testing are sorted and classified, and representative data from each type of data is selected for balanced sampling. Since the amount of some data is very large, random sampling or stratified sampling can be used to reduce the amount of sampled data while maintaining data representativeness.

[0082] More specifically, the database connection operation module manages various database links input by users. The module is called when data is input, written to the target, and executed in a custom language. On this basis, the operation of various heterogeneous data sources is uniformly encapsulated. This module provides a consistent calling method by uniformly encapsulating the operation interfaces of various heterogeneous data sources, greatly simplifying the development complexity of upper-level applications.

[0083] Step S12, performing feature extraction on the sample data obtained after sampling to obtain data features of the sample data.

[0084] It should be noted that the data features are used to describe and analyze the attributes or variables of the data. In financial data analysis, features are those data attributes that are considered to have an important impact on the prediction target.

[0085] Specifically, the sampled data is preprocessed. First, missing values ​​are processed. For abnormal values ​​such as null values ​​and empty values ​​that may exist in the database, a deletion strategy is adopted. Secondly, duplicate data is also removed to avoid deviations in model training. At the same time, data with unclear features are converted or normalized to avoid negative impacts on model training.

[0086] Then, feature extraction is performed on the preprocessed data. First, determine the specific prediction target, such as predicting stock prices, credit default risks, trading volume, etc. Then identify the features that have an important impact on the prediction target. Statistical analysis, visualization and other means can be used to identify features, generate features through aggregation, combination, derivation and other methods, and select the features most relevant to the prediction target to obtain data features.

[0087] For example, Table 1 below shows some of the extracted data features and their specific meanings and values:

[0088] Feature Number Feature Name Value Feature 1 Is all data 0 / 1 Feature 2 String length 1-n Feature 3 Does it include Chinese? 0 / 1 Feature 4 Whether to include time 0 / 1 Feature 5 Is it an enumeration field? 0 / 1 Feature 6 Chinese characters ratio 0-n Feature 7 Digital share 0-n Feature 8 Letter ratio 0-n Feature 9 Special characters ratio 0-n

[0089] Table 1

[0090] In Table 1, feature 1 is "whether all are data", and the value is 0 or 1; when the value is 0, it means that feature 1 is not all data, and the data feature of feature 1 is 0; when the value is 1, it means that feature 1 is all data, and the data feature of feature 1 is 1. Similarly, the specific meanings of the values ​​of feature 3, feature 4, and feature 5 can be inferred; feature 2 in the table is "string length", and "1-n" indicates the range of the string length. When the value is 5, it means that the string length of feature 3 is 5, and then the data feature of feature 3 is 5. Similarly, the specific meanings of the values ​​of feature 6, feature 7, feature 8, and feature 9 can be inferred.

[0091] Step S13, determining the data label of the sample data, establishing a mapping relationship between the data feature and the data label, and obtaining a feature vector of the sample data.

[0092] It should be noted that the data label refers to the classification or rule of data features. Each example sample in the sample data is mapped to a specific label (rule) through a series of features (such as whether it is all numbers, string length, whether it contains Chinese characters, etc.). This label represents the classification or attribute of the sample.

[0093] In addition, it should be noted that the feature vector is a numerical array, in which each numerical value represents the performance of a data point on a certain feature. For example, the feature "whether it is all numbers" is converted into a binary value (0 or 1), and the "length" is converted into a specific numerical value, and so on. For ease of understanding, examples of the numerical value range of the feature vector can be referred to Table 1.

[0094] Specifically, first determine the data labels in the sample data, and then establish a clear mapping relationship between each data feature and data label. You can create a table that can accurately convert the data features of each data point into corresponding data labels or codes; then convert the data features into feature vectors. If the data labels are not direct numerical or coded forms (such as text labels or specific symbols), you need to encode the labels. Common encoding methods include One-Hot Encoding, Label Encoding, etc. The encoded labels can be directly used in the model training and prediction process; combine the encoded feature vectors of all sample data and the corresponding data labels to form the final data set, which will be used for subsequent model training and evaluation.

[0095] Step S14: The pre-created rule recommendation model is trained and updated based on the feature vector to obtain the best rule recommendation model corresponding to the sample data.

[0096] Specifically, the rule recommendation model can be a linear model, decision tree, random forest, deep neural network, etc. Taking the XGBoost algorithm in the decision tree as an example of a rule recommendation model, the target loss function of the model is:

[0097]

[0098] Among them, Obj (t) is the value of the target loss function at the tth iteration; l is the loss function, which is used to calculate the error of a single sample; y t is the value of the true target value at the tth iteration; is the predicted value of sample i in the t-1th iteration; f t (x i ) is the prediction contribution of the newly added model to sample i in the tth iteration; Ω(f t ) is the newly added model f t The regularization term of the complexity is used to prevent overfitting; cons is a constant used to adjust the influence of the tangent function; tant is the tangent function used to adjust the complexity or learning rate of the model.

[0099] During the model training process, a greedy algorithm is used to gradually build the model. This means that the algorithm will make the best choice at each step, hoping to achieve the global optimum through such local optimal choices. In each iteration, the algorithm will learn a new model. Model update is divided into two main steps: leaf node splitting (LeafSplit) and leaf node weight (LeafWeight) update.

[0100] For example, for leaf node j, its weight w j It can be calculated by the following formula:

[0101]

[0102] Among them, w j is the weight of leaf node j; G j is the sum of the first-order derivatives (gradients) of all samples on leaf node j, indicating the sensitivity of the loss function to the predicted value; H j is the sum of the second-order derivatives of all samples on leaf node j, which represents the curvature of the loss function; λ is a regularization parameter, which is used to control the update amplitude of leaf node weights to prevent overfitting of the model.

[0103] Through continuous iteration, the model will continuously update the splits and weights of leaf nodes to gradually reduce the target loss function. This process will continue until a certain stopping condition is reached (such as the loss function no longer significantly decreases, the preset number of iterations is reached, etc.). Finally, the algorithm will output the model that performs best based on the sample data and obtain the best rule recommendation model.

[0104] Through the above scheme, this embodiment desensitizes the sample data, effectively protects the sensitive information in the data, ensures the compliance and security of data use, and reduces the risk of data leakage; balanced sampling technology helps to solve the problem of uneven data distribution, avoids deviations caused by data skew during model training, and improves the quality of training data and the generalization ability of the model.

[0105] Based on the above implementation scheme, in a feasible implementation manner, the dynamically optimizing the optimal rule recommendation model by the feedback optimization algorithm to obtain the target rule recommendation model includes steps S21 to S25:

[0106] Step S21, obtaining a recommended data generation rule based on the optimal rule recommendation model.

[0107] Specifically, the recommended data generation rules are obtained according to the training data through the optimal rule recommendation model.

[0108] Step S22, obtaining the data generation rules actually used by the user.

[0109] Specifically, the data generation rules actually used by users can be obtained from user usage feedback or from historical data.

[0110] Step S23, comparing the recommended data generation rule with the actually used data generation rule, and calculating the correlation coefficient of the recommended data generation rule.

[0111] Specifically, the data generation rules recommended by the model are compared with the data generation rules actually used by the user, and the rules of each field in the data generation task are divided according to the similarity, and they are judged as Good, Bad, and None, and quantitative weights are assigned to obtain correlation coefficients. In one embodiment of the present application, the expression for assigning quantitative weights is:

[0112]

[0113] Among them, Cov j is the correlation coefficient. When the correlation coefficient is 1, it means that the data generation rules recommended by the model are the same as the data generation rules actually used by the user. When the correlation coefficient is 0, it means that the data generation rules recommended by the model are related to the data generation rules actually used by the user but are not completely the same. When the correlation coefficient is -1, it means that the data generation rules recommended by the model are related to the data generation rules actually used by the user but are completely different.

[0114] Step S24, calculating the feature weights of the data generation rules according to the preset weight division of the correlation coefficient.

[0115] Specifically, the feature weight is calculated according to the expression of the correlation coefficient quantization weight in step S23, and the calculation formula of the feature weight is:

[0116]

[0117] Among them, the correlation coefficient Cov j It reflects whether the rule type of a field is correctly identified, and the feature weight W j It reflects the matching related situation for specific data generation tasks.

[0118] Step S25, based on the feature values ​​of the rule similarity, the optimal rule recommendation model is adjusted and optimized to obtain a target rule recommendation model.

[0119] Specifically, for data and rules with low correlation, the system uses updated data to readjust the training model, and then gradually approaches the actual needs of users in the subsequent generation of number-making rules.

[0120] Through the above scheme, this embodiment compares the recommended data generation rules with the data generation rules actually used, and then calculates the feature weights of the recommended data generation rules. The model is dynamically optimized through feedback from actual data, which can solve the problem of overfitting or underfitting caused by the sample data used by the optimal rule-driven model not covering all real data.

[0121] Based on the above implementation scheme, in a feasible implementation manner, the data generation rule for generating recommendations by the model recommendation supplement algorithm includes step S31:

[0122] Step S31, generating a recommended data generation rule by using at least one of a history rule approximate matching algorithm, a constant filling algorithm, an expression language description, and a data canvas dynamic algorithm.

[0123] It should be noted that the historical rule approximate matching algorithm is used to determine the recommended data generation rule through a string similarity matching algorithm according to the field name of the sample data and the historical data rule of the data source.

[0124] In addition, it should be noted that the constant filling algorithm refers to filling the values ​​in the sample data as constants into the expression, generating a constant expression, and obtaining a customized data generation rule. In the process of feature extraction through the rule recommendation model, the extracted features are also selected as fields for calculating string similarity in the historical rule approximate matching algorithm, and can be filled into the expression as constants in the constant filling algorithm to obtain a constant expression.

[0125] More specifically, the recommended data generation rules are generated through the constant filling algorithm. The values ​​in the sample data are filled into the expression as constants. For the data in the sample data that are hit in both the best rule promotion model and the historical rule approximate matching algorithm, the field values ​​are selected as constants to fill into the expression as a more customized data generation rule; if there are multiple field values, the value with the highest frequency is selected to generate a constant expression to obtain the data generation rule.

[0126] In addition, it should be noted that the expression language description refers to the description of data generation rules through expression language. Expression language is a language or syntax used to represent and manipulate data in code. It is usually lighter than a complete programming language and focuses on task-specific functions such as data manipulation, conditional judgment, and loops. Expression language is widely used in scenarios such as template engines, configuration management, and business rule engines to flexibly describe and process data without writing complex code.

[0127] More specifically, the data generation rules for the recommendation are generated by describing the expression language. The data generation rules are described using the expression language; different expression languages ​​can be used for description here, or the expression language can be customized. The expression language used can support various operations and provide basic data processing operation functions such as built-in functions. In one embodiment of the present application, Java Web EL (an expression language widely used in Java Web applications) is used as the expression language to describe the data generation rules. For example, a field in the user data needs to be assembled based on the values ​​of other fields, which can be described using the syntax in the following Table 2:

[0128]

[0129] Table 2

[0130] In addition, it should be noted that the data canvas dynamic algorithm is used to preset field rules in the data canvas, encapsulate the field rules into custom functions, and then uniformly describe the custom functions through expression language, and parse the description through the syntax analyzer to obtain recommended data generation rules. It is often used for constants, enumerations, and when the expression language is insufficient.

[0131] Specifically, the recommended data generation rules are generated by at least one of the historical rule approximate matching algorithm, the constant filling algorithm, the expression language description, and the data canvas dynamic algorithm. In one embodiment of the present application, the data generation rules are first generated by the target rule recommendation model; when the recommended data generation rules cannot be hit, the data generation algorithm is matched by the historical rule approximate matching algorithm; when the recommended data generation rules still cannot be hit, the data generation rules are customized by the constant filling algorithm or the expression language description to obtain the recommended data generation rules; when the recommended data generation rules still cannot be hit, for example, when the constant, enumeration, and expression language expression ability are insufficient, the recommended data generation rules are generated by the data canvas dynamic algorithm.

[0132] Through the above scheme, this embodiment generates recommended data generation rules through at least one of the historical rule approximate matching algorithm, constant filling algorithm, expression language description, and data canvas dynamic algorithm, which can supplement the optimal rule recommendation model, cover scenarios that sample data cannot cover, and alleviate the problem of overfitting or underfitting.

[0133] Based on the above implementation scheme, in a feasible implementation manner, the method of generating recommended data generation rules by using the historical rule approximate matching algorithm includes steps S311 to S313:

[0134] Step S311, obtaining the field name of the sample data.

[0135] It should be noted that the field names of the sample data are features extracted by a feature extraction algorithm, and are fields with significant influence in the sample data.

[0136] Specifically, the field names in the sample data are obtained through word segmentation and feature extraction algorithms.

[0137] Step S312: Obtain historical data that is most similar to the field name of the sample data through a string similarity matching algorithm.

[0138] Specifically, after obtaining the field names of the sample data, the data source of the selected historical data is comprehensively considered, and the historical data with a high degree of matching with the field names in the sample data is calculated through a string similarity matching algorithm. In one embodiment of the present application, the calculation formula of the string similarity matching algorithm is:

[0139]

[0140] Among them, d J (s, t) is the Jaro distance between two strings s and t; l is the number of common characters in the two strings; m is the length of the shorter of the two strings s and t; p is the prefix weight of the field name in the sample data; d J W(s, t) is the adjusted Jaro distance, which takes into account the influence of Jaro distance and prefix matching. If l>0, that is, the two strings have at least one common character, then the similarity will be calculated as d J (s, t) is multiplied by a factor that takes into account the number of common characters and the weight of the prefix. If l = 0, that is, the two strings have no common characters, then d J W(s,t) is 0, which means they are completely dissimilar.

[0141] Step S313, determining the data generation rule of the historical data according to the most similar historical data, and obtaining the data generation rule of the sample data.

[0142] Specifically, after determining the historical data that is most similar to the sample data, the data generation rules of the historical data can be obtained, and the data generation rules of the most similar historical data are used as recommended data generation rules. The historical rule approximate matching algorithm considers both the matching and order of characters and the prefix matching of strings, and has a wider range of applications. For the same data source and the field name approximate matching degree within 0.9 to 1, matching recommendations are made based on the historical data rules.

[0143] Based on the above implementation scheme, in a feasible implementation manner, the data generation rules for generating recommendations through the data canvas dynamic algorithm include steps S341 to S344:

[0144] Step S341, pre-set field rules in the data canvas.

[0145] Specifically, the following functions are pre-built into the data canvas as fragments of field rules. Some scenarios can be preset through the data canvas, for example: 1. Complex rules that cannot be described using expression language; 2. Relying on external seed data; 3. Relying on the syntax provided by the data source for calculations, such as SQL syntax. You can use the corresponding category processing provided by the data canvas to design field fragments.

[0146] Step S342: encapsulate the field rule into a user-defined function.

[0147] Specifically, encapsulate the preset field rules into custom functions

[0148] Step S343: uniformly describe the user-defined functions through an expression language.

[0149] Specifically, custom functions are uniformly described through expression language, functions of various semantic grammars are uniformly described, and delivered to the parser for rule parsing.

[0150] For example, Table 3 below shows some semantic syntax functions and their usage:

[0151]

[0152] Table 3

[0153] Step S344, the user-defined function described in the expression language is parsed by a syntax analyzer to obtain a recommended data generation rule.

[0154] It should be noted that the syntax analyzer is an abstract syntax tree-based syntax analyzer, which is responsible for converting input code or rule strings into a structured representation form, such as an abstract syntax tree.

[0155] Specifically, the custom function described in the expression language is parsed by the syntax analyzer to obtain the recommended data generation rules. First, the specific syntax of the specific segment node is parsed through the abstract syntax tree to identify each specific semantic node:

[0156]

[0157] Among them, u is the semantic node currently being visited. For each child node of node u, the Anal function is recursively called, which is implemented by depth-first search to ensure that each child node is visited and analyzed.

[0158] Then, starting from the root node u, recursively perform abstract syntax analysis on all semantic nodes:

[0159]

[0160] Among them, u is the semantic node currently being visited, v is its child semantic node, and DFS-Anal is a recursive function used to perform depth-first abstract syntax analysis on all semantic nodes starting from the root node u.

[0161] Based on the above implementation scheme, in a feasible implementation manner, the generating of simulation data corresponding to the sample data based on the recommended data generation rule includes steps S41 to S43:

[0162] Step S41, setting the time and frequency of generating simulation data through the built-in timing unit.

[0163] It should be noted that the built-in timing unit is used to set the time and frequency of simulation data generation, which can be set through a timer.

[0164] Specifically, the time and frequency of generating simulation data are set through the built-in timing unit, and simulation data is generated at the time and frequency required by the user; at the same time, a data insurance function is provided, and the data insurance function is realized through the data monitoring unit. For example, when the simulation data consumption reaches a certain level, the simulation data generation is automatically triggered; the data distribution unit performs centralized scheduling according to the load situation of the simulation data generation unit to ensure that the number-generating tasks can be started and completed at the expected time.

[0165] Step S42, based on the recommended data generation rules, generate simulation data under the scheduling of the data generation task thread.

[0166] Specifically, the task thread pool dynamically maintains several data generation task threads. The task threads calculate the amount of data to be constructed according to the parameters provided by the scheduling module, adopt the recommended data generation rules, and clarify the data rules based on the rule generation logic and rule editing logic. After verifying that the data rules are correct, the timing of simulation data generation and the data target landing point are determined to ensure that high-quality and compliant simulation data are generated within a limited time. Batch processing and memory management technologies are used to further improve the speed and efficiency of data generation. Through these measures, the system can achieve rapid generation and distribution of large-scale data while ensuring data quality.

[0167] Step S43, sending the simulation data to a preset sending target for simulation testing.

[0168] It should be noted that the preset sending target is a system or device that needs simulation data for testing, and the receiving target can be a file, a relational database, a system or a device.

[0169] Specifically, after the data generation is completed, according to the pre-configured parameters and settings, the module can send the data to a variety of different target storage systems. These target storage systems can be file systems, or various types of relational and non-relational databases, such as MySQL, Oracle, MongoDB, etc. In this way, the system can flexibly send data in different environments and scenarios. Determine the target system or device to which the simulation data will be sent, including its network address, port number, receiving interface, etc., ensure that the target system or device is correctly configured and in a receivable state, and transmit the simulation data to the preset sending target through the selected transmission method for simulation testing, and generate a test report.

[0170] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the method for generating simulation data of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0171] This application also provides a device for generating simulation data, please refer to Figure 2 , the simulation data generating device comprises:

[0172] The model training module 201 is used to iteratively train the sample data through a pre-created rule recommendation model to obtain the best rule recommendation model corresponding to the sample data;

[0173] A model optimization module 202, configured to dynamically optimize the optimal rule recommendation model through a feedback optimization algorithm to obtain a target rule recommendation model, wherein the target rule recommendation model is used to obtain a recommended data generation rule;

[0174] A supplementary recommendation module 203, used for generating a recommended data generation rule by using a model recommendation supplementary algorithm when the target rule recommendation model fails to match the data generation rule for recommendation;

[0175] The data generation module 204 is used to generate simulation data corresponding to the sample data based on the recommended data generation rules.

[0176] The device for generating simulation data provided by the present application adopts the method for generating simulation data in the above-mentioned embodiment, and can solve the technical problem of improving the efficiency of generating simulation data for testing. Compared with the prior art, the beneficial effects of the device for generating simulation data provided by the present application are the same as the beneficial effects of the method for generating simulation data provided by the above-mentioned embodiment, and other technical features in the device for generating simulation data are the same as the features disclosed in the above-mentioned embodiment method, and are not described in detail here.

[0177] The present application provides a simulation data generation device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the simulation data generation method in the above-mentioned embodiment one.

[0178] Reference below Figure 3 , which shows a schematic diagram of a structure of a device for generating simulation data suitable for implementing the embodiment of the present application. The device for generating simulation data in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 3 The simulation data generation device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0179] like Figure 3As shown, the simulation data generation device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the simulation data generation device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the simulation data generation device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows the simulation data generation device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0180] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0181] The simulation data generation device provided by the present application adopts the simulation data generation method in the above embodiment, which can solve the technical problem of improving the efficiency of generating simulation data for testing. Compared with the prior art, the beneficial effects of the simulation data generation device provided by the present application are the same as the beneficial effects of the simulation data generation method provided by the above embodiment, and other technical features in the simulation data generation device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0182] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0183] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0184] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the method for generating simulation data in the above-mentioned embodiment.

[0185] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0186] The computer-readable storage medium may be included in the device for generating simulation data; or may exist independently without being assembled into the device for generating simulation data.

[0187] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the simulation data generation device, the simulation data generation device: iteratively trains the sample data through a pre-created rule recommendation model to obtain the optimal rule recommendation model corresponding to the sample data; dynamically optimizes the optimal rule recommendation model through a feedback optimization algorithm to obtain a target rule recommendation model, and the target rule recommendation model is used to obtain a recommended data generation rule; when the target rule recommendation model cannot hit the data generation rule for recommendation, generates a recommended data generation rule through a model recommendation supplementary algorithm; based on the recommended data generation rule, generates simulation data corresponding to the sample data.

[0188] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0189] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0190] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0191] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned method for generating simulation data, and can solve the technical problem of improving the efficiency of generating simulation data for testing. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the method for generating simulation data provided by the above-mentioned embodiment, and will not be repeated here.

[0192] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned method for generating simulation data when executed by a processor.

[0193] The computer program product provided by the present application can solve the technical problem of improving the efficiency of generating simulation data for testing. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the method for generating simulation data provided by the above embodiment, which will not be repeated here.

[0194] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A method for generating simulation data, characterized in that: The method comprises: Iteratively train the sample data through the pre-created rule recommendation model to obtain the best rule recommendation model corresponding to the sample data; Dynamically optimizing the optimal rule recommendation model through a feedback optimization algorithm to obtain a target rule recommendation model, wherein the target rule recommendation model is used to obtain a recommended data generation rule; When the target rule recommendation model cannot match the data generation rule for recommendation, generating the recommended data generation rule through the model recommendation supplement algorithm; Based on the recommended data generation rules, simulation data corresponding to the sample data is generated.

2. The method according to claim 1, characterized in that The step of iteratively training the sample data through a pre-created rule recommendation model to obtain the best rule recommendation model corresponding to the sample data includes: Obtain sample data from a data source, and perform balanced sampling on the sample data after desensitization; Perform feature extraction on the sample data obtained after sampling to obtain data features of the sample data; Determine a data label of the sample data, establish a mapping relationship between the data feature and the data label, and obtain a feature vector of the sample data; The pre-created rule recommendation model is trained and updated based on the feature vector to obtain the best rule recommendation model corresponding to the sample data.

3. The method according to claim 1, characterized in that The step of dynamically optimizing the optimal rule recommendation model by a feedback optimization algorithm to obtain a target rule recommendation model comprises: Obtaining recommended data generation rules based on the optimal rule recommendation model; Obtain the data generation rules actually used by users; Compare the recommended data generation rules with the actually used data generation rules, and calculate the correlation coefficient of the recommended data generation rules; Calculate the feature weights of the data generation rules according to the preset weight division of the correlation coefficient; Based on the feature values ​​of the rule similarity, the optimal rule recommendation model is adjusted and optimized to obtain a target rule recommendation model.

4. The method according to claim 1, characterized in that The step of generating the recommended data generation rules by the model recommendation supplement algorithm comprises: The recommended data generation rules are generated by at least one of a historical rule approximate matching algorithm, a constant filling algorithm, an expression language description, and a data canvas dynamic algorithm.

5. The method according to claim 4, characterized in that The step of generating the recommended data generation rules by using the historical rule approximate matching algorithm includes: Get the field name of the sample data; Use the string similarity matching algorithm to obtain the historical data that is most similar to the field name of the sample data; The data generation rule of the historical data is determined according to the most similar historical data, and the data generation rule of the sample data is obtained.

6. The method according to claim 4, characterized in that The step of generating the recommended data generation rules through the data canvas dynamic algorithm includes: Pre-set field rules in the data canvas; Encapsulate the field rule as a custom function; Describing the custom functions in a unified manner through an expression language; The custom function described in the expression language is parsed by the syntax analyzer to obtain the recommended data generation rules.

7. The method according to claim 1, characterized in that The step of generating simulation data corresponding to the sample data based on the recommended data generation rule comprises: Set the time and frequency of generating simulation data through the built-in timing unit; Based on the recommended data generation rules, simulation data is generated under the scheduling of the data generation task thread; The simulation data is sent to a preset sending target for simulation testing.

8. A device for generating simulation data, characterized in that: The device comprises: The model training module is used to iteratively train the sample data through the pre-created rule recommendation model to obtain the best rule recommendation model corresponding to the sample data; A model optimization module, used to dynamically optimize the optimal rule recommendation model through a feedback optimization algorithm to obtain a target rule recommendation model, wherein the target rule recommendation model is used to obtain a recommended data generation rule; A supplementary recommendation module, used for generating a recommended data generation rule by using a model recommendation supplementary algorithm when the target rule recommendation model fails to match the data generation rule for recommendation; The data generation module is used to generate simulation data corresponding to the sample data based on the recommended data generation rules.

9. A device for generating simulation data, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the method for generating simulation data according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the method for generating simulation data according to any one of claims 1 to 7 are implemented.

11. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the method for generating simulation data according to any one of claims 1 to 7 are implemented.

Images