Data generation method, device and system

Abstract

The invention relates to the technical field of computers, and provides a data generation method, device and system.The method comprises the steps that task identification of at least one feature collection task is obtained, the feature collection task indicates data features of data in a collection database, and the task identification is used for indicating the resource usage amount of the feature collection task; and executing at least one feature collection task by using resources of the first computing device to obtain data features of the database, wherein the execution state of the at least one feature collection task is managed according to the task identifier. The process of executing a plurality of feature collection tasks can be fast, the resource consumption is not too much, and the problem that computing equipment of a user does not have enough computing resources to execute business tasks is solved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically to data generation methods, apparatus and systems. Background Technology

[0002] In the field of database testing, this involves simulating and generating database data. By collecting data characteristics from the database and performing data simulation based on these characteristics, it is possible to generate simulated data that closely resembles the data in the database. This simulated data can then be placed into a simulation library, and testing the simulation library can simulate testing the database itself.

[0003] The database is deployed on computing devices, which need to perform numerous feature acquisition tasks to collect data features from the database, consuming significant computing resources. This can lead to slow feature acquisition speeds and insufficient computing resources for executing business tasks. Summary of the Invention

[0004] This application provides a data generation method, apparatus, and system that can accelerate the speed at which computing devices collect data features from databases and generate simulation data.

[0005] In a first aspect, a data generation method is provided. The method includes a first computing device for running a database performing the following steps: obtaining task identifiers for at least one feature acquisition task, wherein the feature acquisition task indicates the acquisition of data features from data in the database, and the task identifier indicates the resource usage of the feature acquisition task; and executing the at least one feature acquisition task using the resources of the first computing device to obtain the data features of the database. This includes managing the execution state of the at least one feature acquisition task based on the task identifier, such that the first computing device executes at least one target feature acquisition task in parallel, and the sum of the resource usage of the at least one target feature acquisition task is less than a resource usage threshold. The data features of the database are used to generate simulation data for the database.

[0006] In the data generation method provided in the above embodiments of this application, each feature acquisition task can indicate its resource usage through a task identifier. Based on this, when the first computing device executes multiple feature acquisition tasks to collect data features from a database for generating simulation data for the database, managing the execution status of at least one feature acquisition task according to the task identifier enables flexible control of the execution strategy of multiple feature acquisition tasks based on the resource usage of the tasks. This can include parallel execution of multiple target feature acquisition tasks without excessive resource consumption. This achieves a faster execution of multiple feature acquisition tasks with minimal resource consumption, reducing the problem of insufficient computing resources for users' computing devices to execute business tasks.

[0007] In some possible implementations, the method further includes: allowing the use of the idle resources of the first computing device to execute at least one feature acquisition task when the idle resources of the first computing device meet the target requirements; and disallowing the use of the resources of the first computing device to execute at least one feature acquisition task when the idle resources of the first computing device do not meet the target requirements.

[0008] Based on this, the resources of the primary computing device can be prioritized for business tasks. This significantly reduces the problem of insufficient computing resources on the user's device to perform business tasks while executing multiple feature acquisition tasks.

[0009] In some other possible implementations, at least one feature acquisition task includes a first type of feature acquisition task and a second type of feature acquisition task. The first type of feature acquisition task indicates the acquisition of data content features from data in the database. The second type of feature acquisition task indicates the acquisition of overall dataset features from the dataset in the database.

[0010] Data content features in a database can characterize the features of the data content. Based on this, the first type of features can be used to guide the generation of simulation data that is similar to or the same as the content features of the source data. The second type of features can be used to guide the adjustment of the generated simulation data so that the overall characteristics of the datasets included in the simulation data are similar to or the same as the overall characteristics of the datasets in the source data.

[0011] In some other possible implementations, the data content characteristics include at least one of the following: data type, data precision, data volume, data business characteristics, and data generation mode.

[0012] In some other possible implementations, the database includes at least one dataset, and the overall characteristics of the dataset include at least one of the following: the data distribution characteristics of at least one dataset, and the relationships between multiple datasets.

[0013] Secondly, another data generation method is provided, the method comprising a second computing device performing the following steps: acquiring data features of a database collected by a first computing device, wherein the data features of the database include a first type of feature and a second type of feature, the first type of feature indicating the data content features of the data in the database, and the second type of feature indicating the overall features of the dataset in the database; generating multiple simulation data subsets in parallel, such that the data content of each simulation data subset conforms to the first type of feature; and performing a data adjustment operation during the parallel generation of multiple simulation data subsets, the data adjustment operation being used to instruct the adjustment of the multiple simulation data subsets so that the multiple simulation data subsets conform to the second type of feature; the multiple simulation data subsets are used to constitute the simulation data of the database.

[0014] In the data generation method provided in the above embodiments of this application, a first type of feature indicates that the generated simulation data is similar to the content features of the source database data. A second type of feature indicates that the overall characteristics of the datasets contained in the generated simulation data (e.g., data distribution patterns, relationships between multiple datasets, etc.) are similar to the overall characteristics of the datasets contained in the source database data. Based on this, multiple subsets of simulation data whose content conforms to the first type of feature can be generated in parallel. These multiple subsets can form the simulation data, thus ensuring the accuracy of the simulation data while accelerating the generation speed and improving the resource utilization of the second computing device. Furthermore, during the parallel generation of multiple subsets of simulation data, the overall characteristics of these subsets are adjusted to conform to the second type of feature, further ensuring the similarity between the simulation data and the source database data.

[0015] In some possible implementations, the total amount of data in multiple simulation data subsets is proportional to the amount of data in the database.

[0016] Based on this, it is possible to generate simulation data that is a multiple of the total amount of the source data more flexibly, and the data distribution of the simulation data is the same as or similar to that of the source data.

[0017] In some other possible implementations, the method further includes: importing multiple subsets of simulation data into a database simulation library in parallel.

[0018] Based on this, the data transfer efficiency of importing simulation data into the simulation library can be improved.

[0019] In some other possible implementations, the method further includes: acquiring data characteristics of the simulation data from the database; and verifying the simulation data based on the data characteristics of the simulation data and the data characteristics of the database. This helps improve the realism of the simulation data.

[0020] In some other possible implementations, the method further includes adjusting the simulation data in the database to conform to the target rules. Based on this, simulation data that meets the requirements can be obtained more flexibly.

[0021] Thirdly, a data generation apparatus is provided, comprising a task identification module and a task execution module. The task identification module is used to acquire task identifiers for at least one feature acquisition task, wherein the feature acquisition task indicates the data characteristics of data collected from a database, and the task identifier indicates the resource usage of the feature acquisition task. The task execution module is used to execute at least one feature acquisition task using the resources of a first computing device to obtain the data characteristics of the database; this includes managing the execution state of the at least one feature acquisition task according to the task identifier, such that the first computing device executes at least one target feature acquisition task in parallel, and the sum of the resource usage of the at least one target feature acquisition task is less than a resource usage threshold. The data characteristics of the database are used to generate simulation data for the database.

[0022] In some possible implementations, the task execution module is further configured to: allow the use of the idle resources of the first computing device to execute at least one feature acquisition task if the idle resources of the first computing device meet the target requirements; and disallow the use of the resources of the first computing device to execute at least one feature acquisition task if the idle resources of the first computing device do not meet the target requirements.

[0023] In some other possible implementations, at least one feature acquisition task includes a first type of feature acquisition task and a second type of feature acquisition task. The first type of feature acquisition task indicates the acquisition of data content features from data in the database. The second type of feature acquisition task indicates the acquisition of overall dataset features from the dataset in the database.

[0024] In some other possible implementations, the data content characteristics include at least one of the following: data type, data precision, data volume, data business characteristics, and data generation mode.

[0025] In some other possible implementations, the database includes at least one dataset, and the overall characteristics of the dataset include at least one of the following: the data distribution characteristics of at least one dataset, and the relationships between multiple datasets.

[0026] Fourthly, another data generation apparatus is provided, comprising a feature acquisition module and a data generation module. The feature acquisition module is used to: acquire data features of a database collected by a first computing device, wherein the data features of the database include a first type of feature and a second type of feature, the first type of feature indicating the data content features of the data in the database, and the second type of feature indicating the overall dataset features of the dataset in the database. The data generation module is used to: generate multiple simulation data subsets in parallel, such that the data content of each simulation data subset conforms to the first type of feature; and, during the parallel generation of multiple simulation data subsets, perform a data adjustment operation, which instructs the adjustment of the multiple simulation data subsets to make the multiple simulation data subsets conform to the second type of feature; the multiple simulation data subsets are used to form the simulation data of the database.

[0027] In some possible implementations, the total amount of data in multiple simulation data subsets is proportional to the amount of data in the database.

[0028] In some other possible implementations, the data generation module is also used to import multiple subsets of simulation data into the database's simulation library in parallel.

[0029] In some other possible implementations, the data generation module is also used to: obtain the data characteristics of the simulation data in the database; and verify the simulation data based on the data characteristics of the simulation data and the data characteristics of the database.

[0030] In some other possible implementations, the data generation module is also used to: adjust the simulation data of the database so that the simulation data of the database conforms to the target rules.

[0031] Fifthly, a data generation system is provided, the system including a first computing device and a second computing device, the first computing device being used to execute the data generation method as described in the first aspect, and the second computing device being used to execute the data generation method as described in the second aspect.

[0032] A sixth aspect provides a computing device including a processor and a memory; the processor is configured to execute instructions stored in the memory to cause the computing device to perform a data generation method as described in the first aspect or a data generation method as described in the second aspect.

[0033] A seventh aspect provides a computing device cluster comprising at least one computing device, wherein each computing device includes a processor and a memory; and the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the computing device cluster to perform a data generation method as described in the first aspect or a data generation method as described in the second aspect.

[0034] Eighthly, a computer program product comprising instructions is provided, which, when executed by at least one computing device, cause the at least one computing device to perform the data generation method as described in the first aspect or the data generation method as described in the second aspect.

[0035] A ninth aspect provides a computer-readable storage medium including computer program instructions, which, when executed by a computing device, enable the computing device to perform a data generation method as described in the first aspect or as described in the second aspect.

[0036] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods.

[0037] The following description includes more specific details about the implementation methods provided for the above aspects. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the architecture of the data generation system provided in the embodiments of this application;

[0039] Figure 2 Flowchart of the data generation method provided in the embodiments of this application Figure 1 ;

[0040] Figure 3 Flowchart of the data generation method provided in the embodiments of this application Figure 2 ;

[0041] Figure 4 This is a schematic diagram of the structure of the data generation apparatus provided in the embodiments of this application;

[0042] Figure 5 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0043] Figure 6 This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application. Detailed Implementation

[0044] A database is a software service that can be deployed on and run on computing devices. Databases are provided to users to store and manage their data. For example, user business data can be stored in a database, and the database can provide data management functions such as modifying, deleting, adding, and querying this business data.

[0045] Databases are provided to users by service providers. Service providers can update databases, such as modifying, adding, or deleting features. Before being released to users, updated databases need to be tested to verify whether the updated database functions correctly or as expected, and to assess aspects such as upgrade paths, reliability, performance, and security. If problems arise with the updated database's functionality, or if upgrade paths, reliability, performance, or security fail to meet requirements, the updated database can be optimized.

[0046] In some scenarios, database testing needs to be conducted based on the customer's actual business data. This improves the reliability of the database test results and ensures the effectiveness and quality of users storing and managing their business data using the updated database. However, some users' business data needs to be kept confidential, and service providers cannot directly obtain this data and place it on their servers to test the updated database.

[0047] In some embodiments, a joint testing lab can be built on the user's server side, where the service provider sends personnel to the user's location to test the updated database using the user's business data. However, this requires on-site personnel for database testing, resulting in high manpower costs and low testing efficiency. Furthermore, the user's server also needs to run the user's business applications, leading to resource constraints and insufficient resources for testing the updated database.

[0048] In some embodiments, business data in the database can also be simulated to generate simulated database data. Simulated database data can refer to data generated by simulating real business operations. Thus, simulated data is not generated during actual business operations, but it can have the same or similar data characteristics as real business data.

[0049] Service providers can deploy a database to be tested (e.g., an updated database obtained by updating a user's database) on their own servers, and make the operating environment of the database to be tested simulate the user's real database operating environment. In this embodiment, the database actually run by the user can be called the source database, and the database running in the operating environment simulating the user's real database operating environment can be called the simulation library. Furthermore, service providers can import the user's simulation data into the database to be tested (e.g., the simulation library) deployed on the service, and then test the database to be tested (e.g., the simulation library).

[0050] In some embodiments, by collecting data features from data in a database and performing data simulation based on those features, simulated data from the database can be generated.

[0051] The source database is deployed on the user's computing devices (such as servers). These devices need to perform numerous feature acquisition tasks to collect data features from the source database, consuming significant computing resources. For example, the computing devices typically execute multiple feature acquisition tasks sequentially in a certain order. This results in slow data feature acquisition speeds and can easily lead to insufficient computing resources for the user's devices to perform business tasks.

[0052] This application provides a data generation method. The method includes a first computing device running a database (e.g., a source database) performing the following steps: obtaining task identifiers for at least one feature acquisition task, wherein the feature acquisition task indicates the acquisition of data features from data in the database, and the task identifier indicates the resource usage of the feature acquisition task; and executing the at least one feature acquisition task using the resources of the first computing device to obtain the data features of the database. This includes managing the execution state of the at least one feature acquisition task according to the task identifier, such that the first computing device executes at least one target feature acquisition task in parallel, and the sum of the resource usage of the at least one target feature acquisition task is less than a resource usage threshold. The data features of the database are used to generate simulation data for the database.

[0053] Thus, in the data generation method provided in the above embodiments of this application, each feature acquisition task can indicate its resource usage through a task identifier. Based on this, when the first computing device executes multiple feature acquisition tasks to collect data features from a database for generating simulation data for the database, managing the execution status of at least one feature acquisition task according to the task identifier enables flexible control of the execution strategy of multiple feature acquisition tasks based on the resource usage of the tasks. This can include parallel execution of multiple target feature acquisition tasks without excessive resource consumption. This achieves a faster execution of multiple feature acquisition tasks with minimal resource consumption, reducing the problem of insufficient computing resources for users' computing devices to execute business tasks.

[0054] This application also provides another data generation method, which includes a second computing device performing the following steps: acquiring data features of a database collected by a first computing device, wherein the data features of the database include a first type of feature and a second type of feature, the first type of feature indicating the data content features of the data in the database, and the second type of feature indicating the overall features of the dataset in the database; generating multiple simulation data subsets in parallel, such that the data content of each simulation data subset conforms to the first type of feature; and performing a data adjustment operation during the parallel generation of multiple simulation data subsets, the data adjustment operation being used to instruct the adjustment of the multiple simulation data subsets so that the multiple simulation data subsets conform to the second type of feature; the multiple simulation data subsets are used to form the simulation data of the database.

[0055] Thus, in the data generation method provided in the above embodiments of this application, the first type of feature can indicate the generation of simulation data with similar content features to the source database data, and the second type of feature can indicate that the overall features of the dataset contained in the generated simulation data (such as the data distribution pattern of the dataset, the correlation between multiple datasets, etc.) are similar to the overall features of the dataset contained in the source database data. Based on this, multiple subsets of simulation data whose data content conforms to the first type of feature can be generated in parallel. These multiple subsets of simulation data can form simulation data, thereby ensuring the accuracy of the simulation data while accelerating the generation speed of simulation data and improving the resource utilization of the second computing device. Furthermore, during the parallel generation of multiple subsets of simulation data, the overall features of the multiple subsets of simulation data are adjusted to conform to the second type of feature, further ensuring the similarity between the simulation data and the source database data.

[0056] The data generation method provided in this application is applicable to generating simulation data from various types of databases. Furthermore, the data in the database can be of various types, such as text, images, and audio data.

[0057] The data generation method provided in this application can also be applied to various scenarios that require generating simulation data from a database. For example, it can be applied to testing a simulation library or other scenarios involving generating simulation data from a database.

[0058] The system architecture provided in the embodiments of this application is described below.

[0059] Figure 1 This is a schematic diagram of the architecture of a data generation system provided in an embodiment of this application. The data generation system 100 can be used to implement the data generation method proposed in the embodiments of this application. Figure 1 As shown, the data generation system 100 includes a first computing device 110, a database 120, a second computing device 130, a simulation library 140, and a testing device 150.

[0060] The first computing device 110 can be used to run the database 120. The database 120 can be of various types and can be used to store and manage user data (such as business data).

[0061] The first computing device 110 may include a single computing device or a cluster of computing devices. The computing device may be a server, a terminal (such as a mobile phone terminal, a laptop computer, etc.), etc.

[0062] The first computing device 110 can be used to execute the data generation method provided in the embodiments of this application, for example... Figure 2 The data generation method provided herein enables the collection of data features from database 120. As an example, the first computing device 110 can obtain task identifiers for at least one feature collection task, wherein the feature collection task indicates the collection of data features from database 120, and the task identifier indicates the resource usage of the feature collection task. Furthermore, the first computing device 110 can also use its resources to execute at least one feature collection task to obtain the data features of database 120. This includes managing the execution state of at least one feature collection task based on the task identifier, such that the first computing device 110 executes at least one target feature collection task in parallel, and the sum of the resource usage of the at least one target feature collection task is less than a resource usage threshold.

[0063] The second computing device 130 may include a single computing device or a cluster of computing devices.

[0064] The second computing device 130 can be used to acquire data features from the database 120 collected by the first computing device 110. The second computing device 130 can acquire the data features from the database 120 collected by the first computing device 110 through various feasible methods. For example, the second computing device 130 can acquire the data features from the database 120 by communicating with the first computing device 110 via a network. Alternatively, if the first computing device 110 cannot or finds it inconvenient to communicate with the second computing device 130, the data features from the database 120 on the first computing device 110 can be downloaded to a storage device first, and then the data features from the database 120 on the storage device can be imported into the second computing device 130.

[0065] The second computing device 130 can also be used to generate simulation data for the database 120 based on the data characteristics of the database 120. The second computing device 130 can be used to execute the data generation method provided in the embodiments of this application, for example... Figure 3The data generation method provided herein generates simulation data for database 120. As an example, the second computing device 130 can acquire data features of database 120 collected by the first computing device 110. These data features include a first type of feature and a second type of feature. The first type of feature indicates the data content features of the data in the database, and the second type of feature indicates the overall features of the dataset in the database. Furthermore, the second computing device 130 can generate multiple subsets of simulation data in parallel, such that the data content of each subset conforms to the first type of feature. These multiple subsets of simulation data are used to form the simulation data for database 120. Additionally, the second computing device 130 can perform a data adjustment operation during the parallel generation of multiple subsets of simulation data. This data adjustment operation instructs the adjustment of the multiple subsets of simulation data to make them conform to the second type of feature.

[0066] Test device 150 can be used to run simulation library 140. Database 120 can be the source database of simulation library 140. Test device 150 may include a computing device or a cluster of computing devices.

[0067] The second computing device 130 can send the generated simulation data to the test device 150 and import it into the simulation library 140 on the test device 150. The test device 150 can run the simulation library 140 to perform simulation library 140 testing.

[0068] Figure 1 This is merely a schematic diagram of the architecture of a data generation system provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc. shown in the figure do not constitute any limitation.

[0069] For example, Figure 1 The second computing device 130 is a device other than the test device 150. In other cases, the second computing device 130 may be a device within the test device 150.

[0070] For example, Figure 1 In this context, the first computing device 110 and the second computing device 130 are different devices. In other cases, the first computing device 110 and the second computing device 130 may also be the same device.

[0071] The data generation method provided in the embodiments of this application is described in detail below. Figure 1 The data generation system 100 shown is implemented on the system.

[0072] Figure 2 Flowchart of the data generation method provided in the embodiments of this application Figure 1 . Figure 2The data generation method shown can be executed by a computing device. For example, it can be executed by the first computing device 110 in the data generation system 100.

[0073] like Figure 2 As shown, the method includes the following steps.

[0074] Step 210: Obtain the task identifier of at least one feature acquisition task.

[0075] Feature acquisition task refers to the task of indicating the data characteristics of data in the acquisition database (which may be called the source database).

[0076] The database may include data, such as user business data. The data in the database can be of various types, such as text, images, and audio. The data in the database can also be data with various data structures; for example, the data in the database may be stored in the form of multiple datasets, and there may be relationships between several datasets. As an example, the database can be a relational database, where the data can be stored in the form of tables (in this embodiment, a table can be considered a dataset), and multiple tables may have relationships between them.

[0077] The first computing device can obtain feature acquisition tasks through various feasible methods. For example, the first computing device can directly generate feature acquisition tasks, or technicians can send feature acquisition features to the first computing device through a client device.

[0078] Feature acquisition tasks can be multiple, and they can be of various types. Different feature acquisition tasks can be used to collect different data features from the data in the database. Thus, multiple feature acquisition tasks can be used to collect multiple data features from the data in the database.

[0079] In the embodiments of this application, for ease of description, the data in the database that needs to be collected for data features can be referred to as source data.

[0080] In some embodiments, the data characteristics of the source data to be collected include one or more of the following: metadata (e.g., metadata of a table), data content characteristics of a single dataset (e.g., data content characteristics of a single table), data distribution characteristics of a dataset (e.g., data distribution characteristics of a single dataset or multiple datasets), relationships between multiple datasets (e.g., relationships between multiple tables), data business characteristics, etc.

[0081] Metadata can include various attributes of the data, such as data type and data precision.

[0082] As an example, data types can include various types such as Boolean, binary, string, date, and numeric.

[0083] Data precision can include fixed-length random values, variable-length values ​​within a range, null / empty values, and multiple references in a dictionary.

[0084] Data content features refer to the characteristics related to data content. Data content features can be used to characterize the features of data content.

[0085] The data content characteristics of a single dataset may include one or more of the following: the most common value (MCV) of the dataset, MCV probability, isofrequency histogram, average width, null value rate, hashing degree, unique values, data volume, data generation pattern, etc.

[0086] In this context, "data generation pattern" refers to a characteristic used to indicate the rules or patterns of data generation. For example, if an array contains phone numbers, and phone numbers have certain data rules or patterns, then the data generation pattern can indicate the phone numbers. As an example, data generation patterns can include types such as phone number, address, name, ID card number, and MAC address.

[0087] In some embodiments, the data attributes in the metadata described above may also be considered data content features.

[0088] The relationships between multiple datasets can include one or more of the following: set relationships between datasets, correspondence relationships between datasets, covariance of multiple datasets, hit rate, multiplier, and expression relationships. For example, set relationships between datasets can include various types such as equality, inclusion, and mutual exclusion. Correspondence relationships between datasets can include various types such as one-to-one, one-to-many, and many-to-many.

[0089] Data business characteristics can include the business characteristics of various business data such as logs, basic data, and core libraries.

[0090] All of the above-mentioned data features can be collected by corresponding feature acquisition features. In some embodiments, depending on the type of data feature, the feature acquisition task may include a first type of feature acquisition task and a second type of feature acquisition task.

[0091] The first type of feature acquisition task indicates the acquisition of data content features from the database. As an example, data content features may include the aforementioned metadata, and data content features of a single dataset (e.g., isofrequency histogram, average width, null value rate, non-repeating values, data volume, etc.).

[0092] The second type of feature acquisition task instructs the collection of overall features of datasets in a database. These overall features can include features related to one or more datasets as a whole. For example, overall dataset features might include the aforementioned data distribution characteristics and the relationships between multiple datasets.

[0093] Each feature acquisition task can be assigned a corresponding task identifier. The task identifier indicates the resource usage of the feature acquisition task. Resource usage includes the usage of various types of resources such as computing resources and storage resources.

[0094] Task identifiers can be represented in various feasible data formats, such as numbers and letters. For example, 0.1U can represent using 10% of the resources of one CPU, and 0.2GB can represent using 0.2GB of storage resources (such as memory resources).

[0095] The resource usage for feature acquisition tasks that collect different data features can vary.

[0096] As examples, the following are some examples of resource usage for feature acquisition tasks:

[0097] Task 1: Calculate the percentage of fields that are empty (which can be represented by null_frac), with resource usage of 0.1U and 0.2GB;

[0098] Task 2: Calculate the average width of the field records in bytes (which can be represented by avg_width), with resource usage of 0.1U and 0.2GB;

[0099] Task 3: Estimate the number of unique values ​​in the record field and the negative number of unique values ​​divided by the row (which can be represented by n_distinct), with resource usage of 0.1U and 0.2GB respectively;

[0100] Task 4: Calculate the most frequently occurring values ​​(which can be represented by most_common_val), with resource usage of 0.1U and 0.2GB;

[0101] Task 5: Calculate the probability of the most frequently occurring value (which can be represented by most_common_freqs), with resource usage of 0.1U and 0.2GB;

[0102] Task 6: Calculate the equal frequency histogram (which can be represented by histogram_bounds) of the values ​​other than null values ​​and MCV values, with resource usage of 0.1U and 0.2GB;

[0103] Task 7: Calculate the hash rate (including values ​​ranging from -1 to +1) (which can be represented by correlation), with resource usage of 0.1U and 0.2GB;

[0104] Task 8: Statistical data volume (can be represented by count), resource usage is 0.1U and 0.2GB;

[0105] Task 9: Statistical summary (can be represented as sum), resource usage is 0.1U and 0.2GB;

[0106] Task 10: Statistically determine the maximum value (which can be represented by max), with resource usage of 0.1U and 0.2GB;

[0107] Task 11: Calculate the minimum statistical values ​​(which can be represented by min), with resource usage of 0.1U and 0.2GB;

[0108] Task 12: Statistical average (can be represented by avg), resource usage is 0.1U and 0.2GB;

[0109] Task 13: Statistical median (can be represented as median), resource usage is 0.1U and 0.2GB;

[0110] Task 14: Statistical standard deviation (represented by stddev_pop), resource usage is 0.1U and 0.2GB;

[0111] Task 15: Analyze the relationships between data sets (e.g., equality, inclusion, mutual exclusion), with resource usage of 0.2U and 0.4GB.

[0112] Task 16: Statistical data grouping and sorting (e.g., group aggregation, window aggregation, sorting, etc.), resource usage is 0.4U and 0.8GB.

[0113] In some embodiments, the data scale of the feature acquisition task can also be set. For example, if the data scale of the feature acquisition task is set to 10%, then the feature acquisition task indicates that 10% of the data in the database should be collected.

[0114] The scale of data collected for a feature acquisition task can be set according to requirements (e.g., the accuracy of the features collected). A smaller data scale results in a shorter execution time for the feature acquisition task. This allows for more flexible and efficient execution of feature data acquisition tasks.

[0115] As an example, some feature acquisition tasks (which can be called high-efficiency tasks) have a data size of 10% and the accuracy of the acquired data features is 95%. Other feature acquisition tasks (which can be called comprehensive tasks) have a data size of 50% and the accuracy of the acquired data features is 99%. Still other feature acquisition tasks (which can be called precise tasks) have a data size of 100% and the accuracy of the acquired data features is 100%.

[0116] Step 220: Use the resources of the first computing device to perform at least one feature acquisition task to obtain data features of the database.

[0117] In step 220, the execution of at least one feature acquisition task can be managed to ensure that the execution process of at least one feature acquisition task meets the execution requirements. The execution requirements can be determined according to the needs, such as requirements that the execution time of at least one feature acquisition task meets the requirements, the resources consumed in executing at least one feature acquisition task are less than the target threshold, and the resources consumed in executing at least one feature acquisition task do not affect the execution of business tasks by the first computing device.

[0118] In some embodiments, managing the execution of at least one feature acquisition task may include: allowing the use of the idle resources of the first computing device to execute the feature acquisition task in at least one feature acquisition task when the idle resources of the first computing device meet the target requirements; and disallowing the use of the resources of the first computing device to execute the feature acquisition task in at least one feature acquisition task when the idle resources of the first computing device do not meet the target requirements. The target requirements may be determined according to needs, such as including idle resources exceeding a target threshold, or the proportion of idle resources to the total resources of the first computing device exceeding a target proportion.

[0119] The availability of resources on the first computing device can be determined through various feasible methods. For example, a heartbeat packet can be installed on the first computing device to check its resource usage at regular intervals, thus dynamically determining the availability of resources.

[0120] Step 220 may include managing the execution status of at least one feature acquisition task based on a task identifier. The execution status of the task may include executing, waiting to be executed, etc.

[0121] Managing the execution status of at least one feature acquisition task based on the task identifier enables flexible control of the execution strategies of multiple feature acquisition tasks according to the resource usage of the task, so that the execution process of at least one feature acquisition task can meet the target requirements.

[0122] In some embodiments, managing the execution status of at least one feature acquisition task based on a task identifier may include: determining at least one target feature acquisition task among the at least one feature acquisition task based on the task identifier of the at least one feature acquisition task, wherein the sum of the resource usage of the target feature acquisition task is less than a resource usage threshold; and causing a first computing device to execute the at least one target feature acquisition task in parallel.

[0123] In this way, executing multiple target feature acquisition tasks in parallel allows at least one feature acquisition task to be executed quickly, while not consuming too many resources, thus reducing the problem that users' computing devices do not have enough computing resources to execute business tasks.

[0124] pass Figure 2Once the data generation method provided in the database obtains the data characteristics of the database, these data characteristics can be used to generate simulation data for the database.

[0125] The following details the method for generating database simulation data based on the data characteristics of the database, as provided in the embodiments of this application.

[0126] Figure 3 Flowchart of the data generation method provided in the embodiments of this application Figure 2 . Figure 3 The data generation method shown can be executed by a computing device. For example, it can be executed by the second computing device 130 in the data generation system 100.

[0127] like Figure 3 As shown, the method includes the following steps.

[0128] Step 310: Obtain the data characteristics of the database.

[0129] The second computing device can acquire data features from the database collected by the first computing device. The second computing device can acquire these data features through various feasible methods. For more detailed information on the acquisition methods, please refer to [link to relevant documentation]. Figure 1 And its related descriptions.

[0130] In some embodiments, according to Figure 2 The data features described in the method are classified according to data feature type. The data features of the database may include a first type of feature and a second type of feature.

[0131] The first type of feature indicates the data content characteristics of the data in the database. The second type of feature indicates the overall characteristics of the dataset in the database. For more detailed descriptions of data content characteristics and overall dataset characteristics, please refer to [link to relevant documentation]. Figure 2 And its related descriptions.

[0132] As mentioned above, the data content characteristics of data in a database can characterize the features of the data content. Based on this, the first type of feature can be used to guide the generation of simulation data that is similar to or identical to the source data in terms of content characteristics.

[0133] The second type of feature can be used to guide the adjustment of the generated simulation data so that the overall features of the datasets included in the simulation data are similar to or the same as the overall features of the datasets in the source data.

[0134] Step 320: Generate multiple subsets of simulation data in parallel. These subsets are used to form the simulation data in the database.

[0135] In this embodiment of the application, the simulation data to be generated can be divided into multiple data subsets, and the data subsets of the simulation data can be referred to as simulation data subsets.

[0136] In step 320, the second computing device can generate multiple simulation data subsets in parallel based on the first type of features, such that the data content of each simulation data subset conforms to the first type of features. This improves the efficiency of generating simulation data while ensuring that the data content of each generated simulation data subset is similar to or identical to the source data content features.

[0137] In some embodiments, the second computing device can execute a first function (also referred to as a basic constraint function in this embodiment) to generate multiple simulation data subsets in parallel based on a first type of feature, such that the data content of each simulation data subset conforms to the first type of feature. The first function is used to instruct the parallel generation of multiple simulation data subsets and that the data content of the multiple simulation data subsets conforms to the first type of feature.

[0138] Step 320 may further include performing a data adjustment operation during the parallel generation of multiple subsets of simulation data. The data adjustment operation is used to instruct the multiple subsets of simulation data to be adjusted so that the multiple subsets of simulation data conform to the second type of characteristics.

[0139] In some embodiments, the second computing device can perform data adjustment operations during the parallel generation of multiple simulation data subsets by executing a second function (also referred to as a dynamic constraint function in this embodiment). The first function instructs the adjustment of data within the multiple simulation data subsets so that the overall characteristics of the datasets of the multiple simulation data subsets conform to a second type of characteristic.

[0140] Adjusting data can include various data adjustment methods, such as deleting data, adding data, and modifying data values.

[0141] To facilitate understanding, as examples, some examples of the first function (also known as the fundamental constraint function) are given below.

[0142] The first type of function indicates that the data type of the generated data is boolean, which can be represented as Get_Random_Bool(int bool Type), and can be used to generate gender data.

[0143] The second type, where the first function indicates that the data type of the generated data is binary, can be represented as Get_Random_Binary(int length), and can be used to generate image data.

[0144] The third type, where the first function indicates that the data type of the generated data is string, can be represented as Get_Random_String(int length), and can be used to generate data such as addresses, ID cards, and names.

[0145] The fourth type of function indicates the data type of the generated data, which is date type. It can be represented as Get_Random_Date(int position) and Get_Random_Timestamp(int position) and can be used to generate date data.

[0146] The fifth type of function indicates that the data type of the generated data is a bit string, which can be represented as Get_Random_bit(int length) and can be used to generate bitmask data.

[0147] The sixth type of function indicates that the data type of the generated data is numeric, which can be represented as Get_Random_int(), and can be used to generate mobile phone number data.

[0148] To facilitate understanding, as examples, some examples of the second function (also known as the dynamic constraint function) are given below.

[0149] The first scenario is: during the data generation process, the second function instructs the algorithm on the data distribution pattern, ensuring that after multiple simulation data subsets are generated in parallel, the final set of simulation data subsets conforms to a normal distribution. The second function may include:

[0150] Random_Binomial_Distribute(table1.column1) / / Randomly generates a single record, and the entire table column1 follows a binomial distribution; Random_Poisson_Distribute(table1.column2) / / Randomly generates a single record, and the entire table column2 follows a Poisson distribution; Random_Normal_Distribute(table1.column3) / / Randomly generates a single record, and the entire table column3 follows a normal distribution; Random_Exponential_Distribute(table1.column4) / / Randomly generates a single record, and the entire table column4 follows an exponential distribution.

[0151] The first scenario is as follows: During the data generation process, the second function instructs a data association algorithm to generate multiple simulation data subsets in parallel, with the total data volume of each subset being 1:1 with the source data volume. Ultimately, the total simulation data volume satisfies a 1:n or n:n relationship with the source data volume. The second function may include:

[0152] Random_Relaition_1ton(table1.column1,table2.column2) / / Generates 2 related records in a 1:1 ratio, resulting in an overall 1:n ratio; Random_Relaition_nton(table1.column1,table2.column2) / / Generates 2 related records in a 1:1 ratio, resulting in an overall n:n ratio.

[0153] In some embodiments, the first function and the second function in this application can be collectively referred to as a data hashing algorithm.

[0154] In step 320, the total amount of simulation data to be generated can be set according to requirements, and the subset data volume of multiple simulation data subsets can be determined according to the set total amount of data, thereby generating multiple simulation data subsets corresponding to the subset data volume.

[0155] In some embodiments, the total amount of simulation data (i.e., the total amount of data in multiple subsets of simulation data) may be the same as or different from the amount of source data.

[0156] In some embodiments, the total amount of simulation data and the total amount of source data can be multiples of each other. For example, if the total amount of source data is A, the total amount of simulation data can be 0.1A, 10A, 100A, etc.

[0157] In some embodiments, Figure 3 The method may also include importing multiple subsets of simulation data into the simulation library in parallel.

[0158] For example, the simulation library is deployed on a test device (e.g., test device 150 in data generation system 100). A second computing device can send multiple subsets of simulation data to the test device in parallel, thus enabling the import of multiple subsets of simulation data into the simulation library on the test device. The multiple subsets of simulation data constitute the simulation data in the simulation library.

[0159] In some embodiments, Figure 3 The method may also include verifying the simulation data in the simulation library. Methods for verifying the simulation data in the simulation library may include: obtaining the data characteristics of the simulation data in the simulation library, and verifying the simulation data in the simulation library based on the data characteristics of the simulation data and the data characteristics of the source database. The method for obtaining the data characteristics of the simulation data in the simulation library can be similar to the method for obtaining the data characteristics of the source database; see [link to relevant documentation] for details. Figure 2 The relevant description of the Chinese method.

[0160] In this case, if the data characteristics of the simulation data are the same as or the similarity between the simulation data and the data characteristics of the source database meets the requirements, the simulation data of the simulation library can be considered to have passed the verification. If the data characteristics of the simulation data are different from or the similarity between the simulation data and the data characteristics of the source database does not meet the requirements, the simulation data of the simulation library can be considered to have failed the verification.

[0161] In some embodiments, Figure 3 The method may also include checking whether the simulation data in the simulation library meets the target rules. If not, the simulation data in the database can be adjusted to make it conform to the target rules. The target rules can be set according to requirements.

[0162] For example, simulation data in the simulation library can be checked by creating indexes to refresh simulation library statistics, performing full table scans, single-rule updates, and checking whether statistics frequently expire. Furthermore, the simulation data in the database can be adjusted using Structured Query Language (SQL).

[0163] Through the embodiments of this application Figure 2 The provided data generation method can extract data features with relatively small data sizes (e.g., MB) from source databases with large data sizes (e.g., TB level). These data features are further divided into two categories: a first category of features and a second category of features with even smaller data sizes (e.g., KB level).

[0164] And, through the embodiments of this application Figure 3 The provided data generation method can generate simulation data subsets with larger data sizes (e.g., GB level) based on data features with smaller data sizes (e.g., first-type features, second-type features). Multiple simulation data subsets can then be combined to form simulation data with even larger data sizes (e.g., TB level).

[0165] Through the embodiments of this application Figure 2 and Figure 3 The provided data generation method can achieve dynamic planning and configurable acquisition cycle for data feature collection and resource consumption actions; maintain only local data and dynamically verify and maintain global constraints during data generation; generate simulation data in a 1:n ratio, and ensure that the data distribution patterns, correlations and other relationships of simulation data of various scales are consistent with the source data; generate data in parallel with local data generation and dynamically maintain global relationships (reducing the dimensionality of the generated data to make full use of resources); and provide simulation degree verification for the source database and simulation library.

[0166] The above text combines Figures 2 to 3 The present application describes in detail the data generation method provided according to the embodiments of this application. The following will combine... Figure 4This application describes the apparatus provided according to the present application. These apparatuses can be used to implement the functions of the processor in the above method embodiments, and thus can also achieve the beneficial effects of the above method embodiments.

[0167] Figure 4 This is a schematic diagram of the data generation apparatus provided in an embodiment of this application. Figure 4 As shown, the data generation device 400 includes a first device 410 and a second device 420. The first device 410 includes a task identification module 412 and a task execution module 414. The second device 420 includes a feature acquisition module 422 and a data generation module 424.

[0168] The task identifier module 412 is used to obtain the task identifier of at least one feature acquisition task, wherein the feature acquisition task indicates the data characteristics of data in the database to be acquired, and the task identifier is used to indicate the resource usage of the feature acquisition task.

[0169] The task execution module 414 is used to execute at least one feature acquisition task using the resources of the first computing device to obtain data features of the database. This includes managing the execution status of the at least one feature acquisition task based on a task identifier, such that the first computing device executes at least one target feature acquisition task in parallel, and the sum of the resource usage of the at least one target feature acquisition task is less than a resource usage threshold. The data features of the database are used to generate simulation data for the database.

[0170] In some implementations, the task execution module 414 is further configured to: allow the use of the idle resources of the first computing device to execute at least one feature acquisition task if the idle resources of the first computing device meet the target requirements; and disallow the use of the resources of the first computing device to execute at least one feature acquisition task if the idle resources of the first computing device do not meet the target requirements.

[0171] In some implementations, at least one feature acquisition task includes a first type of feature acquisition task and a second type of feature acquisition task. The first type of feature acquisition task indicates the acquisition of data content features from data in the database. The second type of feature acquisition task indicates the acquisition of overall dataset features from the dataset in the database.

[0172] In some implementations, data content characteristics include at least one of the following: data type, data precision, data volume, data business characteristics, and data generation mode.

[0173] In some implementations, the database includes at least one dataset, and the overall characteristics of the dataset include at least one of the following: the data distribution characteristics of at least one dataset, and the relationships between multiple datasets.

[0174] The feature acquisition module 422 is used to acquire the data features of the database collected by the first computing device. The data features of the database include a first type of feature and a second type of feature. The first type of feature indicates the data content features of the data in the database, and the second type of feature indicates the overall features of the dataset in the database.

[0175] The data generation module 424 is used to generate multiple simulation data subsets in parallel, such that the data content of each simulation data subset conforms to the first type of feature; and, in the process of generating multiple simulation data subsets in parallel, to perform a data adjustment operation, which is used to instruct the adjustment of multiple simulation data subsets so that the multiple simulation data subsets conform to the second type of feature; the multiple simulation data subsets are used to form the simulation data of the database.

[0176] In some implementations, the total amount of data in multiple simulation data subsets is proportional to the amount of data in the database.

[0177] In some implementations, the data generation module 424 is also used to import multiple subsets of simulation data into the simulation library of the database in parallel.

[0178] In some implementations, the data generation module 424 is also used to: obtain the data characteristics of the simulation data in the database; and verify the simulation data based on the data characteristics of the simulation data and the data characteristics of the database.

[0179] In some implementations, the data generation module 424 is also used to: adjust the simulation data of the database so that the simulation data of the database conforms to the target rules.

[0180] For more details regarding the functions of the above modules, please refer to the aforementioned document. Figures 2-3 And its related descriptions.

[0181] Optionally, the task identification module 412, task execution module 414, feature acquisition module 422, and data generation module 424 may each include multiple sub-modules. These sub-modules can be deployed separately to implement some of the functions of their respective modules, such as implementing the aforementioned... Figures 2-3 One or more steps of the provided data generation method.

[0182] All devices can be implemented in software or hardware. For example, the implementation of the data generation device 400 will be described below.

[0183] As an example of a software functional unit, the data generation device 400 may include code running on a computing instance. The computing instance can be at least one of a physical host (computing device), a virtual machine, a container, or other computing devices. Furthermore, the aforementioned computing instance may be one or more. For example, the data generation device 400 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code can be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code can be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0184] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0185] As an example of a hardware functional unit, the data generation device 400 may include at least one computing device, such as a server. Alternatively, the data generation device 400 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0186] The data generation device 400 includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the multiple computing devices in the data generation device 400 can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices in the data generation device 400 can be distributed within the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0187] This application also provides a computing device 500. For example... Figure 5 As shown, the computing device 500 includes: a bus 502, a processor 504, a memory 506, and a communication interface 508. The processor 504, memory 506, and communication interface 508 communicate with each other via the bus 502. The computing device 500 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 500. Optionally, the processor 504 of the computing device 500 can be connected to a display or input device via the communication interface 508. Figure 5 (Not shown in the image).

[0188] Bus 502 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 5 The bus 502 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 502 may include a path for transmitting information between various components of the computing device 500 (e.g., memory 506, processor 504, communication interface 508).

[0189] Processor 504 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP). In this embodiment, processor 504 is used to execute the functions described in this embodiment. Figures 2-3 The steps or operations in the data generation method proposed in the paper.

[0190] Memory 506 may include volatile memory, such as random access memory (RAM). Processor 504 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0191] The memory 506 stores executable program code, and the processor 504 executes the executable program code to implement the functions of the aforementioned task identification module 412, task execution module 414, feature acquisition module 422, and data generation module 424, thereby realizing the embodiments of this application. Figures 2-3 The data generation method proposed in this application. That is, the memory 506 stores data for executing the embodiments of this application. Figures 2-3 The instructions for the data generation method proposed in the paper.

[0192] The communication interface 508 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 500 and other devices or communication networks.

[0193] The display may include various display devices capable of display functions, such as plasma displays and liquid crystal displays. The computing device 500 can implement the functions described in the embodiments of this application through the display. Figures 2-3 The data generation method proposed in the paper involves display functions.

[0194] Input devices can include various input devices that enable information / signal input, such as keyboards, mice, and touch screens.

[0195] This application also provides a computing device cluster 600. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0196] like Figure 6 As shown, the computing device cluster includes at least one computing device 500. The memory 506 in one or more computing devices 500 of the computing device cluster may store the same memory for executing the embodiments of this application. Figures 2-3 Instructions for the provided data generation method.

[0197] In some possible implementations, the memory 506 of one or more computing devices 500 in the computing device cluster may also store memory for executing the embodiments of this application. Figures 2-3 The provided data generation method includes some instructions. In other words, a combination of one or more computing devices 500 can jointly execute instructions for performing the embodiments of this application. Figures 2-3 Instructions for the provided data generation method.

[0198] It should be noted that the memory 506 in different computing devices 500 within the computing device cluster can store different instructions, each used to implement a portion of the functions of the data generation device 400. That is, the instructions stored in the memory 506 of different computing devices 500 can implement the functions of one or more of the aforementioned task identification module 412, task execution module 414, feature acquisition module 422, and data generation module 424.

[0199] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc.

[0200] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. The instructions contained in the computer program product may include instructions for performing the embodiments of this application. Figures 2-3 The instructions for the steps or operations in the data generation method proposed herein. When the computer program product is run on at least one computing device, the at least one computing device performs the actions described in the embodiments of this application. Figures 2-3 The steps or operations of the data generation method proposed in the paper.

[0201] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the embodiments of this application. Figures 2-3 The steps or operations of the data generation method proposed in the paper.

[0202] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

[0203] The terms “first,” “second,” “third,” and “fourth,” etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to define a specific order.

[0204] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

Claims

1. A data generating method characterized by comprising: The method is executed by a first computing device configured to run a database, and comprises: obtaining task identifiers of at least one feature collection task, the feature collection task indicating collection of data features of data in the database, the task identifier being used to indicate resource usage of the feature collection task; executing the at least one feature collection task using resources of the first computing device to obtain data features of the database; wherein the execution state of the at least one feature collection task is managed according to the task identifier, so that the first computing device executes at least one target feature collection task in the at least one feature collection task in parallel, and the sum of the resource usage of the at least one target feature collection task is less than a resource usage threshold; and the data features of the database are used to generate simulation data of the database.

2. The method of claim 1, wherein, The execution of the at least one feature collection task using the resources of the first computing device further comprises: in the case that the idle resources of the first computing device meet target requirements, allowing the execution of a feature collection task in the at least one feature collection task using the idle resources of the first computing device; in the case that the idle resources of the first computing device do not meet target requirements, not allowing the execution of a feature collection task in the at least one feature collection task using the resources of the first computing device.

3. The method according to claim 1 or 2, characterized in that, The at least one feature collection task comprises a first type of feature collection task and a second type of feature collection task; The first type of feature collection task indicates collection of data content features of data in the database; The second type of feature collection task indicates collection of data set overall features of data sets in the database.

4. The method of claim 3, wherein, The data content features comprise at least one of: data type, data precision, data volume, data service features, and data generation mode.

5. The method according to claim 3 or 4, characterized in that, The database comprises at least one data set, and the data set overall features comprise at least one of: data distribution features of the at least one data set, and association relationships between multiple data sets.

6. The method according to any one of claims 1-5, characterized in that, The data generation system comprises the first computing device and a second computing device, and the method further comprises: The second computing device obtains data features of the database collected by the first computing device, the data features of the database comprising first type features, the first type features indicating data content features of data in the database; The second computing device generates multiple simulation data subsets in parallel, so that the data content of each simulation data subset conforms to the first type features, and the multiple simulation data subsets are used to constitute simulation data of the database.

7. The method of claim 6, wherein, The data features of the database further comprise second type features, the second type features indicating data set overall features of data sets in the database; and the method further comprises: performing a data adjustment operation in the process of generating the multiple simulation data subsets in parallel, the data adjustment operation being used to indicate adjustment of the multiple simulation data subsets so that the multiple simulation data subsets conform to the second type features.

8. The method according to claim 6 or 7, characterized in that, The total amount of data of the multiple simulation data subsets has a multiple relationship with the data volume of the database.

9. The method according to any one of claims 6-8, characterized in that, The method further comprises: The second computing device imports the plurality of simulation data subsets into a simulation library of the database in parallel.

10. The method according to any one of claims 6-9, characterized in that, The method further comprises: Obtaining data features of simulation data of the database; Verifying the simulation data according to the data features of the simulation data and the data features of the database.

11. The method according to any one of claims 6-10, characterized in that, The method further comprises: Adjusting the simulation data of the database so that the simulation data of the database conforms to the target rules.

12. A data generating method characterized by comprising: The method is performed by a second computing device, and the method comprises: Obtaining data features of a database collected by a first computing device, the database being run by the first computing device, the data features of the database comprising first type features and second type features, the first type features indicating data content features of data in the database, and the second type features indicating data set overall features of data sets in the database; Generating a plurality of simulation data subsets in parallel, so that data content of each of the simulation data subsets conforms to the first type features; and performing a data adjustment operation in the process of generating the plurality of simulation data subsets in parallel, the data adjustment operation being used to instruct adjustment of the plurality of simulation data subsets so that the plurality of simulation data subsets conform to the second type features; the plurality of simulation data subsets being used to constitute simulation data of the database.

13. A data generating apparatus characterized by comprising: The apparatus comprises: A task identification module configured to obtain task identifications of at least one feature collection task, the feature collection task being configured to collect data features of data in the database, and the task identification being configured to indicate a resource usage amount of the feature collection task; A task execution module configured to execute the at least one feature collection task using resources of a first computing device to obtain data features of the database; wherein the task execution module is configured to manage execution states of the at least one feature collection task according to the task identifications, so that the first computing device executes at least one target feature collection task in the at least one feature collection task in parallel, and a sum of the resource usage amounts of the at least one target feature collection task is less than a resource usage threshold; and the data features of the database are used to generate simulation data of the database.

14. A data generating system characterized by comprising: The system comprises a first computing device configured to perform the method of any one of claims 1-5, and a second computing device configured to perform the method of any one of claims 6-12.

15. A computing device, comprising: The computing device comprises a processor and a memory; the processor is configured to execute instructions stored in the memory, so that the computing device performs the method of any one of claims 1-12.

16. A cluster of computing devices, characterized in that, The computing device cluster comprises at least one computing device, and each computing device comprises a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method of any one of claims 1-12.

17. A computer program product comprising instructions, characterized in that, The instructions, when executed by the at least one computing device, cause the at least one computing device to perform the method of any one of claims 1-12.

18. A computer-readable storage medium, characterized in that, comprising computer program instructions, which, when executed by a computing device, cause the computing device to perform the method of any of claims 1-12.

Images