A test data generation method, device, electronic device and storage medium

Abstract

The present application discloses a test data generation method, device, electronic device and storage medium, wherein the test data generation method comprises: generating at least one second request log under the corresponding category based on at least one first request log under each category in at least one set category; generating test data based on all or part of the request logs under each category in at least one category; wherein, in the request logs for generating the test data, the difference between the first ratio and the corresponding second ratio of each category is less than a first threshold value; the first request log is a log generated based on a user access request; the first ratio represents the ratio of the request log under the corresponding category to all request logs; the second ratio represents the ratio of the request log under the set corresponding category to all request logs.

Description

Technical Field

[0001] The present application relates to the field of big data, and in particular to a test data generation method, device, electronic device and storage medium. Background Art

[0002] During the e-commerce promotion activities, the number of user visits and business volume increased dramatically, which brought severe challenges to the e-commerce platform and put forward higher requirements for the high concurrent processing capability, large-scale computing power, response speed and stability of the backend system of the e-commerce platform. In order to ensure the normal operation of the e-commerce platform during the activities, the system capacity must be verified in advance through performance testing.

[0003] When performing performance testing, related technologies can only use logs generated by daily user request behaviors as the data source for test data, and cannot simulate user request behaviors during activities with high fidelity. Summary of the invention

[0004] In view of this, embodiments of the present application provide a test data generation method, device, electronic device, and storage medium to at least solve the problem that related technologies cannot simulate user request behaviors during activities with high fidelity.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides a method for generating test data, the method comprising:

[0007] Based on at least one first request log under each category of at least one set category, generating at least one second request log under the corresponding category;

[0008] generating test data based on all or part of the request logs under each category of at least one category;

[0009] Among them, in the request log that generates the test data, the difference between the first ratio of each category and the corresponding second ratio is less than the first threshold; the first request log is a log generated based on a user access request; the first ratio represents the ratio of the request log under the corresponding category to all request logs; the second ratio represents the ratio of the request log under the set corresponding category to all request logs.

[0010] Among them, in the above scheme, when generating test data based on all or part of the request logs under each category in at least one category, the method includes:

[0011] According to the corresponding second ratio, a corresponding number of request logs for generating test data are determined from all request logs under the category.

[0012] In the above solution, when generating at least one second request log under a corresponding category based on at least one first request log under each category of at least one set category, the method includes:

[0013] Based on at least one first request log under the classification and the corresponding second ratio, a corresponding number of second request logs are generated so that the difference between the first ratio corresponding to the classification and the corresponding second ratio is less than a first threshold.

[0014] In the above solution, when generating at least one second request log under a corresponding category based on at least one first request log under each category of at least one set category, the method includes:

[0015] When the first proportion corresponding to the first category is greater than the corresponding second proportion,

[0016] generating a corresponding second request log under each second category of at least one second category;

[0017] When the first proportion corresponding to the first category is not greater than the corresponding second proportion,

[0018] Generate a corresponding second request log under each category in all categories;

[0019] Among them, the first proportion of the first category is the largest among the first proportions of all categories; the second category is other categories among all categories except the first category.

[0020] In the above solution, when generating at least one second request log under a corresponding category based on at least one first request log under each category of at least one set category, the method includes:

[0021] Mapping at least one first user identifier corresponding to the at least one first request log into at least one second user identifier; the at least one second user identifier is all or part of the user identifiers in a set user identifier set;

[0022] At least one second request log under a corresponding category is generated based on the at least one first request log; wherein each generated second request log corresponds to a second user identifier.

[0023] In the above solution, when generating test data based on all or part of the request logs under each category in at least one category, the method includes:

[0024] When the number of request logs in at least one first array is greater than a set threshold, a corresponding number of request logs are taken out from each first array in the at least one first array according to a corresponding second ratio and output as test data; wherein,

[0025] Each first array in the at least one first array stores all request logs under a category.

[0026] In the above solution, taking out a corresponding number of request logs from each first array in the at least one first array according to the corresponding second ratio and outputting them as test data includes:

[0027] A set number of request logs are taken out from each first array of at least one first array at least once, and the taken request logs are stored in a second array and output as test data.

[0028] In the above solution, taking out a corresponding number of request logs from each first array in the at least one first array according to the corresponding second ratio and outputting them as test data includes:

[0029] creating at least one third array; the number of arrays of the at least one third array is determined according to the number of request logs under the third category; the second ratio of the third category is the smallest among the second ratios of all categories;

[0030] The request logs in each first array are evenly distributed to the at least one third array, and the at least one third array is output as test data.

[0031] In the above solution, when generating at least one second request log under a corresponding category based on at least one first request log under each category of at least one set category, the method includes:

[0032] Based on the first log quantity of the at least one first request log, generating the at least one second request log of the second log quantity according to a set integer multiple;

[0033] At least one request log is deleted under a corresponding category so that a difference between a first ratio of the corresponding category and a corresponding second ratio is smaller than the first threshold.

[0034] The present application also provides a test data generating device, including:

[0035] A first generating unit, configured to generate at least one second request log under a corresponding category based on at least one first request log under each category of at least one set category;

[0036] A second generating unit, configured to generate test data based on all or part of the request logs under each category of at least one category;

[0037] Among them, in the request log that generates the test data, the difference between the first ratio of each category and the corresponding second ratio is less than the first threshold; the first request log is a log generated based on a user access request; the first ratio represents the ratio of the request log under the corresponding category to all request logs; the second ratio represents the ratio of the request log under the set corresponding category to all request logs.

[0038] The present application also provides an electronic device, including: a processor and a memory for storing a computer program that can be run on the processor.

[0039] Wherein, the processor is used to execute the steps of the above-mentioned test data generating method when running the computer program.

[0040] An embodiment of the present application further provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned test data generating method are implemented.

[0041] In the embodiment of the present application, the first request log is a log generated based on a user access request, and at least one second request log under the corresponding category is generated based on at least one first request log under each set category in at least one category, and test data is generated based on all or part of the request logs under each category, and the difference between the proportion of the request logs of each category in the test data and the set proportion is less than the set first threshold. In this way, more request logs are generated through the logs generated by daily user request behavior, and then test data is generated based on part or all of the request logs, so that the proportion of the request logs of each category in the test data matches the proportion of the request logs of the corresponding category during the activity, thereby generating test data that simulates the user request behavior during the real activity, and simulating the user request behavior during the activity with high fidelity. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A flow chart of a test data generation method provided in an embodiment of the present application;

[0043] Figure 2 A flow chart of a test data generation method provided for an application example of the present application;

[0044] Figure 3 A flow chart of a test data generation method provided for an application example of the present application;

[0045] Figure 4 A schematic diagram of the structure of a test data generating device provided in an embodiment of the present application;

[0046] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] During the e-commerce promotion activities, the number of user visits and business volume increased dramatically, which brought severe challenges to the e-commerce platform and put forward higher requirements for the system's high concurrent processing capabilities, large-scale computing capabilities, response speed and stability. In order to ensure the smooth progress of the activities, the following preparations must be completed in advance: estimate the number of user requests received by the background system; plan the system capacity and expand the system; verify whether the system capacity can support the peak traffic during the event. Among them, system capacity verification is a key link, which requires efficient and accurate results to guide the direction of system optimization and assist in the decision-making of system expansion.

[0048] A common method to verify whether the system capacity can support the peak traffic during the event is to perform performance testing on the system. To perform performance testing well, the key is to simulate the user requests received by the backend system during the promotion with high fidelity, that is, to prepare test data that conforms to the user request behavior during the event. For example, the proportion of various user requests must be consistent with the actual scenario, the number of user requests must be guaranteed to reach a certain level, multiple user requests from a single user must ensure a reasonable request order, and the test data must remain valid within the test time.

[0049] When performing performance testing, related technologies can only use logs generated by daily user request behaviors as the data source for test data, and cannot simulate user request behaviors during activities with high fidelity.

[0050] Based on this, in various embodiments of the present application, the first request log is a log generated based on a user access request, and at least one second request log under the corresponding category is generated based on at least one first request log under each set category in at least one category, and test data is generated based on all or part of the request logs under each category, and the difference between the proportion of the request logs of each category in the test data and the set proportion is less than the set first threshold. In this way, more request logs are generated through the logs generated by daily user request behavior, and then test data is generated based on part or all of the request logs, so that the proportion of the request logs of each category in the test data matches the proportion of the request logs of the corresponding category during the activity, thereby generating test data that simulates the user request behavior during the real activity, and simulating the user request behavior during the activity with high fidelity.

[0051] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0052] Figure 1 The following is a flow chart of the test data generation method provided in the embodiment of the present application. Figure 1As shown, the test data generation method includes:

[0053] Step 101: Based on at least one first request log in each category of at least one set category, generate at least one second request log in the corresponding category.

[0054] The first request log is a log generated based on a user access request.

[0055] In this embodiment, a correspondence between the first request log and each of the at least one set category is established, and based on the first request log under each of the at least one category, at least one second request log of the corresponding category is generated. The first request log is a log generated based on a user access request. For example, after the user obtains the search result, he needs to access the content corresponding to the search result, and will request to obtain the content corresponding to the search result, and generate the first request log. By establishing a correspondence between the first request log and the at least one set category, the first request log can be mapped to the at least one set category.

[0056] The setting of at least one category can be a set category such as the user cluster where the user initiating the request is located, the user access channel, the scenario in which the user request is generated (such as daily scenario or big promotion scenario), etc.; it can be determined by extracting the Uniform Resource Locator (URL) in the user request and determining the corresponding parameters in the URL, such as "Client_id=AA" in the URL indicates that the user request comes from the business caller AA, "Client_type=BB" indicates that the user request comes from the user access channel BB, and in actual applications, "Client_type=app" indicates that the user request comes from an application (APP, Application), and "Client_type=pc" indicates that the user request comes from a personal computer (PC, Personal Computer); it is also possible to further extract the URL of the user request on the basis of the set category to expand the category of the user request. In this way, the classification of the request log can be made more accurate, thereby generating more simulated test data.

[0057] Step 102: Generate test data based on all or part of the request logs under each category in at least one category.

[0058] Among them, in the request log that generates the test data, the difference between the first ratio of each category and the corresponding second ratio is less than the first threshold; the first ratio represents the ratio of the request log under the corresponding category to all request logs; the second ratio represents the ratio of the request log under the set corresponding category to all request logs.

[0059] Generate test data based on all or part of the request logs under each category in at least one category, and the difference between the first ratio and the corresponding second ratio of each category is less than the first threshold value. Here, the ratio of request logs under a category to all request logs is the first ratio of this category, and the ratio of request logs under a category to all request logs is set as the second ratio of this category. The set value of the second ratio can be determined by predicting the ratio of request logs in each category during the promotion activity, and the first threshold value can be set according to demand.

[0060] In practical applications, such as Figure 2 As shown in the figure, the first threshold is set to 0.05%, and the request logs are divided into three categories according to the user access channel: wx, key, and shop. The corresponding request log numbers are 3000, 2000, and 5000, respectively. The first ratios are 30%, 20%, and 50%, respectively, and the corresponding second ratios are 24%, 16%, and 60%, respectively. The second request log of the corresponding category is generated according to the first request log under each category. After the request logs are generated, the request log numbers corresponding to the three categories of wx, key, and shop are 12000, 2000, and 5000, respectively. 9000 and 30000. Based on all or part of the request logs under each category, the test data with a data volume of 49998 request logs is generated. The number of request logs corresponding to wx, key, and shop are 11988, 8016, and 29994 respectively. The first proportions corresponding to the three categories are 23.98%, 16.03%, and 59.99% respectively. The difference between the first proportion of each category and the corresponding second proportion is 0.02%, 0.03%, and 0.01% respectively, which are all less than the first threshold of 0.05%.

[0061] In one embodiment, when generating at least one second request log under a corresponding category based on at least one first request log under each category of at least one set category, the method includes:

[0062] Based on at least one first request log under the classification and the corresponding second ratio, a corresponding number of second request logs are generated so that the difference between the first ratio corresponding to the classification and the corresponding second ratio is less than a first threshold.

[0063] Based on the first ratio and the second ratio of each category, the number of second request logs that need to be generated for each category is determined, and the determined number of request logs are generated based on the request logs under this category, so that the difference between the ratio of the number of request logs of this category to the number of all request logs and the second ratio set for this category is less than a first threshold. The first threshold can be set according to demand. In this way, when generating test data, the proportion of each category log in the test data matches the proportion of the request log of the corresponding category during the activity, and the obtained test data can simulate the user request behavior during the activity with high fidelity.

[0064] In actual applications, each request log can be stored in an array. When the data length of the array reaches a set threshold (such as 100 logs), check whether the proportion of request logs in the array is consistent with the second proportion. By generating a classified request log whose corresponding request log proportion is not greater than the second proportion, the proportion of request logs matches the second proportion.

[0065] In one embodiment, when generating at least one second request log under a corresponding category based on at least one first request log under each category of at least one set category, the method includes:

[0066] When the first proportion corresponding to the first category is greater than the corresponding second proportion,

[0067] generating a corresponding second request log under each second category of at least one second category;

[0068] When the first proportion corresponding to the first category is not greater than the corresponding second proportion,

[0069] Generate a corresponding second request log under each category in all categories;

[0070] Among them, the first proportion of the first category is the largest among the first proportions of all categories; the second category is other categories among all categories except the first category.

[0071] In the process of generating request logs, the proportion of request logs of each category is adjusted. First, the category with the most request logs among all categories is determined as the first category, and the categories other than the first category among all categories are determined as the second category.

[0072] If the first ratio of the first category is greater than the second ratio of the category, the number of request logs to be generated for each second category is determined based on the first ratio and the second ratio of each second category, and request logs are generated based on the request logs under each second category, so that the proportion of each category matches the corresponding second ratio, and the magnification of the overall number of request logs after the request logs are generated is calculated. If the ratio of the first category is not greater than the second ratio of the category, the number of request logs to be generated for each category in all categories is determined based on the first ratio and the second ratio of each category in all categories, and the second request log is generated based on the first request log under each category in all categories, so that the proportion of each category matches the corresponding second ratio, and the magnification of the overall number of request logs after the request logs are generated is calculated. For example, the first ratios corresponding to category A, category B, and category C are 50%, 10%, and 40%, respectively, and the corresponding second ratios are 80%, 10%, and 10%, respectively. Since the first ratio of category C is greater than the second ratio, without deleting the request log of category C, the number of request logs of category A is magnified by 6.4 times, and the number of request logs of category B is magnified by 4 times. After the request logs are generated, the first ratios of category A, category B, and category C match the corresponding second ratios respectively.

[0073] In this way, the overall number of request logs is increased without deleting the request logs under the first category, and the request logs under the first category can be used as much as possible to generate test data, thereby reducing the loss of request logs in the process of adjusting the proportions and saving the cost of generating request logs.

[0074] In one embodiment, when generating at least one second request log under a corresponding category based on at least one first request log under each category of at least one set category, the method includes:

[0075] Mapping at least one first user identifier corresponding to the at least one first request log into at least one second user identifier; the at least one second user identifier is all or part of the user identifiers in a set user identifier set;

[0076] At least one second request log under a corresponding category is generated based on the at least one first request log; wherein each generated second request log corresponds to a second user identifier.

[0077] At least one first user identifier corresponding to the at least one first request log is mapped to at least one second user identifier, and a mapping relationship between the first user identifier corresponding to the at least one first request log and the at least one second user identifier is established, wherein the second user identifier is all or part of the user identifiers in the set user identifier set, and at least one second request log under the corresponding category is generated based on the at least one first request log, and the user identifier corresponding to the second request log is the second user identifier determined according to the mapping relationship. Here, the user identifier includes the user ID. In this way, when generating at least one second request log under the corresponding category based on the at least one first request log, it is ensured that the multiple request logs generated based on the request log of one first user identifier correspond to different second user identifiers, that is, the generated request log can simulate the effect of generating request logs by user access requests of different users, thereby generating test data that simulates user behavior during real activities, and simulating user behavior during activities with high fidelity.

[0078] In actual applications, prepare a user ID set of the same order as the actual number of users in the promotion activity, and map a list of numbers from 0 to the number of IDs in the user ID set (1, 2, 3...size-1) to each user ID. When a first request log is used to generate a second request log, convert the first user ID of the first request log into a digital type, and then take the modulus according to the number of second user IDs in the user ID set (size-1). In this way, the first user ID can be mapped to at least one second user ID. For example, a user with user ID A generates three first request logs through user requests, and nine second request logs are generated based on the three first request logs. The nine second request logs are grouped in three, corresponding to the second user IDs B, C, and D, respectively. This can simulate the behavior of the three users B, C, and D to generate the effect of request logs.

[0079] In one embodiment, request logs with the same user ID can be distributed to the same process based on the user ID corresponding to the request log, so that multiple request logs of the same user can be aggregated as much as possible to ensure that when performing performance testing, multiple request logs of the same user can be sent to the background system in sequence, thereby ensuring that the test data highly fits the actual user request sequence.

[0080] In one embodiment, when generating at least one second request log under a corresponding category based on at least one first request log under each category of at least one set category, the method includes:

[0081] Based on the first log quantity of the at least one first request log, generating the at least one second request log of the second log quantity according to a set integer multiple;

[0082] At least one request log is deleted under a corresponding category so that a difference between a first ratio of the corresponding category and a corresponding second ratio is smaller than the first threshold.

[0083] According to the first log quantity of the first request log under each category in at least one category, at least one second request log of the second log quantity is generated according to the set integer multiple of the first log quantity, and at least one request log is deleted under the corresponding category, so that the difference between the first ratio of the corresponding category and the corresponding second ratio is less than the first threshold. Here, the first threshold can be set as needed, and can also be further set to a second threshold that is less than the first threshold. In this way, by deleting a small number of request logs, the proportion of request logs under the category can be accurately adjusted.

[0084] In actual applications, request logs with the same user ID can be distributed to the same process according to the user ID corresponding to the request logs, and the process prunes the request logs to make the first ratio match the second ratio.

[0085] In one embodiment, when generating test data based on all or part of the request logs under each category in at least one category, the method includes:

[0086] According to the corresponding second ratio, a corresponding number of request logs for generating test data are determined from all request logs under the category.

[0087] According to the second ratio corresponding to the category, a corresponding number of request logs are determined from all request logs under the category for generating test data. In this way, according to the determined number, test data is generated based on the request logs under each category, and the proportion of request logs of each category for generating test data matches the proportion of request logs of the corresponding category during the activity, thereby generating test data that simulates user behavior during the real activity and simulating user behavior during the activity with high fidelity.

[0088] In one embodiment, when generating test data based on all or part of the request logs under each category in at least one category, the method includes:

[0089] When the number of request logs in at least one first array is greater than a set threshold, taking out a corresponding number of request logs from each first array in the at least one first array according to a corresponding second ratio, and outputting them as test data;

[0090] Each of the at least one first array stores correspondingly all request logs under a category.

[0091] When the number of request logs in at least one first array is greater than a set threshold, a corresponding number of request logs are taken out from each first array in the at least one first array according to the second ratio corresponding to each category, and output as test data. Here, each first array in the at least one first array stores all request logs under a category, and a set threshold is set as needed, such as 1000.

[0092] In this way, by taking out the request logs in the array according to the second ratio and evenly inserting them into an array, the request logs are evenly dispersed in the array, improving the phenomenon of clustering of request logs of the same category. Using such test data can make the system performance more realistic, thereby simulating the user request behavior during the activity with high fidelity.

[0093] In one embodiment, taking out a corresponding number of request logs from each first array of at least one first array according to a corresponding second ratio and outputting them as test data includes:

[0094] A set number of request logs are taken out from each first array of at least one first array at least once, and the taken request logs are stored in a second array and output as test data.

[0095] Here, at least once from each first array in at least one first array, a set number of request logs are taken out, and the taken out request logs are stored in a second array, and the second array is output as test data. The set number can be determined by counting the number of request logs of each category in at least one first array, determining the proportion of each category, and determining the number of request logs taken out each time according to the proportion. In this way, by taking a set number of request logs from the first array at least once and inserting them into the second array, the request logs are evenly dispersed within the distribution range of the test data, improving the phenomenon of clustering of request logs of the same category, and using such test data can make the system performance more realistic, thereby simulating the user request behavior during the activity with high fidelity.

[0096] When there is a phenomenon that the number of request logs sent to different categories is very different (for example, the category with the least number of logs has only 1 / 100 of the number of request logs of the category with the most number of logs), the request logs of the category with the least number of logs will be concentrated at the beginning of the second array, that is, a clustering phenomenon will occur. Therefore, in one embodiment, the corresponding number of request logs are taken out from each first array of at least one first array according to the corresponding second ratio, and output as test data, including:

[0097] creating at least one third array; the number of arrays of the at least one third array is determined according to the number of request logs under the third category; the second ratio of the third category is the smallest among the second ratios of all categories;

[0098] The request logs in each first array are evenly distributed to the at least one third array, and the at least one third array is output as test data.

[0099] According to the number of request logs in the category with the smallest second ratio among all categories, the number of arrays of the third array is determined, and the third array with the determined number of arrays is created, and the request logs in each first array are evenly distributed to each third array. In this way, according to the number of arrays of the third array determined by the number of request logs with the smallest second ratio, the number of request logs of each category to be distributed to each third array is determined, and then the request logs are dispersedly stored in the third array to ensure that the requests of each category in the array finally output are evenly distributed.

[0100] For example, the request logs are classified into A, B, and C, and the corresponding numbers of request logs are 1000, 4000, and 5000 respectively. The corresponding second ratios are 10%, 40%, and 50%. The category with the smallest second ratio is A. The number of arrays of the third array is determined to be 5 according to the number of request logs under category A. The request logs in each first array are evenly distributed to the third array. The number of request logs of category A, category B, and category C stored in each third array is 200, 800, and 1000 respectively. Then, according to the number determined for each category, the request logs of category A, category B, and category C are dispersedly distributed to each third array.

[0101] The present application is further described in detail below in conjunction with application examples.

[0102] Combination Figure 3 ,The test data generation method includes the following steps:

[0103] S1: The backend system publishes the request log.

[0104] After the backend system publishes the request log, it is stored in a high-throughput distributed publish-subscribe messaging system in a real-time asynchronous manner. Since data requests are stored in the publish-subscribe messaging system in real time and centrally, and are implemented through third-party components, the publish-subscribe messaging system is required to ensure that the order of processing subscription messages is consistent with the order of publishing messages, so the order of data subscribed from the publish-subscribe messaging system is sequence-preserving.

[0105] In actual applications, Kafka is used as a publish-subscribe messaging system. It is an open source, high-throughput distributed publish-subscribe messaging system that can process all action stream data of consumers on the website. This action (web browsing, search and other user actions) is ultimately stored in Kafka in the form of request logs, which are stored in string format. The entry module of the system acts as a message publisher for Kafka, and publishes the request logs to Kafka asynchronously and one by one. The request log includes: the content of the request to obtain the message, the time when the request occurred, and can also add additional information such as the user cluster where the requesting user is located, the user access channel, and the scenario in which the user request is generated (such as daily scenarios or promotion scenarios) to help classify user requests.

[0106] S2: Data warehouse subscription message.

[0107] Store the request logs subscribed to through the message system into the data warehouse. The data warehouse is required to have the following functions: based on a distributed file system (to improve the efficiency of reading and writing massive request logs); map structured data files to a database table (using SQL queries to improve the efficiency of code development); be able to aggregate storage based on a column of the data table (to improve reading and sorting efficiency); be able to extract, transform, load, store, query and analyze data from the table. In actual operation, each day's log is a partition, and the time when the request occurs is used as the index for aggregate storage.

[0108] In actual applications, Hive is used as a data warehouse to subscribe to messages in Kafka in real time; generally, it will save the request logs of the last three days for use in production test data. Hive is a data warehouse tool based on Hadoop, which is used for data extraction, transformation, and loading. It is a mechanism for storing, querying, and analyzing large-scale data stored in Hadoop. The Hive data warehouse tool can map structured data files to a database table and provide SQL query functions.

[0109] Using the three types of data models in Hive: table, partition, and bucket, request logs are stored in the same table, with cluster and date as the partition keywords, and stored in aggregate on a daily basis. The data of each partition is stored in multiple buckets, which are distributed on multiple servers to improve storage and reading efficiency.

[0110] S3: retrieve and sort request logs on time within the hour.

[0111] Take the request logs of the same hour from the table, sort them in ascending order according to hour, minute, and second, and store the results in the distributed file system. Since the storage is aggregated by hour, only the requests within the hour are sorted by time, so the sorting granularity is 1 / 2 to 1 / 10 of the existing technology. Therefore, the data query and sorting time can be controlled within 1-3 hours, improving the production efficiency of test data.

[0112] When producing test data, take out the request logs of the same hour, sort them in ascending order according to hour, minute and second, and store the results in Hadoop Distributed File System (HDFS). HDFS is a distributed file system designed to run on general-purpose hardware, which can provide high-throughput data access and is very suitable for large-scale test data production applications.

[0113] In the specific application, the data from 9:00 to 11:00 on September 6th is queried by hour through the Structured Query Language (SQL) method, and sorted by hour, minute and second respectively:

[0114] select t.url,t.urltype,t.hour,t.time from(select tab.url,tab.urltype,unix_timestamp(tab.time_info)as time,hour(tab.time_info)as hour from request_table as tab where tab.dt='2020-09-06'and tab.cluster in('ht')and hour(tab.time_info)in('9','10','11'))t distribute by t.hour sort by t.hour,t.timeasc

[0115] S4: The proportion of request logs under the adjustment category to all request logs.

[0116] Map the user request logs into different categories and classify the request logs according to the caller and the user access channel. Count the proportions of these categories, and generate request logs for categories whose request log proportions are not greater than the second proportion, so that the proportions of the categories corresponding to the request logs match the second proportion. Based on the request logs of each category, accurately output the test data according to the second proportion.

[0117] In addition, in a distributed file system, files are stored in multiple file blocks, and the request log is read and processed in the order of the file blocks, so the processing results are still in order.

[0118] MapReduce implemented using Hadoop. MapReduce is a computing model, framework and platform for parallel processing of big data. It implies the following three meanings: a high-performance parallel computing platform based on clusters; a parallel computing and operation software framework; and a parallel programming model and method.

[0119] S5: Ensure the order of user requests and adjust the proportion.

[0120] In practical applications, request logs with the same user ID can be distributed to the same process for processing according to the user ID corresponding to the request log, and the request logs can be slightly pruned through the process to accurately adjust the first ratio to match the second ratio.

[0121] Among them, S4 and S5 can generate test data that simulates user request behavior during real activities by generating request logs, adjusting the proportion of request logs, and ensuring the order of multiple requests from the same user, thereby simulating user request behavior during the activity with high fidelity.

[0122] In order to implement the method of the embodiment of the present application, the embodiment of the present application also provides a test data generating device, such as Figure 4 As shown, the device comprises:

[0123] The first generating unit 401 is configured to generate at least one second request log under a corresponding category based on at least one first request log under each category of at least one set category;

[0124] A second generating unit 402 is used to generate test data based on all or part of the request logs under each category of at least one category;

[0125] Among them, in the request log that generates the test data, the difference between the first ratio of each category and the corresponding second ratio is less than the first threshold; the first request log is a log generated based on a user access request; the first ratio represents the ratio of the request log under the corresponding category to all request logs; the second ratio represents the ratio of the request log under the set corresponding category to all request logs.

[0126] In one embodiment, the second generating unit 402 is further configured to:

[0127] According to the corresponding second ratio, a corresponding number of request logs for generating test data are determined from all request logs under the category.

[0128] In one embodiment, the first generating unit 401 is used to:

[0129] Based on at least one first request log under the classification and the corresponding second ratio, a corresponding number of second request logs are generated so that the difference between the first ratio corresponding to the classification and the corresponding second ratio is less than a first threshold.

[0130] In one embodiment, the first generating unit 401 is used to:

[0131] When the first proportion corresponding to the first category is greater than the corresponding second proportion,

[0132] generating a corresponding second request log under each second category of at least one second category;

[0133] When the first proportion corresponding to the first category is not greater than the corresponding second proportion,

[0134] Generate a corresponding second request log under each category in all categories;

[0135] Among them, the first proportion of the first category is the largest among the first proportions of all categories; the second category is other categories among all categories except the first category.

[0136] In one embodiment, the first generating unit 401 is used to:

[0137] Mapping at least one first user identifier corresponding to the at least one first request log into at least one second user identifier; the at least one second user identifier is all or part of the user identifiers in a set user identifier set;

[0138] At least one second request log under a corresponding category is generated based on the at least one first request log; wherein each generated second request log corresponds to a second user identifier.

[0139] In one embodiment, the second generating unit 402 is configured to:

[0140] When the number of request logs in at least one first array is greater than a set threshold, a corresponding number of request logs are taken out from each first array in the at least one first array according to a corresponding second ratio and output as test data; wherein,

[0141] Each first array in the at least one first array stores all request logs under a category.

[0142] In one embodiment, the second generating unit 402 is configured to:

[0143] A set number of request logs are taken out from each first array of at least one first array at least once, and the taken request logs are stored in a second array and output as test data.

[0144] In one embodiment, the second generating unit 402 is configured to:

[0145] creating at least one third array; the number of arrays of the at least one third array is determined according to the number of request logs under the third category; the second ratio of the third category is the smallest among the second ratios of all categories;

[0146] The request logs in each first array are evenly distributed to the at least one third array, and the at least one third array is output as test data.

[0147] In one embodiment, the first generating unit 401 is used to:

[0148] Based on the first log quantity of the at least one first request log, generating the at least one second request log of the second log quantity according to a set integer multiple;

[0149] At least one request log is deleted under a corresponding category so that a difference between a first ratio of the corresponding category and a corresponding second ratio is smaller than the first threshold.

[0150] In actual application, the first generation unit 401 and the second generation unit 402 can be implemented by a processor in a test data generation device, such as a central processing unit (CPU), a digital signal processor (DSP), a microcontroller unit (MCU) or a programmable gate array (FPGA).

[0151] It should be noted that: the test data generation device provided in the above embodiment only uses the division of the above program modules as an example when generating test data. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device is divided into different program modules to complete all or part of the processing described above. In addition, the test data generation device provided in the above embodiment and the test data generation method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0152] Based on the hardware implementation of the above program modules, and in order to implement the method for generating test data in the embodiment of the present application, the embodiment of the present application also provides an electronic device, such as Figure 5As shown, the electronic device 500 includes:

[0153] Communication interface 510, capable of exchanging information with other devices such as network devices;

[0154] The processor 520 is connected to the communication interface 510 to implement information exchange with other devices and is used to execute the method provided by one or more technical solutions when running a computer program. The computer program is stored in the memory 530.

[0155] Specifically, the processor 520 is configured to:

[0156] Based on at least one first request log under each category of at least one set category, generating at least one second request log under the corresponding category;

[0157] generating test data based on all or part of the request logs under each category of at least one category;

[0158] Among them, in the request log that generates the test data, the difference between the first ratio of each category and the corresponding second ratio is less than the first threshold; the first request log is a log generated based on a user access request; the first ratio represents the ratio of the request log under the corresponding category to all request logs; the second ratio represents the ratio of the request log under the set corresponding category to all request logs.

[0159] In one embodiment, the processor 520 is configured to:

[0160] According to the corresponding second ratio, a corresponding number of request logs for generating test data are determined from all request logs under the category.

[0161] In one embodiment, the processor 520 is configured to:

[0162] Based on at least one first request log under the classification and the corresponding second ratio, a corresponding number of second request logs are generated so that the difference between the first ratio corresponding to the classification and the corresponding second ratio is less than a first threshold.

[0163] In one embodiment, the processor 520 is configured to:

[0164] When the first proportion corresponding to the first category is greater than the corresponding second proportion,

[0165] generating a corresponding second request log under each second category of at least one second category;

[0166] When the first proportion corresponding to the first category is not greater than the corresponding second proportion,

[0167] Generate a corresponding second request log under each category in all categories;

[0168] Among them, the first proportion of the first category is the largest among the first proportions of all categories; the second category is other categories among all categories except the first category.

[0169] In one embodiment, the processor 520 is configured to:

[0170] Mapping at least one first user identifier corresponding to the at least one first request log into at least one second user identifier; the at least one second user identifier is all or part of the user identifiers in a set user identifier set;

[0171] At least one second request log under a corresponding category is generated based on the at least one first request log; wherein each generated second request log corresponds to a second user identifier.

[0172] In one embodiment, the processor 520 is configured to:

[0173] When the number of request logs in at least one first array is greater than a set threshold, a corresponding number of request logs are taken out from each first array in the at least one first array according to a corresponding second ratio and output as test data; wherein,

[0174] Each first array in the at least one first array stores all request logs under a category.

[0175] In one embodiment, the processor 520 is configured to:

[0176] A set number of request logs are taken out from each first array of at least one first array at least once, and the taken request logs are stored in a second array and output as test data.

[0177] In one embodiment, the processor 520 is configured to:

[0178] creating at least one third array; the number of arrays of the at least one third array is determined according to the number of request logs under the third category; the second ratio of the third category is the smallest among the second ratios of all categories;

[0179] The request logs in each first array are evenly distributed to the at least one third array, and the at least one third array is output as test data.

[0180] In one embodiment, the processor 520 is configured to:

[0181] Based on the first log quantity of the at least one first request log, generating the at least one second request log of the second log quantity according to a set integer multiple;

[0182] At least one request log is deleted under a corresponding category so that a difference between a first ratio of the corresponding category and a corresponding second ratio is smaller than the first threshold.

[0183] Of course, in actual application, the various components in the electronic device 500 are coupled together through the bus system 540. It can be understood that the bus system 540 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 5 Various buses are labeled as bus system 540 .

[0184] The memory 530 in the embodiment of the present application is used to store various types of data to support the operation of the electronic device 500. Examples of such data include: any computer program used to operate on the electronic device 500.

[0185] It can be understood that the memory 530 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAM bus random access memory (DRRAM, Direct Rambus Random Access Memory).The memory 530 described in the embodiments of the present application is intended to include but is not limited to these and any other suitable types of memories.

[0186] The method disclosed in the above embodiment of the present application can be applied to the processor 520, or implemented by the processor 520. The processor 520 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 520 or the instruction in the form of software. The above processor 520 can be a general-purpose processor, a DSP, or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 520 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present application, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in the memory 530, and the processor 520 reads the program in the memory 530 and completes the steps of the above method in combination with its hardware.

[0187] Optionally, when the processor 520 executes the program, the corresponding processes of the various methods of the embodiments of the present application are implemented, which will not be repeated here for the sake of brevity.

[0188] In an exemplary embodiment, the present application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, for example, a memory 530 storing a computer program, and the computer program can be executed by a processor 520 of an electronic device to complete the steps of the aforementioned method. The computer-readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface storage, optical disk, or CD-ROM.

[0189] In the several embodiments provided in the present application, it should be understood that the disclosed devices, electronic devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0190] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0191] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0192] A person of ordinary skill in the art can understand that: all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium, which, when executed, executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, disks or optical disks.

[0193] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0194] It should be noted that the technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict. Unless otherwise specified and limited, the term "connection" should be understood in a broad sense, for example, it can be an electrical connection, or it can be the internal connection of two components, it can be a direct connection, or it can be an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0195] In addition, in the examples of the present application, "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the objects distinguished by "first\second\third" can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein.

[0196] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0197] The various specific technical features in the various embodiments described in the specific implementation methods can be combined in various ways without contradiction. For example, different implementation methods can be formed by combining different specific technical features. In order to avoid unnecessary repetition, the various possible combinations of the specific technical features in this application will not be described separately.

Claims

1. A test data generation method, characterized in that: The method comprises: Based on at least one first request log under each category of at least one set category, at least one second request log under the corresponding category is generated; wherein, the one category represents the category of the request log, and when the at least one second request log under each category of at least one set category is generated, the method includes: based on the at least one first request log under the category and the corresponding second ratio, a corresponding number of second request logs are generated, so that the difference between the first ratio corresponding to the category and the corresponding second ratio is less than a first threshold value; when the first ratio corresponding to the first category is greater than the corresponding second ratio, a corresponding second request log is generated under each second category in at least one second category; when the first ratio corresponding to the first category is not greater than the corresponding second ratio, a corresponding second request log is generated under each category in all categories; wherein, the first ratio of the first category is the largest among the first ratios of all categories; the second category is other categories among all categories except the first category; generating test data based on all or part of the request logs under each category of at least one category; Among them, in the request log that generates the test data, the difference between the first ratio of each category and the corresponding second ratio is less than the first threshold; the first request log is a log generated based on a user access request; the first ratio represents the ratio of the request log under the corresponding category to all request logs; the second ratio represents the ratio of the request log under the set corresponding category to all request logs.

2. The test data generation method according to claim 1, characterized in that: When generating test data based on all or part of the request logs under each category in at least one category, the method includes: According to the corresponding second ratio, a corresponding number of request logs for generating test data are determined from all request logs under the category.

3. The test data generating method according to claim 1, characterized in that: When generating at least one second request log under a corresponding category based on at least one first request log under each category of at least one set category, the method includes: Mapping at least one first user identifier corresponding to the at least one first request log into at least one second user identifier; the at least one second user identifier is all or part of the user identifiers in a set user identifier set; At least one second request log under a corresponding category is generated based on the at least one first request log; wherein each generated second request log corresponds to a second user identifier.

4. The test data generating method according to claim 1, characterized in that: When generating test data based on all or part of the request logs under each category in at least one category, the method includes: When the number of request logs in at least one first array is greater than a set threshold, taking out a corresponding number of request logs from each first array in the at least one first array according to a corresponding second ratio, and outputting them as test data; Each of the at least one first array stores correspondingly all request logs under a category.

5. The test data generating method according to claim 4, characterized in that: The taking out a corresponding number of request logs from each first array in the at least one first array according to the corresponding second ratio and outputting them as test data includes: A set number of request logs are taken out from each first array of at least one first array at least once, and the taken request logs are stored in a second array and output as test data.

6. The test data generating method according to claim 4, characterized in that: The taking out a corresponding number of request logs from each first array in the at least one first array according to the corresponding second ratio and outputting them as test data includes: creating at least one third array; the number of arrays of the at least one third array is determined according to the number of request logs under the third category; the second ratio of the third category is the smallest among the second ratios of all categories; The request logs in each first array are evenly distributed to the at least one third array, and the at least one third array is output as test data.

7. The test data generating method according to claim 1, characterized in that: When generating at least one second request log under a corresponding category based on at least one first request log under each category of at least one set category, the method includes: Based on the first log quantity of the at least one first request log, generating the at least one second request log of the second log quantity according to a set integer multiple; At least one request log is deleted under a corresponding category so that a difference between a first ratio of the corresponding category and a corresponding second ratio is smaller than the first threshold.

8. A test data generating device, characterized in that: include: A first generating unit is configured to generate at least one second request log under a corresponding category based on at least one first request log under each category of at least one set category; wherein the one category represents the category of the request log, and when generating at least one second request log under a corresponding category based on at least one first request log under each category of at least one set category, the method comprises: generating a corresponding number of second request logs based on at least one first request log under the category and a corresponding second ratio, so that a difference between a first ratio corresponding to the category and a corresponding second ratio is less than a first threshold value; generating a corresponding second request log under each second category in at least one second category when a first ratio corresponding to the first category is greater than a corresponding second ratio; generating a corresponding second request log under each category in all categories when a first ratio corresponding to the first category is not greater than a corresponding second ratio; wherein the first ratio of the first category is the largest among the first ratios of all categories; and the second category is other categories in all categories except the first category; A second generating unit, configured to generate test data based on all or part of the request logs under each category of at least one category; Among them, in the request log that generates the test data, the difference between the first ratio of each category and the corresponding second ratio is less than the first threshold; the first request log is a log generated based on a user access request; the first ratio represents the ratio of the request log under the corresponding category to all request logs; the second ratio represents the ratio of the request log under the set corresponding category to all request logs.

9. An electronic device, characterized in that: include: a processor and a memory for storing a computer program capable of being executed on the processor, Wherein, when the processor is used to run the computer program, it executes the steps of the method described in any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images