Data processing method, device, electronic device, storage medium and program product

Abstract

The disclosed embodiment of the present invention discloses a data processing method, device, electronic device, storage medium and program product, the data processing method comprising: receiving a test data request; determining a target test data packet category according to the test data request; taking out the target test data packet from a test data packet storage component corresponding to the target test data packet category and sending it to the test data requester, wherein the test data packet storage component stores one or more test data packets, and the test data packet storage component has a category corresponding to the stored test data packet category. This technical solution can achieve stable acquisition and long-term validity of test data packets, as well as effective generation and management of test data, which not only greatly improves the execution efficiency of data, but also avoids the problem of data unavailability due to data expiration, corruption and data pollution.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a data processing method, device, electronic device, storage medium and program product. Background Art

[0002] Preparation of test data is an important step in software testing. There is an important design principle for test cases: there should be no dependencies between test cases, and the execution result of a test case should not be affected by the execution results of other test cases. Based on this principle, all the test data required by a test case should be prepared by itself. However, since the preparation of test data mostly involves multiple data sources, and the availability of data sources cannot be guaranteed, this greatly reduces the efficiency of data execution, especially for software systems with many types of test data and strong correlations between data. If pre-prepared static test data is used, it will face the problem of data expiration and corruption and data pollution caused by multiple use. Therefore, there is an urgent need for a solution that can not only improve the execution efficiency of data, but also effectively avoid data unavailability due to data expiration and corruption and data pollution. Summary of the invention

[0003] Embodiments of the present disclosure provide a data processing method, device, electronic device, storage medium, and program product.

[0004] In a first aspect, an embodiment of the present disclosure provides a data processing method.

[0005] Specifically, the data processing method includes:

[0006] receiving a test data request;

[0007] Determine a target test data packet category according to the test data request;

[0008] The target test data packet is taken out from a test data packet storage component corresponding to the target test data packet category and sent to a test data requester, wherein the test data packet storage component stores one or more test data packets and the test data packet storage component has a category corresponding to the stored test data packet category.

[0009] In combination with the first aspect, the present disclosure, in a first implementation of the first aspect, further includes:

[0010] A test data packet is generated, and the test data packet is stored in a test data packet storage component corresponding to the category of the test data packet.

[0011] In combination with the first aspect and the first implementation of the first aspect, in a second implementation of the first aspect of the embodiment of the present disclosure, generating a test data packet includes:

[0012] The test data is acquired from a data source corresponding to the category of the test data packet and its associated data source to form a test data packet.

[0013] In combination with the first aspect, the first implementation manner of the first aspect, and the second implementation manner of the first aspect, in a third implementation manner of the first aspect of the embodiment of the present disclosure, the acquiring of test data from a data source corresponding to the category of the test data packet and its associated data source to form a test data packet includes:

[0014] Acquire identification information of the test data, and form a test data packet with the identification information of the test data;

[0015] Storing the data ontology of the test data in a data ontology database;

[0016] A corresponding relationship between the data ontology of the test data and the identification information is established.

[0017] In combination with the first aspect, the first implementation of the first aspect, the second implementation of the first aspect, and the third implementation of the first aspect, the embodiment of the present disclosure, in a fourth implementation of the first aspect, further includes:

[0018] In response to detecting that the number of remaining test data packets in the test data storage component is lower than a first preset number threshold, a test data packet corresponding to the test data storage component category is generated and filled into the test data storage component so that the number of remaining test data packets in the test data storage component is higher than the first preset number threshold.

[0019] In combination with the first aspect, the first implementation of the first aspect, the second implementation of the first aspect, the third implementation of the first aspect, and the fourth implementation of the first aspect, the embodiment of the present disclosure, in a fifth implementation of the first aspect, further includes:

[0020] The target test data packet is marked as used, and the used test data packet is migrated to a used test data packet storage component.

[0021] In combination with the first aspect, the first implementation of the first aspect, the second implementation of the first aspect, the third implementation of the first aspect, the fourth implementation of the first aspect, and the fifth implementation of the first aspect, the embodiment of the present disclosure, in a sixth implementation of the first aspect, further includes:

[0022] In response to detecting that a data clearing condition is met, used test data packets in the used test data packet storage component are cleared.

[0023] In combination with the first aspect, the first implementation of the first aspect, the second implementation of the first aspect, the third implementation of the first aspect, the fourth implementation of the first aspect, the fifth implementation of the first aspect, and the sixth implementation of the first aspect, the embodiment of the present disclosure, in a seventh implementation of the first aspect, further includes:

[0024] The data ontology database is cleaned up with data ontologies corresponding to identification information in the cleaned used test data packets.

[0025] In a second aspect, a data processing device is provided in an embodiment of the present disclosure.

[0026] Specifically, the data processing device includes:

[0027] A receiving module, configured to receive a test data request;

[0028] A determination module, configured to determine a target test data packet category according to the test data request;

[0029] The sending module is configured to take out the target test data packet from the test data packet storage component corresponding to the target test data packet category and send it to the test data requester, wherein the test data packet storage component stores one or more test data packets, and the test data packet storage component has a category corresponding to the stored test data packet category.

[0030] In conjunction with the second aspect, the present disclosure, in a first implementation of the second aspect, further includes:

[0031] The generating module is configured to generate a test data packet and store the test data packet into a test data packet storage component corresponding to the category of the test data packet.

[0032] In combination with the second aspect and the first implementation of the second aspect, in a second implementation of the second aspect of the embodiment of the present disclosure, the generating module is configured as follows:

[0033] The test data is acquired from a data source corresponding to the category of the test data packet and its associated data source to form a test data packet.

[0034] In combination with the second aspect, the first implementation of the second aspect, and the second implementation of the second aspect, in a third implementation of the second aspect of the embodiment of the present disclosure, the part of acquiring test data from a data source corresponding to the category of the test data packet and its associated data source to form the test data packet is configured as follows:

[0035] Acquire identification information of the test data, and form a test data packet with the identification information of the test data;

[0036] Storing the data ontology of the test data in a data ontology database;

[0037] A corresponding relationship between the data ontology of the test data and the identification information is established.

[0038] In combination with the second aspect, the first implementation of the second aspect, the second implementation of the second aspect, and the third implementation of the second aspect, the embodiment of the present disclosure, in a fourth implementation of the second aspect, further includes:

[0039] A filling module is configured to generate test data packets corresponding to the category of the test data storage component in response to detecting that the number of remaining test data packets in the test data storage component is lower than a first preset number threshold, and fill the test data packets into the test data storage component so that the number of remaining test data packets in the test data storage component is higher than the first preset number threshold.

[0040] In combination with the second aspect, the first implementation of the second aspect, the second implementation of the second aspect, the third implementation of the second aspect, and the fourth implementation of the second aspect, the embodiment of the present disclosure, in a fifth implementation of the second aspect, further includes:

[0041] The migration module is configured to mark the target test data packet as used and migrate the used test data packet to the used test data packet storage component.

[0042] In combination with the second aspect, the first implementation of the second aspect, the second implementation of the second aspect, the third implementation of the second aspect, the fourth implementation of the second aspect, and the fifth implementation of the second aspect, the embodiment of the present disclosure, in a sixth implementation of the second aspect, further includes:

[0043] The cleaning module is configured to clean up the used test data packets in the used test data packet storage component in response to detecting that a data cleaning condition is met.

[0044] In combination with the second aspect, the first implementation manner of the second aspect, the second implementation manner of the second aspect, the third implementation manner of the second aspect, the fourth implementation manner of the second aspect, the fifth implementation manner of the second aspect, and the sixth implementation manner of the second aspect, in a seventh implementation manner of the second aspect of the embodiment of the present disclosure, the cleaning module is further configured as follows:

[0045] The data ontology database is cleaned up with data ontologies corresponding to identification information in the cleaned used test data packets.

[0046] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising a memory and at least one processor, wherein the memory is used to store one or more computer instructions, and wherein the one or more computer instructions are executed by the at least one processor to implement the method steps of the above-mentioned data processing method.

[0047] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium for storing computer instructions used by a data processing device, which includes computer instructions involved in the data processing device for executing the above-mentioned data processing method.

[0048] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program / instructions, wherein the computer program / instructions, when executed by a processor, implement the method steps of the above-mentioned data processing method.

[0049] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0050] The above technical solution pre-sets multiple different categories of test data packet storage components to store different categories of test data packets, so as to achieve the decoupling of data production and consumption, and manage the validity of the test data packets in the test data packet storage components. This technical solution can achieve the stable acquisition and long-term validity of test data packets, as well as the effective generation and management of test data, which not only greatly improves the execution efficiency of data, but also avoids the problem of data unavailability due to data expiration, corruption and data pollution.

[0051] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Other features, objectives and advantages of the present disclosure will become more apparent through the following detailed description of non-limiting embodiments in conjunction with the accompanying drawings. In the accompanying drawings:

[0053] Figure 1 A flowchart showing a data processing method according to an embodiment of the present disclosure;

[0054] Figure 2 An overall flow chart of a data processing method according to an embodiment of the present disclosure is shown;

[0055] Figure 3 A structural block diagram of a data processing device according to an embodiment of the present disclosure is shown;

[0056] Figure 4 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown;

[0057] Figure 5 It is a structural diagram of a computer system suitable for implementing a data processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0058] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement them. In addition, for the sake of clarity, parts not related to the description of the exemplary embodiments are omitted in the accompanying drawings.

[0059] In the present disclosure, it should be understood that terms such as "include" or "have" are intended to indicate the presence of features, numbers, steps, behaviors, components, parts, or a combination thereof disclosed in the present specification, and are not intended to exclude the possibility that one or more other features, numbers, steps, behaviors, components, parts, or a combination thereof exist or are added.

[0060] It should also be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0061] The technical solution provided by the embodiment of the present disclosure pre-sets a plurality of different categories of test data packet storage components to store different categories of test data packets, so as to achieve the decoupling of data production and consumption, and to manage the validity of the test data packets in the test data packet storage components. The technical solution can achieve the stable acquisition and long-term validity of the test data packets, as well as the effective generation and management of the test data, which not only greatly improves the execution efficiency of the data, but also avoids the problem of data unavailability due to data expiration, corruption and data pollution.

[0062] Figure 1 A flow chart showing a data processing method according to an embodiment of the present disclosure is shown as follows: Figure 1 As shown, the data processing method includes the following steps S101-S103:

[0063] In step S101, a test data request is received;

[0064] In step S102, a target test data packet category is determined according to the test data request;

[0065] In step S103, the target test data packet is taken out from a test data packet storage component corresponding to the target test data packet category and sent to a test data requester, wherein the test data packet storage component stores one or more test data packets and the test data packet storage component has a category corresponding to the stored test data packet category.

[0066] As mentioned above, the preparation of test data is an important step in software testing. There is an important design principle for test cases: there should be no dependencies between test cases, and the execution results of a test case should not be affected by the execution results of other test cases. Based on this principle, all the test data required for a test case should be prepared by itself, but since the preparation of test data mostly involves multiple data sources, and the availability of data sources cannot be guaranteed, this greatly reduces the efficiency of data execution, especially for software systems with many types of test data and strong correlations between data. If pre-prepared static test data is used, there will be problems of data expiration and corruption and data pollution caused by multiple use. Therefore, there is an urgent need for a solution that can not only improve the execution efficiency of data, but also effectively avoid data unavailability due to data expiration and corruption and data pollution.

[0067] In view of the above defects, in this embodiment, a data processing method is proposed, which pre-sets a plurality of different categories of test data packet storage components to store different categories of test data packets, so as to achieve the decoupling of data production and consumption, and manage the validity of the test data packets in the test data packet storage components. This technical solution can achieve the stable acquisition and long-term validity of test data packets, as well as the effective generation and management of test data, which not only greatly improves the execution efficiency of data, but also avoids the problem of data unavailability due to data expiration, corruption and data pollution.

[0068] In one embodiment of the present disclosure, the data processing method may be applicable to a test data provider such as a computer, a computing device, an electronic device, a server, etc. that processes the test data.

[0069] In one embodiment of the present disclosure, the test data request refers to a request for obtaining test data, which is sent by the test data requester to the test data provider and carries the specific demand for obtaining test data, wherein the test data request may carry information such as the category and quantity of the target test data packet to be obtained, and the category and quantity of the target test data packet to be obtained by the test data requester may be determined based on the test data request. The test data request may be obtained through a unified data interface provided to the outside, that is, the test data requester may send the test data request through the unified data interface provided to the outside by the test data provider.

[0070] In one embodiment of the present disclosure, the test data packet storage component refers to a component for storing one or more test data packets obtained through production, wherein the test data packet storage component has a category corresponding to the category of the stored test data packets. For example, in e-commerce scenarios such as order payment, merchant data, product data, marketing data, store data, user data, etc. may be used. Then, the categories of test data packets that may be used in e-commerce scenarios may include: merchants, products, marketing, stores, users, etc., and test data packets of a certain category may be stored in a test data packet storage component of the corresponding category. That is to say, the category of the test data packet storage component is consistent with the category of the test data packet, that is, the category of the test data packet storage component may include: merchants, products, marketing, stores, users, etc., and each category of the test data packet storage component only stores test data packets of the corresponding category, which will improve the targeted acquisition of test data packets and thereby increase the speed of acquiring test data packets.

[0071] As can be seen from the above, the categories of available test data packets can be one or more, the categories of test data packet storage components used to store the test data packets can be one or more, and for a certain test data request, the categories of the target test data packets carried by the request to be obtained can also be one or more.

[0072] In the above embodiment, after receiving the test data request sent by the test data requester, the category of the target test data packet that the test data requester wants to obtain is determined based on the test data request, and then the target test data packets are taken out from the test data packet storage components of the corresponding categories and sent to the test data requester.

[0073] In one embodiment of the present disclosure, the method may further include the following steps:

[0074] A test data packet is generated, and the test data packet is stored in a test data packet storage component corresponding to the category of the test data packet.

[0075] In order to enable the test data requester to request to obtain the test data packet when he wants, to ensure the stability of the provision of the test data packet, and to improve the targeted acquisition of the test data packet, thereby improving the speed of obtaining the test data packet, in this embodiment, it is necessary to generate a certain number of test data packets of each category in advance, and store these test data packets in a test data packet storage component corresponding to the category of the test data packet, in preparation for subsequent test data requesters to request to obtain, that is, a test data packet storage component only stores one category of test data packets, such as a product test data packet storage component only stores product test data packets, and a store test data packet storage component only stores store test data packets.

[0076] Furthermore, considering that the test data packet has a validity period, the validity periods of different test data packets may be different. Therefore, when the test data packet is stored in the test data packet storage component of the corresponding category, the validity period information of the test data packet can be stored at the same time. In addition, in order to ensure that each test data packet can be used within its validity period, when the test data packet is called, the test data packet with a near validity period can be provided first. For example, assuming that the minimum validity period of a certain type of test data packet is 3 days, there are 120 test data packets stored in the corresponding test data packet storage component, of which 40 test data packets were created 2 days ago and are about to expire. They can be marked as Class A test data packets; 40 commodities were created 1 day ago and can be marked as Class B test data packets; 40 commodities were created today and can be marked as Class C test data packets. When the test data packet is called, the test data packet with a near validity period, that is, the Class A test data packet, can be provided first.

[0077] In one embodiment of the present disclosure, the step of generating a test data packet may include the following steps:

[0078] The test data is acquired from a data source corresponding to the category of the test data packet and its associated data source to form a test data packet.

[0079] As mentioned above, most test data involve multiple data sources. For example, a test case for a certain order transaction requires multiple categories of test data, such as merchant test data, product test data, store test data, user test data, and marketing test data; a test case for a certain commodity management requires not only commodity test data, but also store test data, merchant test data, and other categories of test data. Therefore, in this embodiment, when generating the test data packet, it is first necessary to determine the data source and other related data sources required for the test data packet, and then obtain the corresponding test data from the data source and the related data source to form a test data packet. For example, for the test case of commodity management, the data source it needs to use is the commodity application, but only commodity test data can be obtained from the commodity application, and store test data and merchant test data cannot be provided. Therefore, it is necessary to obtain store test data from other related data sources, such as from the store application, and from the merchant application. The obtained commodity test data, store test data, and merchant test data are combined into a test data packet that can complete the commodity management test.

[0080] The quantity and composition of the data in the test data packet may be determined according to the needs of the actual application, as long as the test data packet can meet the needs of the corresponding test case.

[0081] In one embodiment of the present disclosure, the step of acquiring test data from a data source corresponding to the category of the test data packet and its associated data source to form a test data packet may include the following steps:

[0082] Acquire identification information of the test data, and form a test data packet with the identification information of the test data;

[0083] Storing the data ontology of the test data in a data ontology database;

[0084] A corresponding relationship between the data ontology of the test data and the identification information is established.

[0085] In order to save data storage space and reduce data transmission costs, in this embodiment, only the identification information of the test data is used to form a test data packet, that is, after obtaining the test data from the corresponding data source, the identification information of the test data is extracted, and the identification information of the test data is used to form a test data packet, and the corresponding data body of the test data is not put into the test data packet, but is stored in a pre-set data body database, and then the corresponding relationship between the data body of the test data and the identification information is established, so that the use needs of the test case for the test data can be met, and when the data body is needed later, the data body corresponding to the identification information of the test data can be obtained from the data body database according to the identification information of the test data. Among them, the identification information of the test data can be, for example, the ID of the test data, etc.

[0086] In one embodiment of the present disclosure, the method may further include the following steps:

[0087] In response to detecting that the number of remaining test data packets in the test data storage component is lower than a first preset number threshold, a test data packet corresponding to the test data storage component category is generated and filled into the test data storage component so that the number of remaining test data packets in the test data storage component is higher than the first preset number threshold.

[0088] In order to further ensure the stability and timeliness of the test data packet provision, in this embodiment, when it is found that the remaining test data packets in the test data storage component are insufficient, the filling of the test data packets should be triggered in time. Specifically, if it is detected that the number of remaining test data packets in the test data storage component is lower than the first preset number threshold, the generation and filling of the test data packet is triggered, that is, the test data packet corresponding to the test data storage component category is generated and filled into the test data storage component until the number of remaining test data packets in the test data storage component is higher than the first preset number threshold. In order to save the detection cost, the detection of the number of remaining test data packets in the test data storage component can also be performed regularly, for example, the detection of the number of remaining test data packets in the test data storage component can be performed in units of minutes or hours. In this way, the decoupling of data production and consumption can be achieved, avoiding the situation where the data consumption rate is higher than the data production rate, or the insufficient available test data is caused by the failure of data production, thereby blocking the data testing process, ensuring the adequacy of the test data, so that the test data can be used at any time.

[0089] Among them, the first preset quantity threshold can be determined according to the needs of actual application. It should be particularly noted that the first preset quantity threshold corresponding to each type of test data packet or each type of test data storage component is very important. If it is set too low, the data in the test data storage component will be insufficient. If it is set too high, the test data packet will be backlogged. Over time, the test data packet will eventually "rot", that is, the data will expire. Taking electronic product data as an example, each electronic product data will set a saleable time, which is generally a few weeks or a month. If the electronic product data is always in a backlog state, it will cause the products related to the electronic product data to pass the saleable time but still not be cleaned up, and the product data in the test data storage component will be unusable before it is used. In addition, due to the different characteristics of stores, merchants, marketing activities, user's coupon assets and other data, the effective time length is different, and the usage of each type of data is different. In the end, each type of product data will be required to have a different first preset quantity threshold. For example, if a certain type of test data packet is used more frequently and the number used every day is large, the first preset quantity threshold needs to be set higher. On the contrary, the first preset quantity threshold needs to be set lower. In summary, it can be seen that the first preset quantity threshold corresponding to each type of test data packet or each type of test data storage component must be consistent with the characteristics of the test data packet and must be adjusted in a timely manner according to different commodity data.

[0090] In one embodiment of the present disclosure, the corresponding first preset quantity threshold value can be dynamically adjusted based on the analysis of the historical test data packet usage. Specifically, the usage of a certain type of test data packet with the same or corresponding time characteristics in a preset historical time period can be estimated to obtain the usage of this type of test data packet at the current time, and the first preset quantity threshold value can be determined based on the usage of this type of test data packet at the current time. For example, the usage of each type of test data packet can be recorded on a weekly basis for the past month, and then the data can be classified in a year-on-year manner (compared with the data usage on Mondays, compared with the data usage on Tuesdays, etc.), and then a linear regression algorithm is used to predict the data usage of the day based on the data usage obtained from the above statistics. Taking commodity data as an example, the first preset quantity threshold value of the day (Wednesday) is the value estimated after linear regression based on the commodity data usage on each Wednesday in the past month.

[0091] The detection of the number of remaining test data packets in the test data storage component may be performed at a certain time frequency, for example, the number of remaining test data packets in the test data storage component may be detected every several minutes or hours.

[0092] In addition, considering that the frequency of use of different types of test data packets may be the same or different, therefore, in one embodiment of the present disclosure, the first preset quantity thresholds corresponding to different types of test data storage components may be the same or different, and the detection frequencies of the number of remaining test data packets may be the same or different. That is, the first preset quantity thresholds corresponding to different types of test data storage components and the detection frequencies of the number of remaining test data packets may be set according to the needs of actual applications.

[0093] In one embodiment of the present disclosure, the method may further include the following steps:

[0094] The target test data packet is marked as used, and the used test data packet is migrated to a used test data packet storage component.

[0095] In order to avoid the mutual interference caused by a certain test data packet being used by multiple parties and the possibility of data being polluted, as well as the situation that too many used test data packets are stored and affect the data computing performance and computing functions, in this embodiment, it is also necessary to clean up the used test data packets in a timely manner. Specifically: if it is detected that a certain test data packet is called, the test data packet is marked as used, and the used test data packet is migrated to a pre-set used test data packet storage component, that is, the test data packet is migrated to the used test data packet storage component as soon as it is called, thereby avoiding the data expiration corruption and data pollution caused by the use of pre-prepared static test data in the prior art. The used test data packet storage component can be shared by multiple categories of test data storage components, or it can be used exclusively by the test data storage components of the corresponding category. At this time, the category of the used test data packet storage component is consistent with the category of the test data storage component corresponding to the used test data packet.

[0096] As mentioned above, when a test data packet is stored in a test data packet storage component of a corresponding category, the validity period information of the test data packet is also stored. Therefore, in one embodiment of the present disclosure, in order to ensure the validity of the test data packet, a validity period periodic check may be performed on the test data packet in the test data packet storage component. If the check finds that the expiration time of the test data packet is greater than the current time and the time difference between the expiration time of the test data packet and the current time is less than a preset duration, the test data packet is marked as about to expire, and the test data packet that is about to expire is migrated to the used test data packet storage component. For example, a test data packet validity period check may be performed at 23:00 every day, and the test data packets that will expire within 24 hours found in the check are migrated to the used test data packet storage component, thereby further ensuring the validity of the test data packets in the test data packet storage component.

[0097] In one embodiment of the present disclosure, the method may further include the following steps:

[0098] In response to detecting that a data clearing condition is met, used test data packets in the used test data packet storage component are cleared.

[0099] In order to save data storage space, improve the utilization rate of data storage space, and avoid the situation of abnormal system performance and function caused by excessive data accumulation, for example, if a merchant continuously creates stores, but does not clean up the invalid store information in time, there will be thousands of stores under the name of the merchant after a period of time, which will inevitably affect the system performance. In this embodiment, it is also necessary to clean up the used test data packet storage component in time. Specifically, if it is detected that the data cleaning condition is met, the used test data packets in the used test data packet storage component are cleaned, that is, the polling cleaning of the used test data packet storage component is realized. Among them, the data cleaning condition can be: the number of used test data packets in the used test data packet storage component exceeds the second preset number threshold, or reaches the preset data cleaning time, wherein the second preset number threshold can be determined according to the needs of the actual application. That is to say, when it is found that the number of used test data packets stored in a used test data packet storage component is sufficient, the used test data packet storage component can be cleaned up; the used test data packet storage component can also be cleaned up according to a preset time frequency, wherein the cleaning time frequency of the used test data packet storage component is related to the use or update frequency of the used test data packets, for example, the cleaning time frequency of the used test data packet storage component storing frequently used or updated used test data packets can be set to a higher frequency, such as commodity data can be cleaned up once a day, and the cleaning time frequency of the used test data packet storage component storing seldom used or updated used test data packets can be set to a lower frequency, such as merchant data can be cleaned up once a month.

[0100] In one embodiment of the present disclosure, the method may further include the following steps:

[0101] The data ontology database is cleaned up with data ontologies corresponding to identification information in the cleaned used test data packets.

[0102] In order to further improve the utilization rate of data storage space, in this embodiment, when cleaning up the used test data packets in the used test data packet storage component, it is also necessary to synchronously clean up the data body stored in the data body database and corresponding to the identification information in the cleaned used test data packet.

[0103] Figure 2 A flow chart showing a data processing method according to an embodiment of the present disclosure is shown as follows: Figure 2As shown, taking the single payment scenario as an example, after the test data provider receives the test data request, it is determined that the target test data packets involved in the test data request include: merchant test data packets, commodity test data packets, marketing test data packets, store test data packets and user test data packets, as well as the number of test data packets in each category. Then, the test data provider can respectively retrieve the corresponding number of merchant test data packets, commodity test data packets, marketing test data packets, store test data packets and user test data packets from the merchant test data packet storage component, commodity test data packet storage component, marketing test data packet storage component, store test data packet storage component and user test data packet storage component and send them to the test data requester. Initially, the merchant test data packet storage component, commodity test data packet storage component, marketing test data packet storage component, store test data packet storage component and user test data packet storage component all store test data packets exceeding the first preset number threshold, as shown by the solid circles in the figure. During the test process, the number of remaining test data packets in the merchant test data packet storage component, the commodity test data packet storage component, the marketing test data packet storage component, the store test data packet storage component and the user test data packet storage component is regularly checked. If it is found that the number of remaining test data packets in a certain test data packet storage component is lower than the first preset number threshold, as shown by the long dashed line in the figure, test data is obtained from the data source corresponding to the test data packet storage component and its associated data source to form a test data packet and fill it into the test data packet storage component. The figure only takes the merchant test data packet storage component as an example. During the test process, each test data packet in each test data packet storage component is marked as used after being called, and is migrated to the used test data packet storage component of the corresponding category, wherein the used test data packets stored in the used test data packet storage component will be cleaned up regularly, and the figure also only takes the merchant test data packet storage component as an example.

[0104] The following are embodiments of the apparatus of the present disclosure, which can be used to execute embodiments of the method of the present disclosure.

[0105] Figure 3 FIG. 1 is a block diagram showing a data processing device according to an embodiment of the present disclosure. The device can be implemented as part or all of an electronic device through software, hardware, or a combination of both. Figure 3 As shown, the data processing device includes:

[0106] The receiving module 301 is configured to receive a test data request;

[0107] A determination module 302 is configured to determine a target test data packet category according to the test data request;

[0108] The sending module 303 is configured to take out the target test data packet from the test data packet storage component corresponding to the target test data packet category and send it to the test data requester, wherein the test data packet storage component stores one or more test data packets, and the test data packet storage component has a category corresponding to the stored test data packet category.

[0109] As mentioned above, the preparation of test data is an important step in software testing. There is an important design principle for test cases: there should be no dependencies between test cases, and the execution results of a test case should not be affected by the execution results of other test cases. Based on this principle, all the test data required for a test case should be prepared by itself, but since the preparation of test data mostly involves multiple data sources, and the availability of data sources cannot be guaranteed, this greatly reduces the efficiency of data execution, especially for software systems with many types of test data and strong correlations between data. If pre-prepared static test data is used, there will be problems of data expiration and corruption and data pollution caused by multiple use. Therefore, there is an urgent need for a solution that can not only improve the execution efficiency of data, but also effectively avoid data unavailability due to data expiration and corruption and data pollution.

[0110] In view of the above defects, in this embodiment, a data processing device is proposed, which pre-sets a plurality of different types of test data packet storage components to store different types of test data packets, so as to achieve the decoupling of data production and consumption, and manage the validity of the test data packets in the test data packet storage components. This technical solution can achieve the stable acquisition and long-term validity of test data packets, as well as the effective generation and management of test data, which not only greatly improves the execution efficiency of data, but also avoids the problem of data unavailability due to data expiration, corruption and data pollution.

[0111] In one embodiment of the present disclosure, the data processing device may be implemented as a test data provider such as a computer, a computing device, an electronic device, a server, etc. that processes the test data.

[0112] In one embodiment of the present disclosure, the test data request refers to a request for obtaining test data, which is sent by the test data requester to the test data provider and carries the specific demand for obtaining test data, wherein the test data request may carry information such as the category and quantity of the target test data packet to be obtained, and the category and quantity of the target test data packet to be obtained by the test data requester may be determined based on the test data request. The test data request may be obtained through a unified data interface provided to the outside, that is, the test data requester may send the test data request through the unified data interface provided to the outside by the test data provider.

[0113] In one embodiment of the present disclosure, the test data packet storage component refers to a component for storing one or more test data packets obtained through production, wherein the test data packet storage component has a category corresponding to the category of the stored test data packets. For example, in e-commerce scenarios such as order payment, merchant data, product data, marketing data, store data, user data, etc. may be used. Then, the categories of test data packets that may be used in e-commerce scenarios may include: merchants, products, marketing, stores, users, etc., and test data packets of a certain category may be stored in a test data packet storage component of the corresponding category. That is to say, the category of the test data packet storage component is consistent with the category of the test data packet, that is, the category of the test data packet storage component may include: merchants, products, marketing, stores, users, etc., and each category of the test data packet storage component only stores test data packets of the corresponding category, which will improve the targeted acquisition of test data packets and thereby increase the speed of acquiring test data packets.

[0114] As can be seen from the above, the categories of available test data packets can be one or more, the categories of test data packet storage components used to store the test data packets can be one or more, and for a certain test data request, the categories of the target test data packets carried by the request to be obtained can also be one or more.

[0115] In the above embodiment, after receiving the test data request sent by the test data requester, the category of the target test data packet that the test data requester wants to obtain is determined based on the test data request, and then the target test data packets are taken out from the test data packet storage components of the corresponding categories and sent to the test data requester.

[0116] In one embodiment of the present disclosure, the device may further include:

[0117] The generating module is configured to generate a test data packet and store the test data packet into a test data packet storage component corresponding to the category of the test data packet.

[0118] In order to enable the test data requester to request to obtain the test data packet when he wants, to ensure the stability of the provision of the test data packet, and to improve the targeted acquisition of the test data packet, thereby improving the speed of obtaining the test data packet, in this embodiment, it is necessary to generate a certain number of test data packets of each category in advance, and store these test data packets in a test data packet storage component corresponding to the category of the test data packet, in preparation for subsequent test data requesters to request to obtain, that is, a test data packet storage component only stores one category of test data packets, such as a product test data packet storage component only stores product test data packets, and a store test data packet storage component only stores store test data packets.

[0119] Furthermore, considering that the test data packet has a validity period, the validity periods of different test data packets may be different. Therefore, when the test data packet is stored in the test data packet storage component of the corresponding category, the validity period information of the test data packet can be stored at the same time. In addition, in order to ensure that each test data packet can be used within its validity period, when the test data packet is called, the test data packet with a near validity period can be provided first. For example, assuming that the minimum validity period of a certain type of test data packet is 3 days, there are 120 test data packets stored in the corresponding test data packet storage component, of which 40 test data packets were created 2 days ago and are about to expire. They can be marked as Class A test data packets; 40 commodities were created 1 day ago and can be marked as Class B test data packets; 40 commodities were created today and can be marked as Class C test data packets. When the test data packet is called, the test data packet with a near validity period, that is, the Class A test data packet, can be provided first.

[0120] In one embodiment of the present disclosure, the generating module may be configured as follows:

[0121] The test data is acquired from a data source corresponding to the category of the test data packet and its associated data source to form a test data packet.

[0122] As mentioned above, most test data involve multiple data sources. For example, a test case for a certain order transaction requires multiple categories of test data, such as merchant test data, product test data, store test data, user test data, and marketing test data; a test case for a certain commodity management requires not only commodity test data, but also store test data, merchant test data, and other categories of test data. Therefore, in this embodiment, when generating the test data packet, it is first necessary to determine the data source and other related data sources required for the test data packet, and then obtain the corresponding test data from the data source and the related data source to form a test data packet. For example, for the test case of commodity management, the data source it needs to use is the commodity application, but only commodity test data can be obtained from the commodity application, and store test data and merchant test data cannot be provided. Therefore, it is necessary to obtain store test data from other related data sources, such as from the store application, and from the merchant application. The obtained commodity test data, store test data, and merchant test data are combined into a test data packet that can complete the commodity management test.

[0123] The quantity and composition of the data in the test data packet may be determined according to the needs of the actual application, as long as the test data packet can meet the needs of the corresponding test case.

[0124] In one embodiment of the present disclosure, the part of obtaining the test data from the data source corresponding to the category of the test data packet and its associated data source to form the test data packet may be configured as follows:

[0125] Acquire identification information of the test data, and form a test data packet with the identification information of the test data;

[0126] Storing the data ontology of the test data in a data ontology database;

[0127] A corresponding relationship between the data ontology of the test data and the identification information is established.

[0128] In order to save data storage space and reduce data transmission costs, in this embodiment, only the identification information of the test data is used to form a test data packet, that is, after obtaining the test data from the corresponding data source, the identification information of the test data is extracted, and the identification information of the test data is used to form a test data packet, and the corresponding data body of the test data is not put into the test data packet, but is stored in a pre-set data body database, and then the corresponding relationship between the data body of the test data and the identification information is established, so that the use needs of the test case for the test data can be met, and when the data body is needed later, the data body corresponding to the identification information of the test data can be obtained from the data body database according to the identification information of the test data. Among them, the identification information of the test data can be, for example, the ID of the test data, etc.

[0129] In one embodiment of the present disclosure, the device may further include:

[0130] A filling module is configured to generate test data packets corresponding to the category of the test data storage component in response to detecting that the number of remaining test data packets in the test data storage component is lower than a first preset number threshold, and fill the test data packets into the test data storage component so that the number of remaining test data packets in the test data storage component is higher than the first preset number threshold.

[0131] In order to further ensure the stability and timeliness of the test data packet provision, in this embodiment, when it is found that the remaining test data packets in the test data storage component are insufficient, the filling of the test data packets should be triggered in time. Specifically, if it is detected that the number of remaining test data packets in the test data storage component is lower than the first preset number threshold, the generation and filling of the test data packet is triggered, that is, the test data packet corresponding to the test data storage component category is generated and filled into the test data storage component until the number of remaining test data packets in the test data storage component is higher than the first preset number threshold. In order to save the detection cost, the detection of the number of remaining test data packets in the test data storage component can also be performed regularly, for example, the detection of the number of remaining test data packets in the test data storage component can be performed in units of minutes or hours. In this way, the decoupling of data production and consumption can be achieved, avoiding the situation where the data consumption rate is higher than the data production rate, or the insufficient available test data is caused by the failure of data production, thereby blocking the data testing process, ensuring the adequacy of the test data, so that the test data can be used at any time.

[0132] Among them, the first preset quantity threshold can be determined according to the needs of actual application. It should be particularly noted that the first preset quantity threshold corresponding to each type of test data packet or each type of test data storage component is very important. If it is set too low, the data in the test data storage component will be insufficient. If it is set too high, the test data packet will be backlogged. Over time, the test data packet will eventually "rot", that is, the data will expire. Taking electronic product data as an example, each electronic product data will set a saleable time, which is generally a few weeks or a month. If the electronic product data is always in a backlog state, it will cause the products related to the electronic product data to pass the saleable time but still not be cleaned up, and the product data in the test data storage component will be unusable before it is used. In addition, due to the different characteristics of stores, merchants, marketing activities, user's coupon assets and other data, the effective time length is different, and the usage of each type of data is different. In the end, each type of product data will be required to have a different first preset quantity threshold. For example, if a certain type of test data packet is used more frequently and the number used every day is large, the first preset quantity threshold needs to be set higher. On the contrary, the first preset quantity threshold needs to be set lower. In summary, it can be seen that the first preset quantity threshold corresponding to each type of test data packet or each type of test data storage component must be consistent with the characteristics of the test data packet and must be adjusted in a timely manner according to different commodity data.

[0133] In one embodiment of the present disclosure, the corresponding first preset quantity threshold value can be dynamically adjusted based on the analysis of the historical test data packet usage. Specifically, the usage of a certain type of test data packet with the same or corresponding time characteristics in a preset historical time period can be estimated to obtain the usage of this type of test data packet at the current time, and the first preset quantity threshold value can be determined based on the usage of this type of test data packet at the current time. For example, the usage of each type of test data packet can be recorded on a weekly basis for the past month, and then the data can be classified in a year-on-year manner (compared with the data usage on Mondays, compared with the data usage on Tuesdays, etc.), and then a linear regression algorithm is used to predict the data usage of the day based on the data usage obtained from the above statistics. Taking commodity data as an example, the first preset quantity threshold value of the day (Wednesday) is the value estimated after linear regression based on the commodity data usage on each Wednesday in the past month.

[0134] The detection of the number of remaining test data packets in the test data storage component may be performed at a certain time frequency, for example, the number of remaining test data packets in the test data storage component may be detected every several minutes or hours.

[0135] In addition, considering that the frequency of use of different types of test data packets may be the same or different, therefore, in one embodiment of the present disclosure, the first preset quantity thresholds corresponding to different types of test data storage components may be the same or different, and the detection frequencies of the number of remaining test data packets may be the same or different. That is, the first preset quantity thresholds corresponding to different types of test data storage components and the detection frequencies of the number of remaining test data packets may be set according to the needs of actual applications.

[0136] In one embodiment of the present disclosure, the device may further include:

[0137] The migration module is configured to mark the target test data packet as used and migrate the used test data packet to the used test data packet storage component.

[0138] In order to avoid the mutual interference caused by a certain test data packet being used by multiple parties and the possibility of data being polluted, as well as the situation that too many used test data packets are stored and affect the data computing performance and computing functions, in this embodiment, it is also necessary to clean up the used test data packets in a timely manner. Specifically: if it is detected that a certain test data packet is called, the test data packet is marked as used, and the used test data packet is migrated to a pre-set used test data packet storage component, that is, the test data packet is migrated to the used test data packet storage component as soon as it is called, thereby avoiding the data expiration corruption and data pollution caused by the use of pre-prepared static test data in the prior art. The used test data packet storage component can be shared by multiple categories of test data storage components, or it can be used exclusively by the test data storage components of the corresponding category. At this time, the category of the used test data packet storage component is consistent with the category of the test data storage component corresponding to the used test data packet.

[0139] As mentioned above, when a test data packet is stored in a test data packet storage component of a corresponding category, the validity period information of the test data packet is also stored. Therefore, in one embodiment of the present disclosure, in order to ensure the validity of the test data packet, a validity period periodic check may be performed on the test data packet in the test data packet storage component. If the check finds that the expiration time of the test data packet is greater than the current time and the time difference between the expiration time of the test data packet and the current time is less than a preset duration, the test data packet is marked as about to expire, and the test data packet that is about to expire is migrated to the used test data packet storage component. For example, a test data packet validity period check may be performed at 23:00 every day, and the test data packets that will expire within 24 hours found in the check are migrated to the used test data packet storage component, thereby further ensuring the validity of the test data packets in the test data packet storage component.

[0140] In one embodiment of the present disclosure, the device may further include:

[0141] The cleaning module is configured to clean up the used test data packets in the used test data packet storage component in response to detecting that a data cleaning condition is met.

[0142] In order to save data storage space, improve the utilization rate of data storage space, and avoid the situation of abnormal system performance and function caused by excessive data accumulation, for example, if a merchant continuously creates stores, but does not clean up the invalid store information in time, there will be thousands of stores under the name of the merchant after a period of time, which will inevitably affect the system performance. In this embodiment, it is also necessary to clean up the used test data packet storage component in time. Specifically, if it is detected that the data cleaning condition is met, the used test data packets in the used test data packet storage component are cleaned, that is, the polling cleaning of the used test data packet storage component is realized. Among them, the data cleaning condition can be: the number of used test data packets in the used test data packet storage component exceeds the second preset number threshold, or reaches the preset data cleaning time, wherein the second preset number threshold can be determined according to the needs of the actual application. That is to say, when it is found that the number of used test data packets stored in a used test data packet storage component is sufficient, the used test data packet storage component can be cleaned up; the used test data packet storage component can also be cleaned up according to a preset time frequency, wherein the cleaning time frequency of the used test data packet storage component is related to the use or update frequency of the used test data packets, for example, the cleaning time frequency of the used test data packet storage component storing frequently used or updated used test data packets can be set to a higher frequency, such as commodity data can be cleaned up once a day, and the cleaning time frequency of the used test data packet storage component storing seldom used or updated used test data packets can be set to a lower frequency, such as merchant data can be cleaned up once a month.

[0143] In one embodiment of the present disclosure, the cleaning module may also be configured as follows:

[0144] The data ontology database is cleaned up with data ontologies corresponding to identification information in the cleaned used test data packets.

[0145] In order to further improve the utilization rate of data storage space, in this embodiment, when cleaning up the used test data packets in the used test data packet storage component, it is also necessary to synchronously clean up the data body stored in the data body database and corresponding to the identification information in the cleaned used test data packet.

[0146] The present disclosure also discloses an electronic device, Figure 4 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown. Figure 4 As shown, the electronic device 400 includes a memory 401 and a processor 402; wherein,

[0147] The memory 401 is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor 402 to implement the above method steps.

[0148] Figure 5 It is a structural diagram of a computer system suitable for implementing a data processing method according to an embodiment of the present disclosure.

[0149] like Figure 5 As shown, the computer system 500 includes a processing unit 501, which can perform various processes in the above-mentioned embodiments according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the system 500 are also stored. The processing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0150] The following components are connected to the I / O interface 505: an input part 506 including a keyboard, a mouse, etc.; an output part 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage part 508 including a hard disk, etc.; and a communication part 509 including a network interface card such as a LAN card, a modem, etc. The communication part 509 performs communication processing via a network such as the Internet. The drive 510 is also connected to the I / O interface 505 as needed. Removable media 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., are installed on the drive 510 as needed, so that the computer program read therefrom is installed into the storage part 508 as needed. Among them, the processing unit 501 can be implemented as a processing unit such as a CPU, a GPU, a TPU, an FPGA, an NPU, etc.

[0151] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the road map or block diagram can represent a part of a module, program segment or code, and the part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a different order from the order marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0152] The units or modules involved in the embodiments described in the present disclosure may be implemented by software or hardware. The units or modules described may also be set in a processor, and the names of these units or modules do not constitute limitations on the units or modules themselves in some cases.

[0153] As another aspect, the present disclosure further provides a computer-readable storage medium, which may be a computer-readable storage medium included in the device described in the above embodiment; or a computer-readable storage medium that exists independently and is not assembled into the device. The computer-readable storage medium stores one or more programs, and the programs are used by one or more processors to execute the method described in the present disclosure.

[0154] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present disclosure is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other.

Claims

1. A data processing method, comprising: Acquire identification information of test data, and form a test data packet with the identification information of the test data; Storing the data ontology of the test data in a data ontology database; Establishing a corresponding relationship between the data body of the test data and the identification information; Receive a test data request; the test data request carries the type and quantity of target test data packets to be acquired; Determine a target test data packet category according to the test data request; Taking out the target test data packet from a test data packet storage component corresponding to the target test data packet category and sending it to the test data requester, wherein the test data packet storage component stores one or more test data packets, and the test data packet storage component has a category corresponding to the category of the stored test data packets; In response to detecting that the number of remaining test data packets in the test data packet storage component is lower than a first preset number threshold, generating a test data packet corresponding to the category of the test data packet storage component and filling the test data packet into the test data packet storage component so that the number of remaining test data packets in the test data packet storage component is higher than the first preset number threshold; If it is determined that the target test data packet is called, the target test data packet is marked as used, and the used test data packet is migrated to the used test data packet storage component; or, if it is determined that the expiration time of the target test data packet is greater than the current time, and the time difference between the target test data packet and the current time is less than a preset duration, the target test data packet is marked as about to expire, and the about to expire test data packet is migrated to the used test data packet storage component.

2. The method according to claim 1, further comprising: A test data packet is generated, and the test data packet is stored in a test data packet storage component corresponding to the category of the test data packet.

3. The method according to claim 2, wherein generating a test data packet comprises: The test data is acquired from a data source corresponding to the category of the test data packet and its associated data source to form a test data packet.

4. The method according to claim 1, further comprising: In response to detecting that a data clearing condition is met, used test data packets in the used test data packet storage component are cleared.

5. The method according to claim 4, further comprising: The data ontology database is cleaned up with data ontologies corresponding to identification information in the cleaned used test data packets.

6. A data processing device, comprising: A receiving module is configured to obtain identification information of test data and compose a test data packet with the identification information of the test data; Storing the data ontology of the test data in a data ontology database; Establishing a corresponding relationship between the data body of the test data and the identification information; receiving a test data request; the test data request carries the category and / or quantity of target test data packets to be acquired; A determination module, configured to determine a target test data packet category according to the test data request; a sending module, configured to take out the target test data packet from a test data packet storage component corresponding to the target test data packet category and send it to a test data requester, wherein the test data packet storage component stores one or more test data packets, and the test data packet storage component has a category corresponding to the stored test data packet category; a filling module configured to, in response to detecting that the number of remaining test data packets in the test data packet storage component is lower than a first preset number threshold, generate a test data packet corresponding to the category of the test data packet storage component and fill the test data packet into the test data packet storage component so that the number of remaining test data packets in the test data packet storage component is higher than the first preset number threshold; A migration module, if it is determined that the target test data packet is called, marks the target test data packet as used, and migrates the used test data packet to the used test data packet storage component; or, if it is determined that the expiration time of the target test data packet is greater than the current time, and the time difference between the target test data packet and the current time is less than a preset duration, marks the target test data packet as about to expire, and migrates the about to expire test data packet to the used test data packet storage component.

7. An electronic device comprising a memory and at least one processor; wherein: The memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the at least one processor to implement the method according to any one of claims 1 to 5.

Images