Data extraction method, system, processor and electronic device

Abstract

The application discloses a data extraction method, system, processor and electronic equipment. Relates to the field of cloud computing, and the method comprises the steps of determining a test requirement index, and constructing a data extraction table containing a plurality of field types on a first data source according to the test requirement index; extracting data on the first data source through the data extraction table to obtain a plurality of first extraction data; judging whether the field types of the plurality of first extraction data contain all the field types of the data extraction table; in the case where the field types of the plurality of first extraction data do not contain all the field types of the data extraction table, extracting data on a second data source through the data extraction table to obtain a plurality of second extraction data; and determining the plurality of first extraction data and the plurality of second extraction data as test data meeting the test requirement index. Through the application, the problem of low efficiency in extracting test data caused by manual preparation of data when extracting required test data from a plurality of data sources in the related art is solved.

Description

Technical Field

[0001] This application relates to the field of cloud computing, and more specifically, to a data extraction method, system, processor, and electronic device. Background Technology

[0002] As big data platforms connect to increasingly diverse data sources, different data sources support varying data encoding formats, field types, special characters, and other characteristics. Testers need to have a thorough understanding of the characteristics of each data source to prepare test data that meets their requirements. Therefore, test data preparation places certain demands on the technical skills of testers.

[0003] In related technologies, when preparing test data, testers need to manually extract data from different data sources and then use a data derivation tool to import the extracted data into the system being tested. Different data sources use different data derivation tools, and many factors such as the encoding format of the data derivation tool and the script encoding format can lead to inaccuracies in the test data during the derivation process, making the operation very cumbersome and inconvenient. In addition, in the process of batch processing of big data, there are certain requirements for the degree of data dispersion and the data volume. Manually generating data is difficult to control the data dispersion. Therefore, when using a data derivation tool, the data is directly derived from the production environment corresponding to the data source. The derivation process takes a long time and also affects the testers' control over the test progress.

[0004] There is currently no effective solution to the problem that manual data preparation is required when extracting test data from multiple data sources in related technologies, resulting in low efficiency in extracting test data. Summary of the Invention

[0005] The main objective of this application is to provide a data extraction method, system, processor, and electronic device to solve the problem in related technologies where manual data preparation is required when extracting test data from multiple data sources, resulting in low efficiency in extracting test data.

[0006] To achieve the above objectives, according to one aspect of this application, a data extraction method is provided. The method includes: determining test requirement indicators; constructing a data extraction table containing multiple field types on a first data source based on the test requirement indicators, wherein the first data source is the data source corresponding to the data table type in the test requirement indicators; extracting data conforming to the test requirement indicators from the first data source using the data extraction table to obtain multiple first extracted data; determining whether the field types of the multiple first extracted data already include all the field types of the data extraction table; if the field types of the multiple first extracted data do not include all the field types of the data extraction table, extracting data conforming to the test requirement indicators from a second data source other than the first data source using the data extraction table to obtain multiple second extracted data; and determining the multiple first extracted data and the multiple second extracted data as test data that meets the test requirement indicators.

[0007] Optionally, constructing a data extraction table containing multiple field types on the first data source based on test requirement metrics includes: obtaining the data table structure definition method of the first data source; determining the name of each field type among the multiple field types, and determining the special character metrics and Chinese character metrics among the test requirement metrics; and generating a data extraction table containing multiple field types, the name of each field type, special character metrics, and Chinese character metrics on the first data source according to the data table structure definition method.

[0008] Optionally, extracting data that meets the test requirement indicators from the first data source using a data extraction table to obtain multiple first extracted data includes: determining at least one field type of the data that meets the test requirement indicators in the first data source; adding the data value of each field type in the first data source to the data extraction table to obtain the first extracted data; and extracting data that meets the test requirement indicators from a second data source other than the first data source using a data extraction table to obtain multiple second extracted data includes: determining other field types of the data that meets the test requirement indicators in the second data source besides the field types in the first data source; adding the data value of each other field type in the second data source to the data extraction table to obtain the second extracted data.

[0009] Optionally, before determining multiple first extracted data and multiple second extracted data as test data that meet the test requirement indicators, the method further includes: calculating the dispersion of the extracted data in the data extraction table every first preset period, wherein the extracted data includes first extracted data and second extracted data; determining whether the dispersion of the extracted data is greater than or equal to a dispersion threshold; and stopping the extraction of first extracted data or second extracted data if the dispersion of the extracted data is greater than or equal to the dispersion threshold.

[0010] Optionally, after determining whether the dispersion of the extracted data is greater than or equal to the dispersion threshold, the method further includes: if the dispersion of the extracted data is less than the dispersion threshold, calculating the amount of extracted data in the data extraction table every second preset period; determining whether the amount of extracted data is greater than or equal to the data amount threshold; and stopping the extraction of the first or second extracted data if the amount of extracted data is greater than or equal to the data amount threshold.

[0011] Optionally, after extracting data that meets the test requirements metrics from a second data source other than the first data source through a data extraction table to obtain multiple second extracted data, the method further includes: mapping the multiple second extracted data from the second data source to the first data source to obtain a first data source containing the second extracted data.

[0012] Optionally, after determining multiple first extracted data and multiple second extracted data as test data that meet the test requirement indicators, the method further includes: constructing a test data report table based on the test data; and identifying the data volume, dispersion, and position of special characters of the test data in the test data report table.

[0013] To achieve the above objectives, according to another aspect of this application, a data extraction system is provided. The system includes: a test requirement indicator input unit, comprising an information input module and an information parsing module, wherein the information input module is used to collect descriptive information of test data input by testers, and the information parsing module is used to parse the descriptive information to obtain test requirement indicators; and a data extraction unit, wherein the data extraction unit is used to construct a data extraction table containing multiple field types based on the test requirement indicators, and to extract first extracted data from a first data source or second extracted data from a second data source based on the data extraction table.

[0014] Optionally, the system further includes: a data loading unit, which is used to determine whether to stop extracting the first or second extracted data based on the dispersion and data volume of the extracted data; and a visualization unit, which is used to construct a test data report table based on the test data, wherein the test data report table identifies the data volume, dispersion, and the position of special characters of the test data.

[0015] To achieve the above objectives, according to another aspect of this application, a data extraction apparatus is provided. The apparatus includes: a first determining unit, configured to determine test requirement indicators and construct a data extraction table containing multiple field types on a first data source based on the test requirement indicators, wherein the first data source is the data source corresponding to the data table type in the test requirement indicators; a first extraction unit, configured to extract data conforming to the test requirement indicators from the first data source using the data extraction table, obtaining multiple first extracted data; a judging unit, configured to judge whether the field types of the multiple first extracted data already include all the field types of the data extraction table; a second extraction unit, configured to extract data conforming to the test requirement indicators from a second data source other than the first data source using the data extraction table when the field types of the multiple first extracted data do not include all the field types of the data extraction table, obtaining multiple second extracted data; and a second determining unit, configured to determine the multiple first extracted data and the multiple second extracted data as test data that meets the test requirement indicators.

[0016] This application employs the following steps: determining test requirement indicators, and constructing a data extraction table containing multiple field types on a first data source, where the first data source corresponds to the data table type in the test requirement indicators; extracting data that meets the test requirement indicators from the first data source using the data extraction table, resulting in multiple first extracted data; determining whether the field types of the multiple first extracted data include all the field types of the data extraction table; if the field types of the multiple first extracted data do not include all the field types of the data extraction table, extracting data that meets the test requirement indicators from a second data source other than the first data source, resulting in multiple second extracted data; and determining the multiple first extracted data and multiple second extracted data as test data that meets the test requirement indicators. This solves the problem in related technologies where manual data preparation is required when extracting required test data from multiple data sources, leading to low efficiency in test data extraction. By constructing a data extraction table based on the test requirement indicators and automatically extracting data from multiple data sources using the data extraction table, the efficiency of test data extraction is improved. Attached Figure Description

[0017] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0018] Figure 1 This is a flowchart of a data extraction method provided according to an embodiment of this application;

[0019] Figure 2 This is a flowchart of an optional data extraction method provided according to an embodiment of this application;

[0020] Figure 3 This is a schematic diagram of the structure of the data extraction system provided according to the embodiments of this application;

[0021] Figure 4 This is a schematic diagram of a data extraction device provided according to an embodiment of this application;

[0022] Figure 5 This is a schematic diagram of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0023] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0024] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0027] The present invention will now be described in conjunction with preferred implementation steps. Figure 1 This is a flowchart of the data extraction method provided according to the embodiments of this application, such as... Figure 1 As shown, the method includes the following steps:

[0028] Step S101: Determine the test requirement indicators and construct a data extraction table containing multiple field types on the first data source based on the test requirement indicators. The first data source is the data source corresponding to the data table type in the test requirement indicators.

[0029] Specifically, test requirement metrics can be the characteristic information of the data that testers need to extract. The primary data source can be the data source corresponding to the data table type indicated in the test requirement metrics. For example, if this test requires creating a data table through a HIVE data source, then the HIVE data source is the primary data source. Multiple field types include all field types in the data source, such as text, int, tinyint, datetime, vachar, and char.

[0030] It's important to note that before extracting data from the data source, a connection needs to be established. Testers configure the corresponding data source connection information according to the test requirements metrics. For example, the connection information for an MPP data source is: jdbc:postgresql: / / {ipxxx}:25308 / {dba_name}{username}{paasword}, ​​and the connection information for an Oracle data source is: jdbc:oracle:thin:@{ipxxx}:1521 / {dba_name}{username}{paasword}. Communication connections with each data source are established through this connection information. Test requirements metrics are obtained by testers describing the necessary data characteristics and inputting them according to template prompts. Table 1 shows the template for prompting testers to input data characteristic information. After collecting the data characteristic information using the above template, the collected data characteristic information is categorized through keyword parsing to obtain the test requirements metrics.

[0031] Table 1

[0032]

[0033] Step S102: Extract data that meets the test requirements indicators from the first data source using the data extraction table to obtain multiple first extracted data.

[0034] Specifically, the first data source contains various types of data. Data that meets the test requirements metrics is filtered out using a data extraction table and added to the data extraction table to obtain the first extracted data. For example, if the first data source contains data with storage formats such as Orc, RcFile, and AVRO, and the data storage format in the test requirements metrics is Orc, then only the multiple first extracted data sets with the Orc data storage format will be extracted from the first data source using the data table.

[0035] Step S103: Determine whether the field types of the multiple first extracted data already include all the field types of the data extraction table.

[0036] Specifically, to ensure sufficient test types in the test data, data of all field types is extracted from the data extraction table. However, the field types in the data from the first data source may not cover all field types. Therefore, it is necessary to determine whether the first data source can provide data of all field types that meet the test requirements. If the field types in the data from the first data source cannot cover all field types, data of the remaining field types is extracted from the second data source.

[0037] Step S104: If the field types of multiple first extracted data do not include all the field types of the data extraction table, data that meets the test requirements indicators is extracted from a second data source other than the first data source through the data extraction table, resulting in multiple second extracted data.

[0038] For example, the field types in the first extracted data include text, int, and tinyint. However, data with field types such as datetime, vachar, and char is missing. Therefore, data with field types such as datetime, vachar, and char is extracted from the second data source through a data extraction table; this is the second extracted data.

[0039] Step S105: Determine multiple first extracted data and multiple second extracted data as test data that meet the test requirement indicators.

[0040] Specifically, after the first and second extracted data are extracted through the data extraction table, the first and second extracted data are determined as the test data that meet the test requirements indicators required for this test.

[0041] The data extraction method provided in this application determines test requirement indicators and constructs a data extraction table containing multiple field types on a first data source based on these indicators. The first data source corresponds to the data table type specified in the test requirement indicators. Data conforming to the test requirement indicators is extracted from the first data source using the data extraction table, resulting in multiple first extracted data sets. It is then determined whether the field types of the multiple first extracted data sets include all field types of the data extraction table. If the field types of the multiple first extracted data sets do not include all field types of the data extraction table, data conforming to the test requirement indicators is extracted from a second data source other than the first data source, resulting in multiple second extracted data sets. These multiple first and second extracted data sets are then identified as test data that satisfies the test requirement indicators. This method solves the problem in related technologies where manual data preparation is required when extracting test data from multiple data sources, leading to low efficiency in test data extraction. By constructing a data extraction table based on the test requirement indicators and automatically extracting data from multiple data sources using the data extraction table, the efficiency of test data extraction is improved.

[0042] Before extracting data, a data extraction table is constructed. Optionally, in the data extraction method provided in this application embodiment, constructing a data extraction table containing multiple field types on the first data source according to the test requirement indicators includes: obtaining the data table structure definition method of the first data source; determining the name of each field type among the multiple field types, and determining the special character indicators and Chinese character indicators among the test requirement indicators; generating a data extraction table containing multiple field types, the name of each field type, special character indicators, and Chinese character indicators on the first data source according to the data table structure definition method.

[0043] Specifically, different data sources have different table structure definitions. After determining the first data source based on the data table type in the test requirements metrics, the data table structure definition in the first data source is obtained. Special character and Chinese character metrics indicate whether special characters and Chinese characters need to be introduced in this test. If so, these are added to the extracted data in the data extraction table. By traversing the system table of statistical field types in the first data source, the multiple field types and names of each field type are determined. All existing field types and their corresponding fields in the first data source are extracted, with the specific field names and types based on the first field extracted during the traversal. Then, based on the data table structure definition of the first data source, a data extraction table containing all field types and their corresponding field names, special character metrics, and Chinese character metrics is constructed. The first or second extraction data is then extracted by constructing the data extraction table.

[0044] After constructing the data extraction table, first extracted data is extracted from the first data source or second extracted data is extracted from the second data source. Optionally, in the data extraction method provided in this application embodiment, extracting data that meets the test requirement indicators from the first data source through the data extraction table to obtain multiple first extracted data includes: determining at least one field type of the data that meets the test requirement indicators in the first data source; adding the data value of each field type in the first data source to the data extraction table to obtain the first extracted data; extracting data that meets the test requirement indicators from a second data source other than the first data source through the data extraction table to obtain multiple second extracted data includes: determining other field types of the data that meets the test requirement indicators in the second data source besides the field types in the first data source; adding the data value of each other field type in the second data source to the data extraction table to obtain the second extracted data.

[0045] Specifically, based on the different field types in the data extraction table, the corresponding field values ​​(i.e., data values, including null and non-null values) of the existing field types in the first data source are extracted and populated into the data extraction table. It is then determined whether the data extraction table has a sufficient number of field values ​​for each field type. If not, field values ​​for field types not included in the first data source need to be extracted from the second data source and populated into the data extraction table. Test data is obtained by extracting data from either the first or second data source using the data extraction table.

[0046] In addition to requirements on the type of data, the test requirements also include an indicator for the dispersion of the data obtained from the data extraction table. Optionally, in the data extraction method provided in this application embodiment, before determining multiple first extracted data and multiple second extracted data as test data that meet the test requirements, the method further includes: calculating the dispersion of the extracted data in the data extraction table every first preset period, wherein the extracted data includes first extracted data and second extracted data; determining whether the dispersion of the extracted data is greater than or equal to a dispersion threshold; and stopping the extraction of the first extracted data or the second extracted data if the dispersion of the extracted data is greater than or equal to the dispersion threshold.

[0047] Specifically, the first preset period can be 1 second, and the dispersion can be a quantitative value representing the difference in data. For example, if a set of data contains 10 values, and 3 of these 10 values ​​are duplicates, then the dispersion of this set of data is 3 divided by 10, or 30%. To ensure that the test data can achieve better testing results, it is necessary to reduce the duplicate values ​​in the extracted data. Therefore, a dispersion threshold is set. For example, if the dispersion threshold is set to 10%, then the dispersion of each field type in the data extraction table needs to be calculated every 1 second to determine whether the dispersion of the data of that field type is greater than or equal to the dispersion threshold. If it is, the extraction of data of that field type needs to be stopped immediately. By periodically checking whether the dispersion of the extracted data is greater than or equal to the dispersion threshold, the diversity of the test data is ensured.

[0048] When extracting data, the test requirements indicators also set indicators for the amount of data. Optionally, in the data extraction method provided in this application embodiment, after determining whether the dispersion of the extracted data is greater than or equal to the dispersion threshold, the method further includes: if the dispersion of the extracted data is less than the dispersion threshold, calculating the amount of extracted data in the data extraction table every second preset period; determining whether the amount of extracted data is greater than or equal to the data amount threshold; and stopping the extraction of the first extracted data or the second extracted data if the amount of extracted data is greater than or equal to the data amount threshold.

[0049] Specifically, the data volume can be the storage volume of the extracted data in the data extraction table, and the data volume threshold can be 10MB, meaning that data extraction will stop when the total extracted data volume reaches 10MB. The second preset period can be set to 1 millisecond, counting the data volume in the data extraction table every 1 millisecond. When the extracted data volume is greater than or equal to the data volume threshold, data extraction from the data extraction table will stop. By setting the data volume threshold, a large amount of redundant data not required for testing is avoided.

[0050] The data extracted from the second data source also needs to be mapped to the first data source. Optionally, in the data extraction method provided in this application embodiment, after extracting data that meets the test requirement indicators through the data extraction table from the second data source other than the first data source to obtain multiple second extracted data, the method further includes: mapping the multiple second extracted data from the second data source to the first data source to obtain a first data source containing the second extracted data.

[0051] Specifically, since the test data needs to be tested from the same data source, it is necessary to ensure that all extracted data are in the same data source. After extracting the second data from the second data source, the second data is mapped from the second data source to the first data source for storage.

[0052] After the data extraction is completed, a test data report table is generated based on the data extraction table. Optionally, in the data extraction method provided in this application embodiment, after determining multiple first extracted data and multiple second extracted data as test data that meet the test requirement indicators, the method further includes: constructing a test data report table based on the test data; and identifying the data volume, dispersion, and position of special characters of the test data in the test data report table.

[0053] Specifically, after the data extraction table extracts the required test data, a test data report table is generated based on the data extraction table to facilitate the use of testers. The test data report table marks the data volume, data dispersion, distribution key, position of special characters, and other points that testers need to pay attention to.

[0054] To achieve the above objectives, according to another aspect of this application, an alternative data extraction method is provided. Figure 2 This is a flowchart of an optional data extraction method provided according to an embodiment of this application. For example... Figure 2 As shown, the method includes:

[0055] Step S201: Testers configure the data source.

[0056] Specifically, testers connect to each data source based on the data source connection information. For example, based on the connection information of data source DB1, they connect to the first data source; based on the connection information of data source DB2, they connect to the second data source; and based on the connection information of data source DB3, they connect to the third data source.

[0057] Step S202: The tester inputs data feature information.

[0058] Specifically, testers describe the required data characteristics and input them according to the template prompts.

[0059] Step S203: Generate a standard basic table structure with all field types.

[0060] Specifically, the data extraction module generates a standard basic table structure with all fields based on the configured data source connection information.

[0061] Step S204: Extract the data that meets the requirements from the data source into the standard base table.

[0062] Specifically, based on the connection information of data source DB1, the system connects to the corresponding first data source and extracts data of all field types and values ​​from the first data source into a standard base table. If sufficient data meeting the requirements cannot be extracted from the first data source, data is first extracted from other data sources DB2, DB3, etc., to supplement the data, and then this portion of data is mapped back to data source DB1.

[0063] Step S205: Selectively introduce special characters and Chinese characters based on data characteristic information.

[0064] Specifically, it is determined whether special characters or Chinese characters need to be introduced into the standard base table based on data characteristic information.

[0065] Step S206: Load data into the standard base bar to ensure that the data volume and data dispersion of the final generated test table meet the requirements.

[0066] Specifically, after extracting data into a standard base table, it is periodically determined whether the data in the base table meets the requirements for data volume and data dispersion.

[0067] Step S207: Generate a visual test data report table.

[0068] Specifically, a test data report table is generated, which identifies the data volume, data dispersion, distribution key, position of special characters, and other aspects that testers need to pay attention to.

[0069] The optional data extraction method provided in this application's embodiments automates the entire test data preparation process. Testers only need to input data characteristic information to obtain a test table that meets the testing requirements from a specified data source. The method allows for more precise and convenient customization of the test table's data volume and dispersion, making it easier for testers to prepare test tables suitable for different testing scenarios. It can automatically generate all field types supported on the data source cluster and their corresponding full field values, resulting in richer test data. Furthermore, it allows for customization of whether to introduce special characters, making it more user-friendly for test scenarios requiring abnormal data simulation and ensuring more thorough test preparation for testers.

[0070] To achieve the above objectives, according to another aspect of this application, a data extraction system is provided. Figure 3 This is a schematic diagram of the data extraction system provided according to an embodiment of this application. For example... Figure 3 As shown, the system includes:

[0071] The data feature collection unit 301 includes an information input module and an information parsing module. The information input module is used to collect descriptive information of the test data input by the testers, and the information parsing module is used to parse the descriptive information to obtain test requirement indicators.

[0072] Specifically, the data feature collection module 301 provides a data feature template including data source type, data source connection information, data volume, data dispersion, whether special characters are introduced, whether Chinese characters are included, data encoding format, table type, and distribution key. Testers describe the test data features based on the template. The data feature collection module 301 parses the description information into information that can be recognized by other modules and then transmits it accordingly. Testers only need to describe the data requirements and do not need to understand the relevant characteristics of the data source, which reduces the technical threshold requirements for testers.

[0073] The data extraction unit 302 is used to construct a data extraction table containing multiple field types based on the test requirement indicators, and extract first extracted data from a first data source or extract second extracted data from a second data source based on the data extraction table.

[0074] Specifically, the data extraction unit 302 includes a data source configuration module, which configures a first data source and a second data source, and connects to the first data source based on first connection information and to the second data source based on second connection information; a metadata extraction module, which determines the field types, field names, and data storage formats contained in the first data source; a data extraction table generation module, which generates a data extraction table based on the field types, field names, and data storage formats; a data loading module, which extracts first extracted data from the first data source or extracts second extracted data from the second data source; a data determination module, which determines whether the field types of the first extracted data already include all the field types of the data extraction table; and a data mapping module, which maps the second extracted data from the second data source to the first data source.

[0075] Optionally, in the data extraction system provided in this application embodiment, the system further includes: a data loading unit 303, which is used to determine whether to stop extracting the first extracted data or the second extracted data based on the dispersion and data volume of the extracted data; and a visualization unit 304, which is used to construct a test data report table based on the test data, wherein the test data report table identifies the data volume, dispersion and position of special characters of the test data.

[0076] Specifically, the data loading unit 303 includes a data auto-increment module, which is used to add first extracted data or second extracted data to the data extraction table, and determine whether to stop adding first extracted data or second extracted data based on the statistical information of the data dynamic statistics module and the data dispersion dynamic statistics module; the data dispersion dynamic statistics module is used to calculate the dispersion of the extracted data in the data extraction table every first preset period; and the data dynamic statistics module is used to calculate the amount of data extracted in the data extraction table every second preset period.

[0077] The data extraction system provided in this application fully automates the test data preparation process. Testers only need to input data feature information to obtain a test table that meets the testing requirements from a specified data source. It allows for more precise and convenient customization of the data volume and dispersion of the test table, making it easier for testers to prepare test tables suitable for different testing scenarios. It can automatically generate all field types supported on the data source cluster and their corresponding full field values, resulting in richer test data. Furthermore, it allows for customization of whether to introduce special characters, making it more user-friendly for test scenarios requiring abnormal data simulation and ensuring more thorough test preparation for testers.

[0078] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0079] This application also provides a data extraction device. It should be noted that the data extraction device of this application can be used to execute the data extraction method provided in this application. The data extraction device provided in this application is described below.

[0080] Figure 4 This is a schematic diagram of a data extraction device provided according to an embodiment of this application. For example... Figure 4 As shown, the device includes:

[0081] The first determining unit 10 is used to determine the test requirement indicators and construct a data extraction table containing multiple field types on the first data source based on the test requirement indicators. The first data source is the data source corresponding to the data table type in the test requirement indicators.

[0082] The first extraction unit 20 is used to extract data that meets the test requirements indicators from the first data source through the data extraction table, and obtain multiple first extraction data.

[0083] Judgment unit 30 is used to determine whether the field types of multiple first extracted data have included all the field types of the data extraction table;

[0084] The second extraction unit 40 is used to extract data that meets the test requirements indicators from a second data source other than the first data source when the field types of multiple first extracted data do not include all the field types of the data extraction table, thereby obtaining multiple second extracted data.

[0085] The second determining unit 50 is used to determine multiple first extracted data and multiple second extracted data as test data that meet the test requirement indicators.

[0086] The data extraction device provided in this application embodiment determines test requirement indicators through a first determining unit 10 and constructs a data extraction table containing multiple field types on a first data source based on the test requirement indicators. The first data source is the data source corresponding to the data table type in the test requirement indicators. A first extraction unit 20 extracts data that meets the test requirement indicators from the first data source through the data extraction table, obtaining multiple first extracted data. A judging unit 30 judges whether the field types of the multiple first extracted data include all the field types of the data extraction table. A second extraction unit 40 extracts data that meets the test requirement indicators from a second data source other than the first data source if the field types of the multiple first extracted data do not include all the field types of the data extraction table, obtaining multiple second extracted data. A second determining unit 50 determines the multiple first extracted data and the multiple second extracted data as test data that meets the test requirement indicators. This solves the problem in related technologies where manual data preparation is required when extracting the required test data from multiple data sources, resulting in low efficiency in test data extraction. By constructing a data extraction table based on the test requirement indicators and automatically extracting data from multiple data sources through the data extraction table, the efficiency of test data extraction is improved.

[0087] Optionally, in the data extraction device provided in this application embodiment, the first determining unit 10 includes: an acquisition module, used to acquire the data table structure definition method of the first data source; a first determining module, used to determine the name of each field type among multiple field types, and determine the special character indicators and Chinese character indicators among the test requirement indicators; and a generation module, used to generate a data extraction table containing multiple field types, the name of each field type, special character indicators, and Chinese character indicators in the first data source according to the data table structure definition method.

[0088] Optionally, in the data extraction apparatus provided in this application embodiment, the first extraction unit 20 includes: a second determining module, used to determine at least one field type of data in the first data source that meets the test requirement indicators; a first adding module, used to add the data value of each field type in the first data source to the data extraction table to obtain first extracted data; extracting data that meets the test requirement indicators from a second data source other than the first data source through the data extraction table to obtain multiple second extracted data includes: a third determining module, used to determine other field types of data in the second data source that meet the test requirement indicators besides the field types in the first data source; a second adding module, used to add the data value of each other field type in the second data source to the data extraction table to obtain second extracted data.

[0089] Optionally, in the data extraction device provided in the embodiments of this application, the device further includes: a calculation unit, used to calculate the dispersion of the extracted data in the data extraction table every first preset period, wherein the extracted data includes first extracted data and second extracted data; a dispersion judgment unit, used to judge whether the dispersion of the extracted data is greater than or equal to a dispersion threshold; and a first stop unit, used to stop extracting the first extracted data or the second extracted data when the dispersion of the extracted data is greater than or equal to the dispersion threshold.

[0090] Optionally, in the data extraction device provided in the embodiments of this application, the device further includes: a statistics unit, used to count the amount of data extracted in the data extraction table every second preset period when the dispersion of the extracted data is less than the dispersion threshold; a data amount judgment unit, used to judge whether the amount of data extracted is greater than or equal to the data amount threshold; and a second stop unit, used to stop extracting the first extracted data or the second extracted data when the amount of data extracted is greater than or equal to the data amount threshold.

[0091] Optionally, in the data extraction apparatus provided in the embodiments of this application, the apparatus further includes: a mapping unit, used to map multiple second extracted data from a second data source to a first data source to obtain a first data source containing the second extracted data.

[0092] Optionally, in the data extraction apparatus provided in the embodiments of this application, the apparatus further includes: a construction unit for constructing a test data report table based on the test data; and an identification unit for identifying the data volume, dispersion, and position of special characters of the test data in the test data report table.

[0093] The data extraction device includes a processor and a memory, including the first determining unit 10, the first extraction unit 20, the determining unit 30, the second extraction unit 40, and the second determining unit 50, etc., which are all stored as program units in the memory. The processor executes the program units stored in the memory to achieve the corresponding functions.

[0094] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and the efficiency of test data retrieval can be improved by adjusting kernel parameters.

[0095] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0096] This invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements a data extraction method.

[0097] This invention provides a processor for running a program, wherein the program executes a data extraction method during runtime.

[0098] like Figure 5 As shown, this embodiment of the invention provides an electronic device 501, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs the following steps: a data extraction method. The device in this document can be a server, PC, PAD, mobile phone, etc.

[0099] This application also provides a computer program product that, when executed on a data processing device, is suitable for executing a program with the following initialization method steps: a data extraction method.

[0100] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0101] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0102] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0103] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0104] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0105] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0106] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0107] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0108] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0109] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A data extraction method, characterized in that, include: The test requirement metrics are determined, and a data extraction table containing multiple field types is constructed on a first data source based on the test requirement metrics. The first data source is the data source corresponding to the data table type in the test requirement metrics, and the test requirement metrics are obtained by classifying the collected data feature information. The data table structure definition method of the first data source is obtained. By traversing the system table of statistical field types in the first data source, the multiple field types contained in the first data source and the field name of each field type are determined, and the special character metrics and Chinese character metrics in the test requirement metrics are determined. A data extraction table containing the multiple field types, the field name of each field type, the special character metrics, and the Chinese character metrics is generated on the first data source according to the data table structure definition method. The data extraction table is used to extract data that meets the test requirement indicators from the first data source, resulting in multiple first extracted data. Determine whether the field types of the plurality of first extracted data already include all the field types of the data extraction table; If the field types of the multiple first extracted data do not include all the field types of the data extraction table, data that meets the test requirement indicators is extracted from a second data source other than the first data source through the data extraction table to obtain multiple second extracted data. Every first preset period, the dispersion of the extracted data in the data extraction table is calculated, wherein the extracted data includes the first extracted data and the second extracted data; it is determined whether the dispersion of the extracted data is greater than or equal to a dispersion threshold; if the dispersion of the extracted data is greater than or equal to the dispersion threshold, the extraction of the first extracted data or the second extracted data is stopped. The plurality of first extracted data and the plurality of second extracted data are determined as test data that meet the test requirement indicators.

2. The method according to claim 1, characterized in that, The data that meets the test requirement indicators is extracted from the first data source using the data extraction table, resulting in multiple first extracted data, including: Determine at least one field type in the data from the first data source that meets the test requirement metrics; Add the data value of each field type in the first data source to the data extraction table to obtain the first extracted data; The data extraction table is used to extract data that meets the test requirement indicators from a second data source other than the first data source, resulting in multiple second extracted data, including: Determine the field types of the data in the second data source that meet the test requirement indicators, other than the field types in the first data source; The data values ​​of each of the other field types in the second data source are added to the data extraction table to obtain the second extracted data.

3. The method according to claim 1, characterized in that, After determining whether the dispersion of the extracted data is greater than or equal to a dispersion threshold, the method further includes: If the dispersion of the extracted data is less than the dispersion threshold, the amount of extracted data in the data extraction table is counted every second preset period. Determine whether the amount of extracted data is greater than or equal to a data amount threshold; If the amount of extracted data is greater than or equal to the data amount threshold, stop extracting the first extracted data or the second extracted data.

4. The method according to claim 1, characterized in that, After extracting data that meets the test requirement indicators from a second data source other than the first data source using the data extraction table, and obtaining multiple second extracted data, the method further includes: The plurality of second extracted data are mapped from the second data source to the first data source to obtain a first data source containing the second extracted data.

5. The method according to claim 1, characterized in that, After determining the plurality of first extracted data and the plurality of second extracted data as test data that meet the test requirement indicators, the method further includes: Construct a test data report table based on the test data; The test data report table identifies the amount of data, the degree of dispersion, and the location of special characters in the test data.

6. A data extraction system, characterized in that, include: The test requirement indicator input unit includes an information input module and an information parsing module. The information input module is used to collect descriptive information of the test data input by the testers, and the information parsing module is used to parse the descriptive information to obtain the test requirement indicators. A data extraction unit is configured to construct a data extraction table containing multiple field types based on the test requirement indicators, and extract first extracted data from a first data source or second extracted data from a second data source based on the data extraction table; the data extraction unit includes a metadata extraction module, which is configured to determine the field types, field names and data storage formats contained in the first data source; A data extraction table generation module, which is used to generate a data extraction table based on field type, field name and data storage format; A data loading module, which is used to extract first extracted data from a first data source or extract second extracted data from a second data source; The data determination module is used to determine whether the field types of the first extracted data already include all the field types of the data extraction table; wherein, the data table structure definition method of the first data source is obtained; by traversing the system table of statistical field types of the first data source, the multiple field types contained in the first data source and the field name of each field type are determined, and the special character indicator and the Chinese character indicator in the test requirement indicators are determined; according to the data table structure definition method, a data extraction table containing the multiple field types, the field name of each field type, the special character indicator and the Chinese character indicator is generated in the first data source; A data loading unit is used to determine whether to stop extracting the first extracted data or the second extracted data based on the dispersion and amount of data of the extracted data.

7. The system according to claim 6, characterized in that, The system also includes: A visualization unit is used to construct a test data report table based on the test data, wherein the test data report table identifies the data volume, dispersion, and position of special characters of the test data.

8. A processor, characterized in that, The processor is used to run a program, wherein the program executes the data extraction method according to any one of claims 1 to 5.

9. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the data extraction method according to any one of claims 1 to 5.

Images