A big data testing method, device and equipment and storage medium

Abstract

The big data test method, device and equipment and storage medium provided by the application can be applied to the technical field of big data, can construct a data table interrelation chain based on a data flow link, can convert and merge the conversion rules between data tables based on table interconversion rules, form a link processing rule set of the data table interrelation chain, encapsulate the conversion rules between data tables into job units based on data processing logic, and sequentially concatenate the job units having data transmission relationship to form a job flow. The data flow processing result of a job area in the job flow is verified based on the link processing rule set, wherein the job area at least includes two adjacent job units. Finally, a corresponding storage space is constructed for each job unit, and the monitoring data of each job unit is stored into the corresponding storage space. The big data test method realizes monitoring of each process of data flow processing, automatically tracks and automatically verifies data, and improves test efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a big data testing method, device and equipment and storage medium. BACKGROUND

[0002] Modern society has become a highly developed information circulation society, and big data application has penetrated into the development of various industries. Every time we surf the Internet, every time we search the network, and every electronic payment behavior is closely related to big data. Big data testing refers to testing a large amount of data sets within a certain time range by professional testing tools and technologies. Compared with function testing, big data testing is more about verifying data processing capability. In the existing big data testing process, if you want to know which component and which layer a certain data file is in, you need to consult the test personnel of each component in turn, which is time-consuming and inefficient. SUMMARY

[0003] In order to solve the problem of long time consumption and low efficiency in the prior art, the present application provides a big data testing method, device, equipment and storage medium, which has the characteristics of lower time consumption and higher testing efficiency.

[0004] According to the big data testing method provided by the embodiment of the present application, the method comprises the following steps:

[0005] Based on the data flow link, a data table relationship chain is constructed, and based on the table conversion rule, the conversion rule between the data tables is converted and merged to form a link processing rule set of the data table relationship chain;

[0006] Based on the data processing logic, the conversion rule between the data tables is encapsulated into a job unit, and the job units with data transmission relationship are sequentially connected to form a job flow;

[0007] Based on the link processing rule set, the data flow processing result of the job area in the job flow is verified, and the job area at least includes two adjacent job units.

[0008] For each job unit, a corresponding storage space is constructed, and the monitoring data of each job unit is stored in the corresponding storage space.

[0009] Further, the data table relationship chain is constructed based on the data flow link, which comprises:

[0010] Obtain the data processing component selected by the user;

[0011] Connect each data processing component based on the connection link in the link library to constitute the data flow link;

[0012] Correlate data tables of each data processing component based on blood analysis, and form a relationship chain between the data tables.

[0013] Further, the conversion rules between the data tables are converted and merged based on the inter-table conversion rules to form a link processing rule set of the relationship chain between the data tables, including:

[0014] Determine the inter-table conversion rules based on business requirements;

[0015] Combine each of the inter-table conversion rules to constitute the link processing rule set, and the link processing rule set includes sequentially increasing processing rule links, wherein the first processing rule link only includes conversion rules between the first two data tables, and the last processing rule link includes conversion rules between all adjacent data tables.

[0016] Further, the conversion rules between the data tables are encapsulated into job units based on data processing logic, and job units with data transmission relationship are sequentially concatenated to form a job flow, including:

[0017] The conversion rules between the data tables are encapsulated using the development logic of the component, the job scheduling and triggering mechanism to generate the job unit;

[0018] Based on the data transmission between the job units, each of the job units is sequentially concatenated to form the job flow.

[0019] Further, the data flow processing results of the job area in the job flow are verified based on the link processing rule set, including:

[0020] Obtain the target link processing rule in the link processing rule set that matches the job unit in the job area;

[0021] Verify the target result data obtained after processing the target link processing rule based on a test tool.

[0022] Further, the monitoring data at least includes job status data and test status data, and the method further includes:

[0023] The generation time of the test status data and the job status data are fed back to a display terminal.

[0024] Further, the big data testing method further includes:

[0025] Set the data type and field type in the data table based on a set rule.

[0026] According to the big data testing device provided in the specific embodiment of the application,

[0027] a data flow access module, configured to construct a data table interrelation chain based on a data flow link, and to convert and merge conversion rules between data tables based on table interconversion rules, to form a link processing rule set of the data table interrelation chain;

[0028] a job flow construction module, configured to encapsulate the conversion rules between the data tables into job units based on data processing logic, and to sequentially concatenate job units having a data transmission relationship to form a job flow;

[0029] a data verification module, configured to verify data flow processing results of a job region in the job flow based on the link processing rule set, the job region including at least two adjacent job units; and

[0030] a data storage module, configured to construct a corresponding storage space for each job unit, and to store monitoring data of each job unit into the corresponding storage space.

[0031] According to the device provided in the embodiment of the present application, the memory and the processor are included.

[0032] The memory is configured to store a program.

[0033] The processor is configured to execute the program, to realize each step of the big data testing method.

[0034] According to the storage medium provided in the embodiment of the present application, the computer program is stored on the storage medium, and when the computer program is executed by the processor, each step of the big data testing method is realized.

[0035] The big data testing method provided in the present application can construct a data table interrelation chain based on a data flow link, and convert and merge conversion rules between data tables based on table interconversion rules, to form a link processing rule set of the data table interrelation chain. Then the conversion rules between the data tables are encapsulated into job units based on data processing logic, and job units having a data transmission relationship are sequentially concatenated to form a job flow. The data flow processing results of a job region in the job flow are verified based on the link processing rule set, wherein the job region includes at least two adjacent job units. Finally, a corresponding storage space is constructed for each job unit, and monitoring data of each job unit is stored into the corresponding storage space. The big data testing method realizes monitoring of each process of data flow processing, can automatically track and automatically verify the correctness of data, reduces communication cost, and improves testing efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only constitute a part of the embodiments of the present application, and for those skilled in the art, other drawings can also be obtained based on the provided drawings without creative labor.

[0037] Figure 1 is a flow chart of a big data testing method provided according to an exemplary embodiment;

[0038] Figure 2 is a flow chart of construction of a data table interrelation chain provided according to an exemplary embodiment;

[0039] Figure 3 is a flow chart of construction of a link processing rule set provided according to an exemplary embodiment;

[0040] Figure 4 is a flow chart of construction of a job flow provided according to an exemplary embodiment;

[0041] Figure 5 is a flow chart of data verification provided according to an exemplary embodiment;

[0042] Figure 6 is a structural diagram of a big data testing device provided according to an exemplary embodiment;

[0043] Figure 7 is a structural diagram of an apparatus provided according to an exemplary embodiment. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only constitute a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0045] Referring to Figure 1 The embodiments of the present application provide a big data testing method, which can include the following steps:

[0046] 101, constructing a data table interrelation chain based on a data flow link, and converting and merging the conversion rules between data tables based on table interconversion rules to form a link processing rule set of the data table interrelation chain.

[0047] Firstly, data stream access is performed, a data stream link is initialized through big data component access and metadata construction, a data table relationship chain is formed, and the change range of all tables involved is determined. Meanwhile, table conversion rules can be determined according to business requirements, and rule conversion and merging are performed to form a link processing rule set of the data table relationship chain, and to lay the foundation for local link rule testing.

[0048] 102. Encapsulating the conversion rules between data tables into job units based on data processing logic, and sequentially concatenating the job units that exist in data transmission relationship to form a job flow.

[0049] The development logic of each data table is encapsulated to generate a job unit, and the data transmission between the job units is concatenated to form a job flow. For example, the development flow between P8 and P8 components can be linked to ensure that the multi-node test can be directly initiated.

[0050] 103. Verifying the data stream processing result of the job area in the job flow based on the link processing rule set, the job area at least including two adjacent job units.

[0051] The local link rule is used as the test basis to schedule data stream jobs to generate the target result data obtained by development. The local link rule is compared by using automatic test tools such as field conversion comparison, type conversion comparison, and numerical value conversion comparison. Whether the result data obtained by the development process meets the link rule of the business requirement is tested to test the correctness of the rule conversion.

[0052] 104. Constructing a corresponding storage space for each job unit, and storing the monitoring data of each job unit into the corresponding storage space.

[0053] By establishing a test node monitoring mechanism, the test job state and test data state can be monitored, and temporary storage space is provided for each stage type data, which is fed back to the front-end display. For example, the generation time of the test data and whether the test task is successful can be displayed, so as to track the data flow in real time.

[0054] The big data test method can automatically track the data stream and automatically verify the correctness of the data, reducing the communication cost and improving the test efficiency.

[0055] As a feasible implementation manner of the above embodiment, referring to FIG. 1, the data stream link is used to construct the data table relationship chain, which can include the following steps: Figure 2

[0056] 201. Obtain the data processing components selected by the user.

[0057] 202. Connect the data processing components based on the connection links in the link library to constitute a data stream link.​

[0058] 203, based on the blood analysis of each data processing component data table is associated, forming the relationship between data table chain.

[0059] Specifically, first, from the big data component library to obtain the user selected components, such as A, B, C, D, E, F six components, and then from the link library to obtain the link between the six components, such as the connection order is B, C, E, F, D, A. Then based on the meta database for data table connection is also B, C, E, F, D, A, wherein the data table includes field type, data type and other data. Through the blood analysis of the correlation between the link between the tables is associated and stored in the blood table library, facilitating subsequent analysis and query. At the same time, according to the business requirements to determine the table conversion rule and rule conversion merge, forming a head and tail conversion rule link, which is for the subsequent local link rule test to lay the foundation. Blood table library corresponding to each component table tailor made specific type of data, and support field level statistics, in the establishment of the correlation between multiple tables. Among them, the data type can include bus type, ORC type, relational data and other data types. Field type can include regular, enumeration, primary and foreign key association, anti case data, etc.

[0060] Referring to Figure 3 As shown, based on the conversion rule between the data table conversion rule is converted and merged, forming a link processing rule set of data table relationship chain, which can include the following steps:

[0061] 301, based on the business requirements to determine the conversion rule between the tables.

[0062] 302, each table conversion rule is combined to constitute a link processing rule set, the link processing rule set includes a number of incremental processing rule link, wherein the first processing rule link only includes the conversion rule between the first two data tables, and the last processing rule link includes the conversion rule between all adjacent data tables.

[0063] Specifically, the conversion rule between the tables can be determined according to the business requirements and the rule conversion merge, forming a head and tail conversion rule link, for example, the conversion rule 1 between table B and table C, the conversion rule 2 between table C and table E, the conversion rule 3 between table E and table F, the conversion rule 4 between table F and table D, and the conversion rule 5 between table F and table A. Then the link processing rule set can include: 1->2, 1->2->3, 1->2->3->4, etc.

[0064] Referring to Figure 4 As shown, based on the data processing logic, the conversion rule between the data tables is encapsulated into a job unit, and the job units with data transmission relationship are sequentially concatenated to form a job flow, which can include the following steps:

[0065] 401, encapsulate the conversion rules between data tables using the development logic, job scheduling and triggering mechanism of the components to generate a job unit.

[0066] 402, sequentially concatenate each job unit based on the data transmission between the job units to form a job flow.

[0067] Specifically, the development logic of each component is encapsulated to generate a job, supporting trigger, manual trigger and time trigger, and the data transmission between jobs is concatenated to form a job flow.

[0068] Referring to Figure 5 The data flow processing result of the job area in the job flow based on the link processing rule set can include the following steps:

[0069] 501, obtain the target link processing rule in the link processing rule set that matches the job unit in the job area.

[0070] 502, verify the target result data obtained after processing the target link processing rule based on the test tool.

[0071] The scheduling data flow job is generated by the development of the target result data by the local link rule, and the local link rule comparison is performed by the field conversion comparison, type conversion comparison and numerical conversion comparison automatic test tool. Test whether the result data obtained in the development process meets the link rule of the business requirement, so as to test the correctness of the rule conversion.

[0072] The monitoring data at least includes: job state data and test state data, the generation time of the test state data and the job state data can be fed back to the display terminal, so as to track the data flow in real time.

[0073] And the data type and field type in the data table can be set based on the set rule. For example, when supporting the generation of new and old line format text data, the test data support is provided for the bus type product as the bus type data. When supporting the generation of ORC format data, the modified partition can be directly used for large data type components. Regular field support generates test data according to regular expression to meet the generation requirements of specific data format and related combination type. The generated data value of the enumeration type is randomly selected from the predetermined enumeration value.

[0074] Based on the same design idea, referring to Figure 6 The embodiment of the application also provides a big data test device, which can realize the steps of the above-mentioned big data test method when running, and the device can include:

[0075] The data stream access module 601 is configured to construct a data table interrelation chain based on a data stream link, and convert and merge the conversion rules between data tables based on table interconversion rules to form a link processing rule set of the data table interrelation chain.

[0076] The job flow construction module 602 is configured to encapsulate the conversion rules between data tables into job units based on data processing logic, and sequentially concatenate the job units having data transmission relations to form a job flow.

[0077] The data verification module 603 is configured to verify the data stream processing result of a job region in the job flow based on the link processing rule set, the job region at least including two adjacent job units.

[0078] The data storage module 604 is configured to construct a corresponding storage space for each job unit, and store the monitoring data of each job unit into the corresponding storage space.

[0079] The big data testing device has the same beneficial effects as the big data testing method described above, and the specific implementation manner can refer to the embodiments of the big data testing method provided by the above-described embodiments, which will not be described herein again.

[0080] Referring to Figure 7 The embodiments of the present application also provide a device, which can include a memory 701 and a processor 702.

[0081] The memory 701 is configured to store a program.

[0082] The processor 702 is configured to execute the program to implement each step of the big data testing method described in the above embodiments.

[0083] The embodiments of the present application also provide a storage medium having a computer program stored thereon, which is executed by a processor to implement each step of the big data testing method described in the above embodiments.

[0084] It should be noted that the big data testing method, device, equipment and storage medium provided by the present application can be used in the fields of artificial intelligence, block chain, distribution, cloud computing, big data, Internet of Things, mobile Internet, network security, chips, virtual reality, augmented reality, holographic technology, quantum computing, quantum communication, quantum measurement, digital twin or finance. The above are only examples and do not limit the application field of the big data testing method, device, equipment and storage medium provided by the present application.

[0085] For each method embodiment described above, for the sake of simple description, it is expressed as a combination of a series of actions, but those skilled in the art should know that the present application is not limited by the order of the described actions, because according to the present application, certain steps can be performed in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily necessary for the present application.

[0086] It should be noted that each embodiment in the specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between each embodiment can be referred to each other. For device embodiments, since they are basically similar to method embodiments, they are described more simply, and the relevant parts are referred to the part of the method embodiment.

[0087] The steps in the method of each embodiment of the present application can be adjusted in order, combined and reduced according to actual needs. The technical features recorded in each embodiment can be replaced or combined.

[0088] The modules and sub-modules in the device and terminal of each embodiment of the present application can be combined, divided and reduced according to actual needs.

[0089] In several embodiments provided by the present application, it should be understood that the disclosed terminal, device and method can be implemented by other ways. For example, the terminal embodiments described above are only schematic, for example, the division of modules or sub-modules is only a logical function division, and actual implementation can have another division manner, for example, a plurality of sub-modules or modules can be combined or integrated into another module, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be indirect coupling or communication connection between some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0090] The modules or sub-modules described as separate components can or can not be physically separated, and the components of the modules or sub-modules can or can not be physical modules or sub-modules, that is, they can be located in one place, or can be distributed to multiple network modules or sub-modules. According to actual needs, some or all of the modules or sub-modules can be selected to achieve the purpose of the embodiment.

[0091] In addition, each functional module or sub-module in various embodiments of the present application can be integrated in one processing module, or each module or sub-module can exist physically alone, or two or more modules or sub-modules can be integrated in one module. The integrated module or sub-module can be realized in the form of hardware or in the form of a software functional module or sub-module.

[0092] Those skilled in the art will further appreciate that the functions or steps of the examples described herein can be implemented using electronic hardware, computer software, or any combination thereof. To clearly illustrate this interchangeability of hardware and software, various examples have been described herein in terms of their functional or logical implementation. Whether such functions are implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Skilled artisans can implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0093] The steps of a method or algorithm described in connection with the examples disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0094] Finally, it should be noted that the terms "first", "second", and the like, herein do not denote any order, quantity, combination, or arrangement, but are used to distinguish one element from another. Also, the terms "include", "comprise", or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements does not include only those elements recited, but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without more limitations, an element defined by the phrase "comprising a" does not exclude the existence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0095] The foregoing description of the disclosed embodiments enables a person skilled in the art to make or use the application. Modifications of these embodiments will occur to persons of skill in the art, and that the appended claims are intended to cover all such modifications that do not depart from the true spirit and scope of the application. Therefore, the application is not limited to the embodiments shown but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A big data testing method, characterized in that, include: A relationship chain between data tables is constructed based on the data flow link, and the transformation rules between data tables are transformed and merged based on the transformation rules between tables to form a set of link processing rules for the relationship chain between data tables; Building relationship chains between data tables based on data flow links includes: Obtain the data processing component selected by the user; The data processing components are connected based on the connection links in the link library to form the data flow link; Based on lineage analysis, the data tables of each of the data processing components are associated to form a relationship chain between the data tables; Based on data processing logic, the conversion rules between the data tables are encapsulated into job units, and job units with data transfer relationships are sequentially connected to form a job flow; The data stream processing results of the job area in the job flow are verified based on the set of link processing rules, wherein the job area includes at least two adjacent job units; A corresponding storage space is constructed for each of the job units, and the monitoring data of each job unit is stored in the corresponding storage space.

2. The method according to claim 1, characterized in that, The transformation and merging of inter-table transformation rules based on inter-table transformation rules to form a set of link processing rules for the inter-table relationship chain includes: The table conversion rules are determined based on business requirements; The table conversion rules are combined to form the link processing rule set. The link processing rule set includes processing rule links with an increasing number of links. The first processing rule link only includes conversion rules between the first two data tables, and the last processing rule link includes conversion rules between all adjacent data tables.

3. The method according to claim 1, characterized in that, The process of encapsulating the conversion rules between the data tables into job units based on data processing logic, and sequentially connecting job units with data transfer relationships to form a job flow, includes: The conversion rules between the data tables are encapsulated using the component's development logic, job scheduling, and triggering mechanism to generate the job unit; The work units are sequentially connected based on the data transfer between them to form the work flow.

4. The method according to claim 1, characterized in that, The verification of the data stream processing results of the job area in the job flow based on the link processing rule set includes: Obtain the target link processing rule from the link processing rule set that matches the work unit within the work area; The target result data obtained after processing the target link processing rules using testing tools is verified.

5. The method according to claim 1, characterized in that, The monitoring data includes at least: job status data and test status data, and the method further includes: The generation time of the test status data and the operation status data are fed back to the display terminal.

6. The method according to claim 1, characterized in that, Also includes: The data types and field types in the data table are set based on the established rules.

7. A big data testing device, characterized in that, include: The data flow access module is used to construct a relationship chain between data tables based on the data flow link, and to convert and merge the conversion rules between data tables based on the conversion rules between the tables, forming a set of link processing rules for the relationship chain between the data tables; Building relationship chains between data tables based on data flow links includes: Obtain the data processing component selected by the user; The data processing components are connected based on the connection links in the link library to form the data flow link; Based on lineage analysis, the data tables of each of the data processing components are associated to form a relationship chain between the data tables; The workflow construction module is used to encapsulate the conversion rules between the data tables into job units based on data processing logic, and to connect the job units with data transfer relationships in sequence to form a workflow. A data verification module is used to verify the data stream processing results of a work area in the job flow based on the link processing rule set, wherein the work area includes at least two adjacent work units; and The data storage module is used to construct a corresponding storage space for each of the work units and to store the monitoring data of each work unit into the corresponding storage space.

8. An electronic device, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement the various steps of the big data testing method as described in any one of claims 1 to 6.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the various steps of the big data testing method as described in any one of claims 1 to 6.

Images