A big data testing platform and data processing method
By utilizing the data preparation and automated testing modules of the big data testing platform, the problems of high testing costs and low efficiency in existing technologies have been solved, enabling efficient testing without the need for separate development of testing tools.
Patent Information
- Application Number
- CN202111092917.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-17
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2041-09-17
AI Technical Summary
Existing technologies require the development of separate testing tools for each application system, resulting in high testing costs and low efficiency.
A big data testing platform is provided, including a data preparation module and an automated testing module, which are used to determine and send test information, call the test tools corresponding to the business functions of the object under test, automatically compare and process the results, reduce testing costs and improve efficiency.
By using a big data testing platform, various application systems or data platforms can be tested without the need to develop separate testing tools, thus reducing testing costs and improving testing efficiency.
Smart Images

Figure CN113742244B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a big data testing platform and a data processing method. Background Art
[0002] With the development of the Internet, more and more application systems have been developed. Before the application system is actually used, it is necessary to test the application system. The current testing method for the application system is to develop a corresponding test tool for the application system to be tested, and then use the test tool to test the application system to be tested. However, due to the large number and variety of application systems, developing a test tool for each application system will waste a lot of manpower and material resources, resulting in high testing costs and low testing efficiency. Summary of the invention
[0003] In view of this, an embodiment of the present invention provides a big data testing platform and a data processing method to solve the problems of high testing cost and low testing efficiency existing in the existing testing application system.
[0004] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:
[0005] A first aspect of an embodiment of the present invention discloses a big data testing platform, the big data testing platform at least comprising: a data preparation module and an automated testing module;
[0006] The data preparation module is used to determine multiple test information and send the multiple test information to the object to be tested for processing, wherein the test information is pre-built information or information obtained from a source system, and the object to be tested is a big data platform or an application system;
[0007] The automated testing module is used to call a testing tool corresponding to the business function of the object to be tested, process each item of the test information to obtain a corresponding first processing result, and obtain a second processing result obtained by the object to be tested processing each item of the test information;
[0008] The automated testing module is further configured to: for each item of the test information, compare the first processing result and the second processing result of the test information to obtain a corresponding comparison result.
[0009] Preferably, the data preparation module for constructing the test information is specifically used to: determine the dimensions used to construct the test information in response to a dimension selection instruction, wherein the dimensions are one or more of a type dimension, a system dimension, and a date dimension; and construct the test information according to the dimensions, preset file distribution rules, and preset data structure rules.
[0010] Preferably, the test information is data or a file, and the automated testing module is further used to: count multiple pieces of the test information to obtain a first data quantity and a first file quantity; obtain a second data quantity and a second file quantity obtained by counting multiple pieces of the test information by the object to be tested; compare the first data quantity and the second data quantity to obtain a first quantity comparison result, and compare the first file quantity and the second file quantity to obtain a second quantity comparison result.
[0011] Preferably, the automated testing module is specifically used to: for each item of the test information, calculate a first hash value of the first processing result of the test information, and calculate a second hash value of the second processing result of the test information; for each item of the test information, compare the first hash value and the second hash value corresponding to the test information to obtain a corresponding comparison result.
[0012] Preferably, the big data testing platform also includes:
[0013] A management module is used to manage and schedule the specified information of the big data testing platform.
[0014] Preferably, the automated testing module is further used to: print the comparison result corresponding to each item of the test information, and highlight the comparison result indicating that the first processing result and the second processing result are inconsistent.
[0015] A second aspect of an embodiment of the present invention discloses a data processing method, the method comprising:
[0016] Determine multiple pieces of test information, and send the multiple pieces of test information to the object to be tested for processing, wherein the test information is pre-built information or information obtained from a source system, and the object to be tested is a big data platform or an application system;
[0017] Calling a test tool corresponding to the business function of the object to be tested, and processing each item of the test information to obtain a corresponding first processing result;
[0018] Acquire a second processing result obtained by the object to be tested processing each item of the test information;
[0019] For each item of the test information, the first processing result and the second processing result of the test information are compared to obtain a corresponding comparison result.
[0020] Preferably, the process of constructing the test information includes:
[0021] In response to the dimension selection instruction, determining a dimension for constructing the test information, the dimension being one or more of a type dimension, a system dimension, and a date dimension;
[0022] The test information is constructed according to the dimension, the preset file distribution rule and the preset data structure rule.
[0023] Preferably, for each item of the test information, comparing the first processing result and the second processing result of the test information to obtain a corresponding comparison result includes:
[0024] For each item of the test information, calculating a first Hash value of the first processing result of the test information, and calculating a second Hash value of the second processing result of the test information;
[0025] For each item of the test information, the first Hash value and the second Hash value corresponding to the test information are compared to obtain a corresponding comparison result.
[0026] Preferably, it also includes:
[0027] The comparison result corresponding to each item of the test information is printed, and the comparison result indicating that the first processing result and the second processing result are inconsistent is highlighted.
[0028] Based on the above-mentioned embodiment of the present invention, a big data testing platform and data processing method are provided, and the big data testing platform at least includes: a data preparation module and an automated testing module; the data preparation module determines multiple test information, and sends the multiple test information to the object to be tested for processing, and the test information is pre-built information or information obtained from the source system; the automated testing module calls the test tool corresponding to the business function of the object to be tested, processes each test information to obtain the corresponding first processing result, and obtains the second processing result obtained by the object to be tested for processing each test information; for each test information, the automated testing module compares the first processing result and the second processing result of the test information to obtain the corresponding comparison result. The testing of various application systems or data platforms can be completed through the big data testing platform, and there is no need to develop corresponding testing tools separately, thereby reducing testing costs and improving testing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0030] Figure 1 A structural block diagram of a big data testing platform provided by an embodiment of the present invention;
[0031] Figure 2 A functional schematic diagram of a data preparation module provided in an embodiment of the present invention;
[0032] Figure 3 A schematic diagram of the execution principle of the data preparation module provided in an embodiment of the present invention;
[0033] Figure 4 A schematic diagram of the execution principle of the automated test module provided in an embodiment of the present invention;
[0034] Figure 5 A schematic diagram of the interaction between the data preparation module and the automated testing module provided in an embodiment of the present invention;
[0035] Figure 6 A functional schematic diagram of a management module provided in an embodiment of the present invention;
[0036] Figure 7 A schematic diagram of the architecture of a big data testing platform provided by an embodiment of the present invention;
[0037] Figure 8 A flowchart of a data processing method provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0039] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.
[0040] It should be noted that the big data testing platform and data processing method provided by the present invention can be used in the field of big data. The above is only an example and does not limit the application field of the big data testing platform and data processing method provided by the present invention.
[0041] As can be seen from the background technology, when testing an application system, it is necessary to develop a corresponding test tool for the application system to be tested, and then use the test tool to test the application system to be tested. However, due to the large number and variety of application systems, developing a test tool for each application system will waste a lot of manpower and material resources, resulting in high testing costs and low testing efficiency.
[0042] Therefore, an embodiment of the present invention provides a big data testing platform and a data processing method, wherein the big data testing platform includes a data preparation module and an automated testing module. The data preparation module determines multiple test information and sends the multiple test information to the object to be tested for processing. The automated testing module calls the test tool corresponding to the business function of the object to be tested, processes each test information to obtain a corresponding first processing result, and obtains a second processing result obtained by the object to be tested processing each test information. For each test information, the automated testing module compares the first processing result and the second processing result of the test information to obtain a corresponding comparison result. The testing of various application systems or data platforms can be completed through the big data testing platform, and there is no need to develop corresponding testing tools separately, so as to reduce testing costs and improve testing efficiency.
[0043] See also Figure 1 , shows a structural block diagram of a big data testing platform provided by an embodiment of the present invention, and the big data testing platform at least includes: a data preparation module 100 and an automated testing module 200.
[0044] Specifically, the data preparation module 100 is used to determine multiple test information and send the multiple test information to the object to be tested for processing. The test information is pre-constructed information or information obtained from a source system, and the object to be tested is a big data platform or an application system; the automated testing module 200 is used to call a test tool corresponding to the business function of the object to be tested, process each test information to obtain a corresponding first processing result, and obtain a second processing result obtained by the object to be tested for processing each test information; the automated testing module 200 is also used to compare the first processing result and the second processing result of the test information for each of the test information to obtain a corresponding comparison result.
[0045] It is understandable that the test information determined by the data preparation module 100 can be information constructed according to the selected dimensions (such as system dimension, date dimension and type dimension, etc.), or it can be information obtained from the source system connected to the big data testing platform (which can be understood as real source information). In other words, the test information determined by the data preparation module 100 can be constructed or obtained from the source system. For example, the test information determined by the data preparation module 100 includes but is not limited to: data (or files) sent by the source system, functional test environment data (or files), desensitized production data (or files) and constructed data (or files).
[0046] It should be noted that one of the functions of the data preparation module 100 is to determine the test information mentioned above. The data preparation module 100 can also store the data obtained by other modules of the big data test platform processing related information for use by various modules of the big data test platform. Figure 2 The functional schematic diagram of the data preparation module shown in the figure shows that the data preparation module 100 can obtain test information from the source system through interface access or transmission protocol, and construct structured or semi-structured test information according to dimensions such as system dimension, date dimension and type dimension, and store the structure obtained by processing the test information by the test module of the big data testing platform in the database for use by various test modules.
[0047] In some specific embodiments, the specific process of the data preparation module 100 to construct the test information is as follows: in response to the dimension selection instruction, determine the dimension used to construct the test information, the dimension being one or more of the type dimension, the system dimension and the date dimension; construct the test information according to the determined dimension, the preset file distribution rule and the preset data structure rule. Specifically, the construction of the test data is carried out according to the standard library (including table data information and table structure, etc.), and the corresponding fields (such as system number and provincial association line number, etc.) in the basic table data information of the standard library are extracted and spliced according to the standard file name naming specification, and the table structure in the standard library is assigned, so as to generate test information that complies with the file distribution rules and data structure rules.
[0048] The above embodiments are related explanations about the data preparation module 100 determining multiple test information. It can be seen that the multiple test information determined by the data preparation module 100 can be constructed or obtained from the source system; specifically, according to different test rounds and different test stages of the object to be tested, choose to construct the test information or obtain the test information from the source system. For example: in the early stage of the test of the object to be tested, since the real information is not in place (such as the desensitized production information is not in place), the construction method can be used to construct the test information for testing. After the real information is in place, the test information can be obtained from the source system for testing.
[0049] To better explain how the data preparation module 100 determines multiple test information, Figure 3 The execution principle diagram of the data preparation module shown is used as an example to illustrate that: Figure 3 For illustration purposes only, and Figure 3 The test information from the source system is production information. Figure 3 The following steps are involved:
[0050] Step S301: Determine whether to construct test information according to the user's operation instruction. If the test information is to be constructed, execute steps S302 to S303; if the test information is not to be constructed, execute steps S304 to S306.
[0051] In the specific implementation of step S301 , the data preparation module 100 may determine whether to construct test data in response to an operation instruction of a user.
[0052] Step S302: In response to the dimension selection instruction of the user, determine the dimension of the test information, and construct and read the test information.
[0053] In the specific implementation of step S302, in response to the user's dimension selection instruction, the dimensions of the test information (system dimension, type dimension and date dimension) are determined, and the test information is constructed and read according to the selected dimensions combined with the file distribution rules and data structure rules.
[0054] Step S303: Determine whether the construction and reading of the test information are completed. If so, end the process. If not, return to continue executing step S302.
[0055] Step S304: In response to the user's information selection instruction, select test information.
[0056] It should be noted that, in this example, the selected test information is production information.
[0057] Step S305: Access test information from the source system.
[0058] It should be noted that in this example, the source systems are the basic system and the direct connection system.
[0059] Step S306: Determine whether the access of the test information is completed. If yes, end the process; if no, return to step S305.
[0060] The contents of the above embodiments are related to the data preparation module 100 determining the test information.
[0061] It is understandable that after the data preparation module 100 determines the multiple test information, it transmits the determined multiple test information to the automated test module 200 and the object to be tested, and the automated test module 200 and the object to be tested process the multiple test information respectively. The automated test module 200 calls the test tool corresponding to the business function of the object to be tested to process each test information and obtain a corresponding first processing result; the object to be tested processes each test information to obtain a corresponding second processing result, and the automated test module 200 obtains the second processing result corresponding to each test information from the object to be tested.
[0062] It should be noted that the automated testing module 200 includes multiple pre-built testing tools, such as: interface automated testing tools, interface automated testing tools, data analysis automated testing tools and full comparison testing tools, among which the full comparison testing tools at least include file processing automated testing tools, data processing automated testing tools, data storage automated testing tools and data model automated testing tools.
[0063] In one example, the automated testing module 200 calls the full comparison testing tool to process each item of test information to obtain a corresponding first processing result. Specifically, each item of test information can be processed by executing a program. Similarly, the object to be tested can also process each item of test information by executing a program.
[0064] To better explain the process of the automated test module 200 and the object to be tested processing each test information, Figure 4 The schematic diagram of the execution principle of the automated test module is shown as an example. It should be noted that: Figure 4 For illustration purposes only, Figure 4 The following steps are involved:
[0065] Step S401: In response to the user's execution instruction, determine whether to execute the program. If the program is to be executed, execute steps S402 and S403. If it is determined not to execute the program, return to continue waiting for the execution instruction of the program.
[0066] Step S402: The big data testing platform executes a program to process each item of test information to obtain a corresponding first processing result.
[0067] In the specific process of implementing step S402, the automated testing module 200 executes a program and calls a testing tool to process each item of test information to obtain a corresponding first processing result.
[0068] Step S403: the object to be tested executes the program, and processes each item of test information to obtain a corresponding second processing result.
[0069] Step S404: Determine whether the program has been executed. If it has been executed, execute step S405; if it has not been executed, return to continue to execute steps S402 and S403.
[0070] Step S405: The automated testing module compares the first processing result and the second processing result of the test information to obtain a corresponding comparison result.
[0071] The above contents are examples of how the automated testing module 200 and the object to be tested process each item of test information.
[0072] In some specific embodiments, the specific manner in which the automated testing module 200 compares the first processing result and the second processing result of each test information is as follows: for each test information, the automated testing module 200 calculates the first hash value (such as MD5 value) of the first processing result of the test information, and calculates the second hash value of the second processing result of the test information; for each test information, the automated testing module 200 compares the first hash value and the second hash value corresponding to the test information to obtain a corresponding comparison result. For each test information, if the first hash value of the first processing result and the second hash value of the second processing result corresponding to the test information are consistent, it means that the result of the automated testing module 200 processing the test information is consistent with the result of the test object processing the test information, that is, it means that the first processing result and the second processing result of the test information are consistent.
[0073] In some specific embodiments, after obtaining the comparison results corresponding to each item of test information, the automated testing module 200 prints the comparison results corresponding to each item of test information, and highlights the comparison results indicating that the first processing result and the second processing result are inconsistent; specifically, a comparison tool can be used to print the comparison results corresponding to each item of test information, and highlight the comparison results indicating that the first processing result and the second processing result are inconsistent.
[0074] It can be understood that each test information is data or a file, and the automated testing module 200 can verify the file integrity and data correctness of all test information. Specifically, verifying the file integrity mainly includes verifying the number and content of files corresponding to all test information, and verifying the data correctness mainly includes verifying the number and content of data corresponding to all test information.
[0075] It should be noted that verifying the contents corresponding to all test information is to compare the first processing results and the second processing results corresponding to the test information mentioned in the above embodiment to obtain the corresponding comparison results. For specific contents, please refer to the relevant contents about comparing the first processing results and the second processing results corresponding to the test information in the above embodiment, which will not be repeated here.
[0076] The specific process of verifying the number of files and the number of data corresponding to all test information is as follows: the automated testing module 200 counts multiple test information to obtain a first data number and a first file number. Specifically, the first data number is obtained by counting multiple test information through a data item counting function, and the first file number is obtained by counting multiple test information through a file number counting function; the automated testing module 200 obtains the second data number and the second file number obtained by counting multiple test information by the object to be tested. It should be noted that the object to be tested can count multiple test information through a monitoring site to obtain the second data number and the second file number; the automated testing module 200 compares the first data number and the second data number to obtain a first quantity comparison result, and compares the first file number and the second file number to obtain a second quantity comparison result.
[0077] The automated testing module 200 can verify the file integrity and data correctness of all test information based on the first quantity comparison result, the second quantity comparison result, and the comparison result corresponding to each item of test information determined in the above embodiment.
[0078] It can be understood that when the test information is a file, the processing of the test information by the automated testing module 200 includes but is not limited to file decompression, file transcoding and file backup; when the test information is data, the processing of the test information by the automated testing module 200 includes but is not limited to table structure verification, model verification and semi-structured verification.
[0079] It can be seen from the contents of the above embodiments that the data preparation module 100 determines the test information and transmits it to the object to be tested and the automated test module 200 for subsequent processing. Figure 5 The interactive diagram of the data preparation module and the automated test module shown is used as an example. It should be noted that: Figure 5 In this paper, the object to be tested is the big data platform.
[0080] See also Figure 5 The data preparation module 100 can construct the test information, or access and obtain the test information from the source system through FTP or HIVE, and store the test information in a database, a file system or a server.
[0081] The automated test module 200 and the big data platform can call the test information from the database, file system or server, the big data platform uses its corresponding business function to process the test information to obtain the second processing result, and the automated test module 200 uses the test tool to process the test information to obtain the first processing result. The automated test module 200 can compare the first processing result and the second processing result corresponding to each test information and output the comparison result. The automated test module 200 can also verify the file integrity and data correctness of the test information and output the verification result.
[0082] By utilizing the big data testing platform involved in the above embodiments, the testing of various application systems or data platforms can be completed without the need to develop corresponding testing tools separately, thereby reducing testing costs and improving testing efficiency.
[0083] In some embodiments of the present invention, the big data testing platform further includes: a management module;
[0084] The management module is used to manage and schedule the specified information of the big data testing platform; specifically, the management module is used to manage and schedule the scripts and testing tools in the big data testing platform; the management module is connected to the internal information system and precise testing middle platform of the big data testing platform, so as to realize functions such as user login and test data statistics.
[0085] The functions of the management module are mainly test management, scheduling management and public management, such as Figure 6 The functional schematic diagram of the management module is shown, and the management module is divided into a test management module, a scheduling management module and a public management module.
[0086] Among them, the test management module is connected with the automated testing platform and the quality intelligent management platform, and the functions of the test management module include script management, tool management, task management, etc.; the scheduling management module is connected with the precise testing middle platform, the automated testing platform and the JIRA platform, and the functions of the scheduling management module include data scheduling, script scheduling and tool scheduling, etc.; the public management module is connected with the internal information system, and the functions of the public management sub-module include user management, authority management and log management.
[0087] It can be seen from the contents of the above embodiments that the big data testing platform in the embodiments of the present invention mainly includes a data preparation module, an automated testing module, and a management module, wherein the management module is mainly divided into a scheduling management module, a test management module, and a public management module; Figure 7 The schematic diagram of the architecture of the big data testing platform is shown, and the big data testing platform can be used to complete tests such as the big data platform waiting for test objects,
[0088] In some specific embodiments, Figure 7The schematic diagram of the architecture of the big data testing platform is shown. Each module of the big data testing platform is built based on microservices. Any module in the big data testing platform can be called individually, or multiple modules in the big data testing platform can be called in series through pipelines, and the execution records and results of each module can be automatically recorded.
[0089] It should be noted that if Figure 7 The schematic diagram of the architecture of the big data testing platform shown, the front end of the big data testing platform is built based on Vue.js, the middle platform of the big data testing platform is built based on Python, and the big data testing platform uses the back-end to store related data.
[0090] The above content is some description about the big data testing platform. Some embodiments of the present invention also disclose a data processing method.
[0091] See also Figure 8 , shows a flow chart of a data processing method provided by an embodiment of the present invention, the data processing method comprising:
[0092] Step S801: Determine multiple test information and send the multiple test information to the object to be tested for processing.
[0093] It should be noted that the test information is pre-built information or information obtained from a source system, and the object to be tested is a big data platform or an application system.
[0094] In some specific embodiments, the process of constructing test information includes: responding to a dimension selection instruction, determining the dimensions used to construct the test information, the dimensions being one or more of a type dimension, a system dimension, and a date dimension; and constructing the test information according to the dimensions, preset file distribution rules, and preset data structure rules.
[0095] Step S802: calling a test tool corresponding to the business function of the object to be tested, and processing each item of test information to obtain a corresponding first processing result.
[0096] Step S803: obtaining a second processing result obtained by the object to be tested processing each item of test information.
[0097] Step S804: for each item of test information, compare the first processing result and the second processing result of the test information to obtain a corresponding comparison result.
[0098] In the specific implementation of step S804, for each test information, a first hash value of the first processing result of the test information is calculated, and a second hash value of the second processing result of the test information is calculated. For each test information, the first hash value and the second hash value corresponding to the test information are compared to obtain a corresponding comparison result.
[0099] Preferably, the comparison result corresponding to each item of test information is printed, and the comparison result indicating that the first processing result and the second processing result are inconsistent is highlighted.
[0100] Preferably, the test information is data or files, and multiple pieces of test information are counted to obtain a first data quantity and a first file quantity; a second data quantity and a second file quantity are obtained by counting multiple pieces of test information from the object to be tested; the first data quantity and the second data quantity are compared to obtain a first quantity comparison result, and the first file quantity and the second file quantity are compared to obtain a second quantity comparison result.
[0101] It should be noted that Figure 8 The execution principle of each step in the above text has been described in detail in the big data testing platform, which will not be repeated here.
[0102] In an embodiment of the present invention, multiple pieces of test information are determined, and the multiple pieces of test information are sent to the object to be tested for processing. The test tool corresponding to the business function of the object to be tested is called, each piece of test information is processed to obtain a corresponding first processing result, and a second processing result obtained by the object to be tested processing each piece of test information is obtained. For each piece of test information, the first processing result and the second processing result of the test information are compared to obtain a corresponding comparison result. The testing of various application systems or data platforms can be completed through the big data testing platform, without the need to develop corresponding testing tools separately, thereby reducing testing costs and improving testing efficiency.
[0103] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.
[0104] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0105] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A big data testing platform, characterized in that: The big data testing platform at least includes: a data preparation module and an automated testing module; The data preparation module is used to determine multiple test information and send the multiple test information to the object to be tested for processing, wherein the test information is pre-built information or information obtained from a source system, and the object to be tested is a big data platform or an application system; The automated testing module is used to call a testing tool corresponding to the business function of the object to be tested, process each item of the test information to obtain a corresponding first processing result, and obtain a second processing result obtained by the object to be tested processing each item of the test information; The automated testing module is further configured to: for each item of the test information, compare the first processing result and the second processing result of the test information to obtain a corresponding comparison result; The data preparation module determines multiple test information, including: Determine whether to construct test information according to the user's operation instructions; If the test information is constructed, in response to the dimension selection instruction of the user, the dimension of the test information is determined, and the test information is constructed and read; If the test information is not constructed, in response to the information selection instruction of the user, selecting the test information and accessing the test information from the source system; The automated testing module calls a testing tool corresponding to the business function of the object to be tested, processes each item of the test information to obtain a corresponding first processing result, and obtains a second processing result obtained by the object to be tested processing each item of the test information, including: In response to an execution instruction of the user, determining whether to execute a program; If it is determined to execute the program, the program is executed to call a test tool corresponding to the business function of the object to be tested to process each item of the test information to obtain a corresponding first processing result; A second processing result is obtained from the object to be tested, where the second processing result is obtained by the object to be tested executing the program to process each item of the test information.
2. The big data testing platform according to claim 1, characterized in that: The data preparation module for constructing the test information is specifically used to: determine the dimensions for constructing the test information in response to a dimension selection instruction, wherein the dimensions are one or more of a type dimension, a system dimension, and a date dimension; and construct the test information according to the dimensions, preset file distribution rules, and preset data structure rules.
3. The big data testing platform according to claim 1, characterized in that: The test information is data or files, and the automated test module is further used to: obtain a first data quantity and a first file quantity by counting multiple test information; obtain a second data quantity and a second file quantity obtained by counting multiple test information by the object to be tested; The first data quantity and the second data quantity are compared to obtain a first quantity comparison result, and the first file quantity and the second file quantity are compared to obtain a second quantity comparison result.
4. The big data testing platform according to claim 1, characterized in that: The automated testing module is specifically configured to: for each item of the test information, calculate a first hash value of the first processing result of the test information, and calculate a second hash value of the second processing result of the test information; For each item of the test information, the first Hash value and the second Hash value corresponding to the test information are compared to obtain a corresponding comparison result.
5. The big data testing platform according to claim 1, characterized in that: The big data testing platform also includes: A management module is used to manage and schedule the specified information of the big data testing platform.
6. The big data testing platform according to claim 1, characterized in that: The automated testing module is further configured to print the comparison result corresponding to each item of the test information, and highlight the comparison result indicating that the first processing result and the second processing result are inconsistent.
7. A data processing method, characterized in that: The method comprises: Determine multiple pieces of test information, and send the multiple pieces of test information to the object to be tested for processing, wherein the test information is pre-built information or information obtained from a source system, and the object to be tested is a big data platform or an application system; Calling a test tool corresponding to the business function of the object to be tested, and processing each item of the test information to obtain a corresponding first processing result; Acquire a second processing result obtained by the object to be tested processing each item of the test information; For each item of the test information, compare the first processing result and the second processing result of the test information to obtain a corresponding comparison result; Identify multiple test information, including: Determine whether to construct test information according to the user's operation instructions; If the test information is constructed, in response to the dimension selection instruction of the user, the dimension of the test information is determined, and the test information is constructed and read; If the test information is not constructed, in response to the information selection instruction of the user, selecting the test information and accessing the test information from the source system; Calling a test tool corresponding to the business function of the object to be tested, processing each item of the test information to obtain a corresponding first processing result; obtaining a second processing result obtained by the object to be tested processing each item of the test information, including: In response to an execution instruction of the user, determining whether to execute a program; If it is determined to execute the program, the program is executed to call a test tool corresponding to the business function of the object to be tested to process each item of the test information to obtain a corresponding first processing result; A second processing result is obtained from the object to be tested, where the second processing result is obtained by the object to be tested executing the program to process each item of the test information.
8. The method according to claim 7, characterized in that The process of constructing the test information includes: In response to the dimension selection instruction, determining a dimension for constructing the test information, the dimension being one or more of a type dimension, a system dimension, and a date dimension; The test information is constructed according to the dimension, the preset file distribution rule and the preset data structure rule.
9. The method according to claim 7, characterized in that: For each item of the test information, the first processing result and the second processing result of the test information are compared to obtain a corresponding comparison result, including: For each item of the test information, calculating a first Hash value of the first processing result of the test information, and calculating a second Hash value of the second processing result of the test information; For each item of the test information, the first Hash value and the second Hash value corresponding to the test information are compared to obtain a corresponding comparison result.
10. The method according to claim 7, characterized in that Also includes: The comparison result corresponding to each item of the test information is printed, and the comparison result indicating that the first processing result and the second processing result are inconsistent is highlighted.
