Pyspark-based automated testing method and device, computer device and storage medium

Abstract

The present application relates to the technical field of big data, and discloses an automatic test method and device based on pyspark, computer equipment and a storage medium. The method comprises the following steps: obtaining missing monitoring tasks in a task scheduling platform at a regular time; obtaining pyspark logic written in a preset editing platform when the missing monitoring tasks are running; executing automatic test scripts corresponding to a plurality of data clusters respectively through the pyspark logic; monitoring monitoring data in each data cluster through the automatic test scripts to obtain monitoring results; and displaying the monitoring results to a preset receiving platform through a task flow display platform. The present application is used for improving data interaction, thereby improving data display capability and the efficiency of automatic test.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing in big data, and particularly discloses a pyspark-based automatic testing method and device, computer equipment and a storage medium. BACKGROUND

[0002] As one of the current popular emerging technologies, big data is used more and more widely in various enterprises. The automatic testing technology is a process of simulating manual testing by recording or writing scripts, and the testing cases are executed by recycling or running scripts to replace manual verification of product systems and avoid a large amount of repetitive work. However, the automatic testing technology in big data still has some deficiencies, and it is not perfect in automatic testing and visualization, which leads to high development and testing threshold, complicated processing flow, and low automatic testing efficiency. In addition, the components of the ecological new system corresponding to the big data are relatively independent, and the data interaction mechanism between the components is not perfect, which leads to relatively difficult data interaction, ineffective management of development tasks, and friendly display.

[0003] In summary, the technical personnel in the field urgently need to find a new solution to solve the above problems. SUMMARY

[0004] Therefore, it is necessary to provide a pyspark-based automatic testing method and device, computer equipment and a storage medium to improve data interaction, thereby improving data display capability and automatic testing efficiency through process cooperation.

[0005] A pyspark-based automatic testing method, comprising:

[0006] acquiring an omitted monitoring task in a task scheduling platform at a regular time;

[0007] obtaining a pyspark logic written in a preset editing platform when the omitted monitoring task is running, executing an automatic testing script corresponding to each data cluster through the pyspark logic, monitoring monitoring data in each data cluster through the automatic testing script, and obtaining a monitoring result;

[0008] displaying the monitoring result to a preset receiving platform through a task flow display platform.

[0009] A pyspark-based automatic testing device, comprising:

[0010] an acquisition module configured to acquire an omitted monitoring task in a task scheduling platform at a regular time;

[0011] The monitoring module is configured to, when the missing monitoring task is executed, acquire a pyspark logic written in a preset editing platform, execute an automatic test script corresponding to each data cluster through the pyspark logic, and monitor monitoring data in each data cluster through the automatic test script to obtain a monitoring result.

[0012] The display module is configured to display the monitoring result to a preset receiving platform through a task flow display platform.

[0013] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the above-mentioned automatic test method based on pyspark.

[0014] A computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the above-mentioned automatic test method based on pyspark.

[0015] The above-mentioned automatic test method based on pyspark, device, computer device and storage medium are based on the mutual cooperation between the task scheduling platform, the preset editing platform and the task flow of the three main modules. After the monitoring task is executed, the automatic test script corresponding to each data cluster is executed through the pyspark logic written in the preset editing platform, the monitoring result corresponding to each data cluster is obtained, and finally the monitoring result is visualized and displayed. The data interaction ability of the processing flow is improved, the problem of imperfect data interaction in the flow can be avoided in the overall flow, and the data display ability and the efficiency of big data automatic test and development are improved. At the same time, the task scheduling platform, the preset editing platform and the task flow are used to reduce most of the tedious processing flow, and the processing flow is solidified. The tool can be reused to execute the processing flow in the future, and the efficiency of automatic test is improved. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0017] Figure 1 is an application environment diagram of a kind of automatic test method based on pyspark in an embodiment of the present application;

[0018] Figure 2is a flowchart of a pyspark-based automatic testing method according to an embodiment of the present application;

[0019] Figure 3 is a structural diagram of a pyspark-based automatic testing device according to an embodiment of the present application;

[0020] Figure 4 is a schematic diagram of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0022] The present application provides a pyspark-based automatic testing method, which can be applied in an application environment as shown in Figure 1 , wherein the client communicates with the server through the network. The client can include but is not limited to various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices.

[0023] In an embodiment, as shown in Figure 2 , a pyspark-based automatic testing method is provided, which is applied in the server as shown in Figure 1 . The architecture of the method is a pyspark-based automatic testing tool running in the server, which includes the following steps S10-S30:

[0024] S10, periodically obtaining a missing monitoring task in a task scheduling platform;

[0025] Understandably, the architecture of the pyspark-based automatic testing tool includes a task scheduling platform, a preset editing platform and a task flow display platform. There are a plurality of pre-set missing monitoring tasks in the task scheduling platform. The missing monitoring tasks can be periodically scheduled and run through the platform. The missing monitoring task refers to the re-monitoring of a new product or a new system or a new program (there may be a missed detection problem previously, which needs to be monitored again). The missing monitoring task is associated with the automatic testing of big data, which includes but is not limited to testing data quality (including data field consistency, accuracy, repeatability, validity and data integrity), performance testing (including job completion time, memory occupancy and data throughput).

[0026] S20, when running the missing monitoring task, obtaining pyspark logic written in a preset editing platform, executing an automatic test script corresponding to each data cluster through the pyspark logic, and monitoring monitoring data in each data cluster through the automatic test script to obtain a monitoring result;

[0027] Understandably, the preset editing platform is jupyter, which is essentially a Web application for creating and sharing literate program documents, supporting real-time code, mathematical equations, visualizations, and markdown. Its uses include data cleaning and transformation, numerical simulation, and statistical modeling. Pyspark is a development language edited by users through the jupyter platform, and the pyspark logic is a business logic edited through the platform (the business logic is configured in the rule configuration module of the pyspark-based automated testing tool as a monitoring rule when the tool is executed). The pyspark logic includes automatic test scripts for executing multiple data cluster tests, such as automatic test script 1 for executing hive cluster, automatic test script 2 for executing spark cluster, and automatic test script 3 for executing hbase cluster. The monitoring result reflects the result corresponding to this automated test, such as whether the data quality of the monitoring data has a problem or whether the product performance has a defect.

[0028] S30, displaying the monitoring result to a preset receiving platform through a task flow display platform.

[0029] Understandably, the task flow display platform is a visualization platform in the pyspark-based automated testing tool that displays various running states of the entire monitoring task in the running phase. Through the platform, the entire automated test flow can be clearly understood.

[0030] In the embodiments of steps S10 to S30, based on the mutual cooperation between the three main modules (task scheduling platform, preset editing platform, and task flow display platform), after running the monitoring task, the pyspark logic written in the preset editing platform is executed to execute the automatic test script corresponding to each data cluster, and the monitoring result corresponding to each data cluster is obtained. Finally, the monitoring result is visualized to improve the data interaction capability of the processing flow, avoid the problem of imperfect data interaction in the flow, and improve the data display capability and the efficiency of big data automated testing and development. Through the three main modules, most of the tedious processing flow is reduced, the processing flow is solidified, and the tool can be reused to execute the processing flow in the future, improving the efficiency of automated testing.

[0031] Further, the pyspark logic written in the preset editing platform is acquired, the automatic test scripts corresponding to the plurality of data clusters are executed through the pyspark logic, and the monitoring data in each data cluster is monitored through the automatic test scripts to obtain monitoring results, comprising:

[0032] The pyspark logic containing the automatic test scripts corresponding to the plurality of data clusters is acquired, and the pyspark logic is used as a monitoring rule;

[0033] The monitoring rule is parsed to obtain a parent-child monitoring rule; all the automatic test scripts corresponding to the plurality of data clusters in the parent-child monitoring rule are arranged according to an execution sequence;

[0034] The automatic test scripts corresponding to the plurality of data clusters are executed according to the execution sequence in the parent-child monitoring rule, and the monitoring data in each data cluster is monitored in sequence through the executed automatic test scripts to obtain a plurality of monitoring results.

[0035] Understandably, the pyspark logic used as a monitoring rule can enable the tool to run automatically, the parent-child rule containing the execution sequence, the priority of the parent rule being higher than that of the child rule, and the automatic scripts in the pyspark logic being executed according to the execution sequence, such as executing the automatic test script 1 of the hive data cluster-first and then executing the automatic test script 2 of the spark data cluster, or the automatic test script 1 containing a plurality of test branches, the data 1 being automatically tested first and then the data 2 being automatically tested, wherein the specific operations of the pyspark logic include but are not limited to counting the missing values, filling the missing values, deleting a column of data, inserting a hive partition table, calculating the time difference between two columns, multi-condition filtering, replacing the value of a column according to the value of another column, replacing a value in a column, and filtering the rows with missing values; the parent-child monitoring rule can be obtained through a rule engine, the rule engine can be arranged in the architecture in the tool, and the rule engine can also be used to issue the execution commands of the monitoring rule; the embodiment mainly parses the execution sequence in the monitoring rule into the parent-child monitoring rule to enable the automatic test scripts to be executed according to the execution sequence, and the execution process is standardized.

[0036] Further, the running of the missing monitoring task comprises:

[0037] When it is determined that there is a dependent task having a dependency relationship with the missing monitoring task, the dependent task having the dependency relationship with the missing monitoring task is acquired, and the missing monitoring task is run after the dependent task is run.

[0038] Understandably, the dependent task refers to a task that needs to be executed before the current monitoring task is executed, and the dependent task and the monitoring task also have a sequential execution order. After the dependent task is executed, the monitoring task can be executed (the monitoring task needs to depend on the result of the execution of the dependent task). For example, the dependent task is to combine data in a data cluster, and the monitoring task is to judge the quantity and quality of the data in the data cluster (including the consistency, accuracy, repeatability, effectiveness and data integrity of the fields in the data). The setting of the dependent task and the monitoring task is dynamically managed on the task scheduling platform (that is, dynamically set on the corresponding page of the task scheduling platform through the Web, and dynamically changed according to actual needs). The dynamic management includes but is not limited to new, edit and delete. This embodiment sets the relationship between the dependent task and the monitoring task, and stipulates that the monitoring task can be executed only after the dependent task is executed, so that the execution process is standardized.

[0039] Further, the monitoring result is displayed to the preset receiving platform through the task flow display platform, including:

[0040] The task information table, the task log corresponding to the missing monitoring task and the dependent task, and the task running state determined from the monitoring result are obtained through the task scheduling platform;

[0041] The missing monitoring task, the task running state, the task information table and the task running state are spliced into a combined JSON through JAVA microservices;

[0042] The combined JSON is displayed to the preset receiving platform in the form of a preset box line chart through the task flow display platform.

[0043] Understandably, the task information table refers to the information corresponding to the monitoring task and the dependent task, and the information includes but is not limited to the primary key ID of the task, the task type, and the task field; the task log refers to the record of the process experienced at each time point in the entire task execution process; the task running state refers to whether the task execution is successful (whether the monitoring result can be obtained) in the task execution process; the JAVA microservice is a traditional Web development mode, which can package all functions in a package and run multiple functions at the same time, and JSON is a data structure that replaces XML, which is more compact than XML but has no less description ability. Because it is small, network data transmission will reduce more traffic and speed up. Specifically, a combined JSON is formed by JAVA microservice for a variety of visual information, and after the combined JSON is parsed, a preset style box plot is obtained and displayed to a preset display platform. Among them, the preset box plot form can link various related data in the form of lines and box plots, and an independent data can be represented by a preset box plot. The embodiment combines multiple information together and displays all information to the preset display platform, improving the observation effect of all information for users.

[0044] Further, after obtaining the pyspark logic written in the preset editing platform, the method further comprises:

[0045] obtaining the changed pyspark logic corresponding to the new missing monitoring task;

[0046] obtaining the new automatic test script corresponding to the new parent-child monitoring rule and having a execution sequence according to the changed pyspark logic;

[0047] monitoring the monitoring data in each data cluster in sequence according to the execution sequence through the new automatic test script to obtain a plurality of new monitoring results.

[0048] Understandably, the embodiment provides a change page of the preset editing platform for updating the pyspark logic. In the automatic test process, the automatic test script needs to be changed in real time according to different test requirements, and the test case needs to be iterated quickly. The tool provided in the embodiment does not need to go through any re-submission and handover script process, and can realize automatic test after changing the script, thereby improving the efficiency of automatic test.

[0049] Further, the monitoring the monitoring data in each data cluster in sequence according to the execution sequence through the new automatic test script to obtain a plurality of new monitoring results, comprises:

[0050] In the preset editing platform, different contexts are obtained according to different data clusters, and distributed data sets are created based on the monitoring data in the different data clusters through the contexts;

[0051] The monitoring data in each distributed data set is monitored in sequence through the automatic test script, and a plurality of monitoring results are obtained.

[0052] Understandably, the Context is the most core module in pyspark, and can be understood as an access point of the monitoring data, connecting various clusters, and then a distributed data set RDD can be obtained, wherein the corresponding Context of Spark is SparkContext, and the corresponding Context of Hive is HiveContext; the distributed data set RDD is a resilient distributed data set, which is an abstract concept of distributed memory, and the RDD is composed of a plurality of partitions, a function operation is performed on the RDD, and the same function operation (such as a map function) is performed on all partitions of the RDD, the RDD can depend on other RDDs, and specifically, there are two methods for creating a distributed data set: 1. an object (such as a Python list object) on an existing data set is automatically partitioned (partitioned) by calling the parallelize method in the SparkContext, and each block is a distributed data set that can be operated in parallel; 2. text data is directly read into a distributed data set to create a distributed data set.

[0053] Further, after the distributed data set is created based on the monitoring data in the data cluster through the Context, the method further comprises:

[0054] The selected preset data conversion mode is concatenated to obtain a total preset data conversion mode;

[0055] After the distributed data set is obtained, the total preset data conversion mode is used for preset data conversion, and a new distributed data set after conversion is obtained;

[0056] The monitoring data in the new distributed data set is monitored again through the automatic test script.

[0057] Understandably, the preset data conversion is carried out according to the preset data conversion mode, so as to obtain a new RDD, one preset data conversion corresponds to one preset data conversion mode, which is realized by a function, the function includes but is not limited to map(), flatMap(), filler(), distinct() and sortBy(), map() is used for executing the same operation on each item of the RDD, flatMap() is used for executing the same operation on the item in the PDD to obtain a list, then all the results of the list are combined to form a new list in a flat manner, filler() screens out the item meeting the condition, distinct() is used for removing the duplicate items in the RDD, and sortBy() is used for sorting the items in the RDD; the total preset data conversion mode is a data conversion mode obtained by concatenating at least two preset data conversion modes selected by the user, the preset data conversion modes can be run in sequence, so as to obtain a new RDD containing multiple processing modes; the embodiment is a new distributed data set creation process, the monitoring data can be continuously updated according to the demand through the new distributed data set, and the new monitoring data is monitored again.

[0058] In another embodiment, for more than two distributed data sets, a new distributed data set is obtained through set operation (such as union, intersection, difference and orthogonal sum); for example, assuming that there are two distributed data sets, RDD1 and RDD2, union: RDD1.union(RDD2): all the items in RDD1 and RDD2 are combined; intersection: RDD1.intersection(RDD2): the intersection of RDD1 and RDD2; difference: RDD1.substract(RDD2): all the items in RDD1 but not in RDD2; orthogonal sum: RDD1.cartesian(RDD2): the Cartesian product of all the elements in RDD1 and RDD2.

[0059] In summary, the above provides a pyspark-based automated testing method. Based on the cooperation between the task scheduling platform, the preset editing platform and the task flow, after the monitoring task is run, the automatic test script corresponding to each data cluster is executed through the pyspark logic written in the preset editing platform, the monitoring result corresponding to each data cluster is obtained, and finally the monitoring result is visualized to improve the data interaction capability of the processing flow. The overall process can avoid the problem of imperfect data interaction in the process, thereby improving the data display capability and the efficiency of big data automated testing and development. At the same time, through the task scheduling platform, the preset editing platform and the task flow, most of the tedious processing flow is reduced and solidified. The tool can be reused to execute the processing flow in the future, improving the efficiency of automated testing. In addition, the parent-child rule or the monitoring task is executed according to the execution sequence, and the execution process is standardized. In addition, a variety of display information is combined together and displayed to the preset display platform, improving the observation effect of the user on all information. In addition, the provided tool does not need to go through any re-submission and handover script process to realize automated testing after changing the script, improving the efficiency of automated testing. In addition, new distributed data sets can be obtained through the preset data conversion mode to realize re-monitoring of new monitoring data, and the efficiency of automated testing can be improved.

[0060] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0061] In an embodiment, the present application also provides a pyspark-based automated testing device, which corresponds one-to-one to the above-mentioned embodiment of the pyspark-based automated testing method. As shown in the figure, the pyspark-based automated testing device comprises an acquisition module 11, a monitoring module 12 and a display module 13. The functions of each module are described in detail as follows: Figure 3

[0062] Further, the pyspark-based automated testing device further comprises:

[0063] The acquisition module 11 is used for acquiring the missed monitoring task in the task scheduling platform at a time;

[0064] The monitoring module 12 is used for acquiring the pyspark logic written in the preset editing platform when the missed monitoring task is run, executing the automatic test script corresponding to each data cluster through the pyspark logic, and monitoring the monitoring data in each data cluster through the automatic test script to obtain the monitoring result.​

[0065] The display module 13 is configured to display the monitoring result to a preset receiving platform through a task flow display platform.

[0066] Further, the monitoring module comprises:

[0067] The first obtaining unit is configured to obtain a written pyspark logic containing an automatic test script corresponding to each data cluster in a preset editing platform, and take the pyspark logic as a monitoring rule;

[0068] The parsing unit is configured to parse the monitoring rule to obtain a parent-child monitoring rule; all automatic test scripts corresponding to each data cluster contained in the parent-child monitoring rule are arranged according to an execution sequence;

[0069] The first monitoring unit is configured to execute the automatic test script corresponding to each data cluster according to the execution sequence in the parent-child monitoring rule, and sequentially monitor the monitoring data in each data cluster through the executed automatic test script to obtain a plurality of monitoring results.

[0070] Further, the monitoring module comprises:

[0071] The determining unit is configured to, when determining that there is a dependent task having a dependency relationship with the missing monitoring task, obtain the dependent task having the dependency relationship with the missing monitoring task, and run the missing monitoring task after the dependent task is run.

[0072] Further, the display module comprises:

[0073] The determining unit is configured to obtain a task information table, a task log corresponding to the missing monitoring task and the dependent task, and a task running state determined from the monitoring result through the task scheduling platform;

[0074] The splicing unit is configured to splice the missing monitoring task, the task running state, the task information table and the task running state into a combined JSON through JAVA microservices;

[0075] The display unit is configured to display the combined JSON in the form of a preset box line chart to the preset receiving platform through the task flow display platform.

[0076] Further, the monitoring module comprises:

[0077] The second obtaining unit is configured to obtain a changed pyspark logic corresponding to a new missing monitoring task;

[0078] The third acquisition unit is configured to obtain a new parent-child monitoring rule by changing the pyspark logic, and obtain a new automatic test script corresponding to the new parent-child monitoring rule and having a sequence of execution;

[0079] The second monitoring unit is configured to sequentially monitor the monitoring data in each data cluster according to the sequence of execution by using the new automatic test script, and obtain a plurality of new monitoring results.

[0080] Further, the first monitoring unit comprises:

[0081] The creation sub-unit is configured to obtain different contexts according to different data clusters in the preset editing platform, and create distributed data sets based on the monitoring data in different data clusters by using the contexts;

[0082] The first monitoring sub-unit is configured to sequentially monitor the monitoring data in each distributed data set by using the automatic test script, and obtain a plurality of monitoring results.

[0083] Further, the first monitoring unit further comprises:

[0084] The concatenation sub-unit is configured to concatenate the selected preset data conversion modes to obtain a total preset data conversion mode.

[0085] The data conversion sub-unit is configured to perform preset data conversion on the obtained distributed data sets according to the total preset data conversion mode, and obtain new distributed data sets after conversion.

[0086] The second monitoring sub-unit is configured to monitor the monitoring data in each new distributed data set again by using the automatic test script.

[0087] The specific limitations of the automatic test device based on pyspark can be referred to the limitations of the automatic test method based on pyspark in the above, which will not be repeated here. Each module in the automatic test device based on pyspark can be realized by software, hardware and their combinations. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each module.

[0088] In one embodiment, a computer device, which can be a server, is provided, and the internal structure diagram thereof can be as shown in Figure 4As shown in the figure. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the data involved in the pyspark-based automated testing method. The network interface of the computer device is used to communicate with the external terminal through the network connection. The computer program is executed by the processor to implement a pyspark-based automated testing method.

[0089] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the pyspark-based automated testing method in the above-mentioned embodiments when executing the computer program, for example Figure 2 The steps S10 to S30 shown in the figure. Alternatively, the processor implements the functions of each module / unit of the pyspark-based automated testing device in the above-mentioned embodiments when executing the computer program, for example Figure 3 The functions of the modules 11 to 13 shown in the figure. To avoid repetition, it will not be repeated here.

[0090] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by the processor to implement the steps of the pyspark-based automated testing method in the above-mentioned embodiments, for example Figure 2 The steps S10 to S30 shown in the figure. Alternatively, the computer program is executed by the processor to implement the functions of each module / unit of the pyspark-based automated testing device in the above-mentioned embodiments, for example Figure 3 The functions of the modules 11 to 13 shown in the figure. To avoid repetition, it will not be repeated here.

[0091] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, storage, database or other medium used in each embodiment of the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0092] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of functional units and modules is exemplified. In actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the above-described functions.

[0093] The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, but not to limit it. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features. Such modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A pyspark-based automated testing method, characterized in that, The method comprises: Timing obtains the missing monitoring task in the task scheduling platform; When running the missing monitoring task, obtain the pyspark logic written in the preset editing platform, execute the automatic test script corresponding to each data cluster through the pyspark logic, and monitor the monitoring data in each data cluster through the automatic test script to obtain the monitoring result; The method comprises: Obtain the pyspark logic written in the preset editing platform, execute the automatic test script corresponding to each data cluster through the pyspark logic, and monitor the monitoring data in each data cluster through the automatic test script to obtain the monitoring result, comprising: Obtain the pyspark logic written in the preset editing platform, execute the automatic test script corresponding to each data cluster through the pyspark logic, and monitor the monitoring data in each data cluster through the automatic test script to obtain the monitoring result, comprising: Obtain the pyspark logic written in the preset editing platform, execute the automatic test script corresponding to each data cluster through the pyspark logic, and monitor the monitoring data in each data cluster through the automatic test script to obtain the monitoring result, comprising: Obtain the pyspark logic written in the preset editing platform, execute the automatic test script corresponding to each data cluster through the pyspark logic, and monitor the monitoring data in each data cluster through the automatic test script to obtain the monitoring result, comprising:

2. The pyspark based automated testing method as claimed in claim 1, wherein, Obtain the pyspark logic written in the preset editing platform, execute the automatic test script corresponding to each data cluster through the pyspark logic, and monitor the monitoring data in each data cluster through the automatic test script to obtain the monitoring result, comprising: Display the monitoring result to the preset receiving platform through the task flow display platform. The method comprises: When it is determined that there is a dependent task having a dependency relationship with the missing monitoring task, obtain the dependent task having a dependency relationship with the missing monitoring task, and then run the missing monitoring task after the dependent task is run.

3. The automatic test method based on pyspark according to claim 2, wherein The method comprises: Obtain the task information table, task log corresponding to the missing monitoring task and the dependent task, and task running state determined from the monitoring result through the task scheduling platform; 4. The pyspark based automated testing method as claimed in claim 1, wherein, Splice the missing monitoring task, task running state, task information table and task running state into combined JSON through JAVA microservice; Display the combined JSON to the preset receiving platform in the form of a preset box line chart through the task flow display platform. After obtaining the pyspark logic written in the preset editing platform, the method further comprises: Obtain the changed pyspark logic corresponding to the new missing monitoring task; Obtain the new automatic test script corresponding to the new missing monitoring task according to the new parent-child monitoring rule obtained according to the changed pyspark logic; Monitor the monitoring data in each data cluster in turn through the new automatic test script according to the execution order to obtain multiple new monitoring results.

5. The pyspark based automated testing method as claimed in claim 1, wherein, The monitoring data in each data cluster is sequentially monitored by the executed automation test script to obtain a plurality of monitoring results, including: In the preset editing platform, different contexts are obtained according to different data clusters, and distributed data sets are created based on the monitoring data in different data clusters through the contexts; The monitoring data in each distributed data set is sequentially monitored by the automation test script to obtain a plurality of monitoring results.

6. The pyspark based automated testing method as claimed in claim 5, wherein, After the distributed data sets are created based on the monitoring data in the data clusters through the contexts, the method further includes: The selected preset data conversion mode is concatenated to obtain a total preset data conversion mode; The obtained distributed data sets are preset data converted according to the total preset data conversion mode to obtain converted new distributed data sets; The monitoring data in each new distributed data set is monitored again by the automation test script.

7. A pyspark based automated testing device, characterized in that, The device includes: The acquisition module is configured to acquire the missing monitoring tasks in the task scheduling platform at a regular time; The monitoring module is configured to acquire the pyspark logic written in the preset editing platform when the missing monitoring tasks are executed, execute the automation test scripts corresponding to the plurality of data clusters through the pyspark logic, and monitor the monitoring data in each data cluster through the automation test scripts to obtain monitoring results; The monitoring module is further configured to: Acquire the pyspark logic containing the automation test scripts corresponding to the plurality of data clusters written in the preset editing platform, and use the pyspark logic as a monitoring rule; the pyspark logic contains the automation test scripts for testing the plurality of data clusters; Parse the monitoring rule to obtain parent-child monitoring rules; all automation test scripts corresponding to the plurality of data clusters contained in the parent-child monitoring rules are arranged in a sequence of execution; Execute the automation test scripts corresponding to the plurality of data clusters according to the sequence of execution in the parent-child monitoring rules, and sequentially monitor the monitoring data in each data cluster through the executed automation test scripts to obtain a plurality of monitoring results; The display module is configured to display the monitoring results to the preset receiving platform through the task flow display platform.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the pyspark-based automation test method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. The computer program is executed by the processor to implement the pyspark-based automation test method according to any one of claims 1 to 6.

Images