Spark task testing methods, devices, equipment, and media

Abstract

The present invention belongs to the field of artificial intelligence and provides a method, apparatus, device, and medium for testing a Spark task. The method includes: obtaining a target Spark task including multiple functional objects, wherein the functional objects are preset with verification parameters; determining a functional object to be authenticated, and determining a calling functional object of the functional object to be authenticated based on the calling statement; obtaining target authentication information from a preset authentication information table based on the verification parameters of the calling functional object; determining a call authentication result of the functional object to be authenticated based on the target authentication information, and performing a functional test to obtain a target test result after all functional objects to be authenticated have passed authentication. According to the technical solution of this embodiment, the calling authority of the functional object can be authenticated before testing the target Spark task based on the authentication information recorded in the authentication information table, and the test can be performed after the authentication passes, effectively avoiding test failures caused by call errors and improving test efficiency.

Description

Technical Field

[0001] The present invention belongs to the field of big data, and in particular relates to a Spark task testing method, device, equipment, and medium. Background Art

[0002] Spark is a computing engine for large-scale data processing. In big data business systems using Spark, efficient big data processing is achieved by distributing Spark tasks that compute large datasets across multiple devices. Due to the large program size of Spark tasks, calls between functional objects such as assemblies or functions are common. Errors in calling functional objects can lead to data conflicts or overwriting of result data, impacting the normal operation of Spark tasks. To prevent errors in calling functional objects, the current method involves testers backtracking erroneous data during Spark task testing and reporting the root cause to developers for maintenance, which is very inefficient. Summary of the Invention

[0003] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0004] The embodiments of the present invention provide a Spark task testing method, apparatus, device, and medium, which can authenticate the call of a function object before testing, automatically discover erroneous calls of the function object, and improve the testing efficiency of the Spark task.

[0005] In a first aspect, an embodiment of the present invention provides a Spark task testing method, including:

[0006] Obtain a target Spark task to be tested, wherein the target Spark task includes multiple function objects, and each function object is preset with verification parameters;

[0007] Determine the function object having the call statement as the function object to be authenticated, and determine the calling function object of the function object to be authenticated according to the call statement;

[0008] Obtain target authentication information from a preset authentication information table according to the verification parameter of the calling function object;

[0009] Determine the call authentication result of the function object to be authenticated according to the target authentication information;

[0010] When the call authentication result indicates that the authentication of the function object to be authenticated is passed, the function object to be authenticated is set as a testable function object;

[0011] When all the functional objects to be authenticated are determined to be the testable functional objects, the functional test of the target Spark task is executed to obtain the target test result of the target Spark task.

[0012] In some embodiments, the target authentication information includes a Spark identifier and a verification identifier, and determining the call authentication result of the function object to be authenticated according to the target authentication information includes:

[0013] Determine the task identifier of the target Spark task;

[0014] When the task identifier matches the Spark identifier, and the verification identifier indicates that the verification is passed, the call authentication result of the function object to be authenticated is determined to be authentication passed.

[0015] In some embodiments, the target authentication information further includes target user information. Before obtaining the target Spark task to be tested, the method further includes:

[0016] Acquire update information for the functional object, the update information carrying operation user information and object modification information;

[0017] When the operating user information matches the target user information, and the corresponding verification identifier indicates that the verification is passed, the functional object is updated according to the object modification information.

[0018] In some embodiments, the target Spark task includes a plurality of task files, each of the task files records an assembly corresponding to at least one of the functional objects, and executing the functional test of the target Spark task to obtain a target test result of the target Spark task includes:

[0019] Saving all of the task files to a target storage disk;

[0020] Parsing and functionally testing the task file to obtain functional test data and functional test results;

[0021] The functional test data and the functional test results corresponding to each task file are saved to the target storage disk.

[0022] In some embodiments, saving all of the task files to a target storage disk includes:

[0023] determining a file format of each of the task files;

[0024] When the file format of the task file belongs to a preset format, saving the task file to the target storage disk;

[0025] or,

[0026] When the file format of the task file does not belong to the preset format, converting the file format of the task file into the preset format and saving the converted file to the target storage disk;

[0027] The preset format is a file format that the Spark program supports parsing.

[0028] In some embodiments, performing a functional test of the target Spark task includes:

[0029] Obtain a target test script corresponding to the target Spark task, and determine the resource requirements of the target test script;

[0030] Input the target test script into a script execution queue, and determine the number of available resources in the script execution queue;

[0031] When the number of available resources meets the resource requirement, allocating target test resources to the target test script according to the resource requirement, or when the number of available resources does not meet the resource requirement, allocating the target test resources according to the number of available resources;

[0032] The target test script is run according to the target test resource.

[0033] In some embodiments, after allocating target test resources to the target test script according to the resource requirement when the available resources meet the resource requirement, the method further includes:

[0034] Determine the number of test resources and production resources of the script execution queue, wherein the number of test resources is the number of resources in the script execution queue used for running functional tests, and the number of production resources is the number of resources in the script execution queue used for system operation;

[0035] Determine the ratio of the number of test resources to the number of production resources as a target ratio;

[0036] The target proportion is reduced according to a preset ratio so that the script execution queue increases the number of production resources.

[0037] In a second aspect, an embodiment of the present invention provides a Spark task testing device, including:

[0038] A task acquisition unit, configured to acquire a target Spark task to be tested, wherein the target Spark task includes a plurality of function objects, and each function object is preset with verification parameters;

[0039] an object determining unit, configured to determine a function object having a call statement recorded therein as a function object to be authenticated, and determine a calling function object of the function object to be authenticated according to the call statement;

[0040] An authentication information acquisition unit, configured to acquire target authentication information from a preset authentication information table according to the verification parameter of the calling function object;

[0041] A calling authentication unit, configured to determine a calling authentication result of the function object to be authenticated according to the target authentication information;

[0042] An object authentication unit, configured to, when the call authentication result indicates that the authentication of the functional object to be authenticated is passed, set the functional object to be authenticated as a testable functional object;

[0043] The testing unit is configured to, when all the function objects to be authenticated are determined to be the testable function objects, execute the function test of the target Spark task and obtain the target test result of the target Spark task.

[0044] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the Spark task testing method as described in the first aspect is implemented.

[0045] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program is used to execute the Spark task testing method as described in the first aspect.

[0046] The embodiment of the present invention includes: obtaining a target Spark task to be tested, the target Spark task including multiple function objects, the function objects having preset verification parameters; determining a function object having a call statement as a function object to be authenticated, and determining the calling function object of the function object to be authenticated according to the call statement; obtaining target authentication information from a preset authentication information table according to the verification parameters of the calling function object; determining a call authentication result of the function object to be authenticated according to the target authentication information; when the call authentication result indicates that the authentication of the function object to be authenticated has passed, the function object to be authenticated is a testable function object; when all the function objects to be authenticated are determined to be testable function objects, executing a functional test of the target Spark task to obtain a target test result of the target Spark task. According to the technical solution of this embodiment, the calling authority of the function object can be authenticated before testing the target Spark task according to the target authentication information recorded in the authentication information table, and the test can be executed after the authentication passes, effectively avoiding test failures caused by call errors and improving test efficiency.

[0047] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purposes and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings are used to provide a further understanding of the technical solution of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the technical solution of the present invention and do not constitute a limitation to the technical solution of the present invention.

[0049] Figure 1 This is a flowchart of a Spark task testing method provided by one embodiment of the present invention;

[0050] Figure 2 is a flowchart of call authentication provided by another embodiment of the present invention;

[0051] Figure 3 is a flowchart of modifying a function object provided by another embodiment of the present invention;

[0052] Figure 4 is a flowchart of saving test data provided by another embodiment of the present invention;

[0053] Figure 5 is a flowchart of file format conversion provided by another embodiment of the present invention;

[0054] Figure 6 is a flowchart of allocating test resources provided by another embodiment of the present invention;

[0055] Figure 7 is a flowchart of adjusting system resources provided by another embodiment of the present invention;

[0056] Figure 8 is a structural diagram of a Spark task testing device provided by another embodiment of the present invention;

[0057] Figure 9 is a device diagram of an electronic device provided in another embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0059] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "objective," and the like in the specification, claims, or accompanying drawings are used to distinguish similar objects and are not necessarily intended to describe a specific sequence or precedence.

[0060] The present invention provides a method, apparatus, device, and medium for testing a Spark task. The method includes: obtaining a target Spark task to be tested, wherein the target Spark task includes multiple function objects, and the function objects are preset with verification parameters; determining a function object with a call statement as a function object to be authenticated, and determining the calling function object of the function object to be authenticated according to the call statement; obtaining target authentication information from a preset authentication information table according to the verification parameters of the calling function object; determining a call authentication result of the function object to be authenticated according to the target authentication information; when the call authentication result indicates that the authentication of the function object to be authenticated is passed, the function object to be authenticated is determined to be a testable function object; when all the function objects to be authenticated are determined to be testable function objects, performing a functional test of the target Spark task to obtain a target test result of the target Spark task. According to the technical solution of this embodiment, the calling authority of the function object can be authenticated before testing the target Spark task according to the target authentication information recorded in the authentication information table, and the test is performed after the authentication is passed, effectively avoiding test failures caused by call errors and improving test efficiency.

[0061] The embodiments of the present application can compile, acquire, and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) refers to the theories, methods, technologies, and application devices that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.

[0062] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive devices, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0063] Spark in the embodiment of the present application is a fast and general computing engine designed for large-scale data processing. In the process of using Spark to perform big data processing, the intermediate results of the data calculation are generated by MapReduce and stored in the RDD of each worker node of Spark. MapReduce is a computing model for parallel processing of big data, and is usually used for parallel operations of large-scale data sets. RDD is called Resilient Distributed Dataset, which is the most basic data abstraction in Spark. It represents an immutable, partitionable set whose elements can be calculated in parallel. Specifically, the data to be monitored is selected by the user (developer). After the system obtains the data selection instruction input by the user, it extracts the data to be monitored pointed to by the data selection instruction from the corresponding RDD. Spark has the advantages of Hadoop MapReduce; but unlike MapReduce, the intermediate output results of the Job can be stored in memory, so there is no need to read and write HDFS. Therefore, Spark is better suitable for MapReduce algorithms that require iteration, such as data mining and machine learning. Spark is an open source cluster computing environment similar to Hadoop, but with some differences that make it more advantageous for certain workloads. In other words, Spark enables in-memory distributed datasets, which in addition to providing interactive queries, can also optimize iterative workloads.

[0064] It should be noted that the data of the embodiments of the present invention can be stored in a server. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0065] like Figure 1 As shown, Figure 1 The flowchart of a Spark task testing method provided by one embodiment of the present invention includes but is not limited to the following steps:

[0066] Step S110: obtaining a target Spark task to be tested, where the target Spark task includes multiple functional objects, each of which has preset verification parameters.

[0067] Step S120, determining the function object having the call statement as the function object to be authenticated, and determining the calling function object of the function object to be authenticated according to the call statement;

[0068] Step S130, obtaining target authentication information from a preset authentication information table according to the verification parameters of the calling function object;

[0069] Step S140, determining the call authentication result of the function object to be authenticated according to the target authentication information;

[0070] Step S150: when the authentication result indicates that the authentication of the function object to be authenticated is successful, the function object to be authenticated is set as a testable function object;

[0071] Step S160: When all the functional objects to be authenticated are determined to be testable functional objects, the functional test of the target Spark task is executed to obtain the target test result of the target Spark task.

[0072] It should be noted that the assembly of Spark tasks can be saved in the database to facilitate joint development by multiple developers and improve development efficiency. At the same time, the database can simultaneously save multiple different Spark tasks. When testing is required, the technical solution of this embodiment can be executed independently for each Spark task once. There is no excessive limit on the number of Spark tasks here.

[0073] It should be noted that the functional object of a Spark task can be an assembly or a function, as long as it can realize complete functions. It is understandable that, in order to facilitate authentication, a verification parameter can be set in each assembly or function. The verification parameter does not play an actual operating function in the assembly or function, but is used to match the target authentication information from the authentication information table. The verification parameter can be the assembly name, the Spark task number, etc. The verification parameter can be uniquely corresponding to the functional object, or, for actual needs, the same verification parameter can be configured for multiple functional objects with the same call permission and applicable scope. This embodiment does not impose too many restrictions on the correspondence between the verification parameter and the functional object. It is sufficient to be able to obtain the target authentication information of the functional object from the authentication information table through the verification parameter.

[0074] It should be noted that the authentication information table can be a global index file, such as the global Map file commonly used in Spark programs. The Map file records the target authentication information of all functional objects stored in the same database. During the development of the Spark program, when it is detected that a new assembly or function is generated, its verification parameters can be saved in the Map file, and the authentication information can be configured for the verification parameters in the Map file. For example, if a functional object developed by a developer cannot be called by other functional objects, the target authentication information corresponding to the verification parameters corresponding to the functional object can be set to non-callable in the Map file. For example, a common Flag identifier can be set, with Yes indicating that the functional object can be called, and No indicating that the functional object cannot be called. The specific form of the target authentication information is not limited in this embodiment, and it is sufficient to be able to represent the authority of the functional object to be called.

[0075] It should be noted that Spark tasks typically have many function objects, and not all function objects have call relationships. For function objects that do not have call relationships, since the issue of whether the call is wrong does not arise, the authentication operation described in this embodiment can be omitted, thereby improving the efficiency of function object call authentication. Based on this, after obtaining the target Spark task, the function objects that record the call statement can be determined as the function objects to be authenticated. The call statement usually records the information of the function object to be called. Therefore, the calling function object of the function object to be authenticated can be determined based on the call statement, and the call permission of the calling function object can be authenticated to determine the call authentication result. For example, if function object A calls function object B, then function object A is the function object to be authenticated, and function object B is the calling authentication object. The target authentication information is obtained from the authentication information table according to the verification parameters of function object B. When the target authentication information records the calling permission for function object A, for example, the Flag mark for function object A is recorded as Yes, then it can be determined that the calling authentication result of function object A is authentication passed, function object B can be called by function object A, and there will be no calling error for function object B when performing functional testing on function object A; if the Flag mark for function object A is recorded as No, then it can be determined that the calling authentication result of function object A is authentication failure, and a calling error will occur if function object A is tested. Since the test has not started, the developer can be informed through error prompts and other means to perform timely maintenance, thereby automatically discovering calling errors and improving the automation level of self-inspection of the target Spark task assembly.

[0076] It should be noted that after the call authentication is completed for each functional object and the call authentication results are all authenticated, it can be determined that no errors caused by call permissions will occur during the test process of the target Spark task, which can effectively improve the efficiency of error troubleshooting during the test process, thereby improving test efficiency. The specific function test process is a technology well known to those skilled in the art. This embodiment does not elaborate on the specific process of the function test. It is sufficient to obtain the target test result of the target Spark task and that the target test result does not have errors caused by call permissions.

[0077] In addition, in one embodiment, the target authentication information includes a Spark identifier and a verification identifier. Figure 2 , Figure 1 Step S150 shown also includes but is not limited to the following steps:

[0078] Step S210, determining the task identifier of the target Spark task;

[0079] Step S220: When the task identifier matches the Spark identifier and the verification identifier indicates that the verification is passed, the call authentication result of the function object to be authenticated is determined to be authentication passed.

[0080] It should be noted that the target authentication information may include a Spark identifier and a check identifier. The Spark identifier can be the name or number of a Spark task, used to distinguish different Spark tasks. For common function objects, the same name may be used in different Spark tasks. To avoid errors in calling data results, such as Task A calling Task B's data, this embodiment sets the Spark identifier in the target authentication information to prevent data calls between different Spark tasks. The check identifier can be the Flag identifier described in the above embodiment, used to indicate whether the function object can be called. By setting the check identifier and the Spark identifier, it is possible to effectively avoid incorrect calls to function objects and prevent erroneous data in function objects from being overwritten, effectively improving the efficiency of testing and program development and reducing development time and costs.

[0081] It is worth noting that the verification flag is not a fixed flag, but can be adjusted according to actual needs. For example, in the subsequent expansion development process, due to functional adjustments, the calling relationship between functional objects changes, the verification flag can also be adjusted in the authentication identification table to achieve changes in calling permissions, which can effectively improve the calling flexibility of Spark tasks.

[0082] In addition, in one embodiment, the target authentication information also includes target user information, see Figure 3 , in execution Figure 1Before step S110, the following steps are also included but not limited to:

[0083] Step S310: Acquire update information for the functional object, where the update information carries operating user information and object modification information;

[0084] Step S320: When the operating user information matches the target user information and the corresponding verification identifier indicates that the verification is passed, the functional object is updated according to the object modification information.

[0085] It should be noted that the authentication of functional objects can be performed not only in the testing phase, but also in the development process. Since Spark tasks are usually developed by multiple people, it is very likely that the assembly of the functional object will be overwritten due to the developer's misoperation, resulting in functional errors of the functional object. In order to avoid this situation, user information can be added to the target authentication information. After the developer writes the update information, the developer's logged-in operation user information is matched to determine whether the developer has the modification permission, thereby effectively improving the security of the functional object code.

[0086] It should be noted that, combined with Figure 2 In the description of the embodiment, the target authentication information may include target user information, Spark identifier and verification identifier. Therefore, in the authentication information table, the form of each target authentication information can refer to the table below. Of course, the content of the target authentication information can also be increased or decreased according to actual needs, and no excessive restrictions are made here.

[0087] Target user information Spark logo Flag User A Spark Task A Yes

[0088] Table 1: Example of authentication information table

[0089] In addition, in one embodiment, the target Spark task includes multiple task files, each of which records a program set corresponding to at least one functional object. Figure 4 , Figure 1 Step S160 shown also includes but is not limited to the following steps:

[0090] Step S410, saving all task files to the target storage disk;

[0091] Step S420: parsing and functional testing the task file to obtain functional test data and functional test results;

[0092] Step S430: Save the function test data and function test results corresponding to each task file to the target storage disk.

[0093] It should be noted that during the operation and testing of the target Spark task, most of the data will be stored in the memory, and the memory is an erasable storage medium, which can easily delete the corresponding data after the execution is completed. In this embodiment, the task files, functional test data and functional test results are stored in the target storage disk, which can facilitate the later viewing, analysis and verification of the intermediate data of the program. This will greatly reduce the time and effort spent on tracing abnormal data for big data scenarios with high precision requirements and complex scenarios, and improve the efficiency of development and testing.

[0094] In addition, in one embodiment, referring to Figure 5 , Figure 4 Step S410 shown also includes but is not limited to the following steps:

[0095] Step S510, determining the file format of each task file;

[0096] Step S520: When the file format of the task file is a preset format, the task file is saved to a target storage disk;

[0097] or,

[0098] Step S530 , when the file format of the task file does not belong to the preset format, convert the file format of the task file into the preset format and save it to the target storage disk;

[0099] The preset format is the file format that the Spark program supports parsing.

[0100] It should be noted that the file formats that Spark programs can support are limited, such as the common text, csv, json, and parquet formats. However, during the development process, it is difficult to ensure that all task files conform to the above formats. For example, SQL is usually stored in xml. The data hierarchy of task files in these formats is relatively complex, and errors are likely to occur during data reading, data parsing, and data structuring, resulting in errors in testing and operation. Based on this, after obtaining the target Spark task, the file format can be verified for each task file, and the file format of task files that do not conform to the preset format can be converted so that the converted task files can be directly parsed and recognized by the Spark program, and can be saved to the target storage disk, improving the convenience of use.

[0101] It should be noted that by converting the file format, the dataset of the target Spark task is expanded, which improves the convenience for developers to write programs. The conversion of file format is a technology well known to those skilled in the art and will not be described in detail here.

[0102] In addition, in one embodiment, referring to Figure 6 , Figure 1 Step S160 shown also includes but is not limited to the following steps:

[0103] Step S610: Obtain a target test script corresponding to a target Spark task and determine the resource requirements of the target test script;

[0104] Step S620: input the target test script into the script execution queue and determine the number of available resources in the script execution queue;

[0105] Step S630 , when the number of available resources meets the resource requirement, allocating target test resources to the target test script according to the resource requirement, or when the number of available resources does not meet the resource requirement, allocating target test resources according to the number of available resources;

[0106] Step S640: Run the target test script according to the target test resource.

[0107] It should be noted that the test of the target Spark task is performed through a script. After the target test script is input into the script running queue, automatic testing is achieved by executing the script in the queue. Therefore, the number of available resources in the script running queue can be determined first, and the test resources can be allocated according to the available resources to ensure that the test can proceed smoothly.

[0108] It is worth noting that after determining the target test resources, the target test script can be automatically modified and uploaded to the test cluster for script replacement, thereby ensuring the accuracy of resource allocation.

[0109] It is worth noting that when there are multiple Spark tasks that need to be tested, you can set the number of resources for each task separately, or you can allocate test resources uniformly and adjust according to actual needs.

[0110] In addition, in one embodiment, referring to Figure 7 After executing Figure 6 After step S630, the following steps are also included but not limited to:

[0111] Step S710, determining the number of test resources and production resources in the script execution queue, where the test resource number is the number of resources in the script execution queue used for function testing, and the production resource number is the number of resources in the script execution queue used for system operation;

[0112] Step S720: determining the ratio of the number of test resources to the number of production resources as the target ratio;

[0113] Step S730: Reduce the target proportion according to a preset ratio so that the script execution queue increases the number of production resources.

[0114] It should be noted that the task resources required for big data tasks are often configured according to the production environment and are often not specifically configured for the test environment. The target Spark task to be tested needs to share the resources of a queue with the running Spark task. Usually, resources are allocated to the running Spark task first, that is, production resources are allocated first. In this case, the number of resources that can be allocated to testing is limited. If the target ratio of production resources to test resources is set in advance, when the number of production resources is small, it will lead to waste of available resources. It is very likely that a test task will occupy most or even all the test resources of a queue for a long time, causing the remaining test tasks to remain in a waiting state for a long time. However, in a test environment, there are quite a number of small tasks that do not actually require too many resources and will not run for too long. For example, a large Spark task may block many Sqoop derivative tasks, or a task is only run by developers or testers for testing, resulting in a shortage of resources in the entire test cluster and blocking the submission of tasks in the entire cluster. Therefore, after determining the target ratio, this embodiment adjusts the target ratio according to the available resources. When there are sufficient remaining resources in the queue, the number of production resources is increased to reduce the target ratio proportionally, allowing more resources to be allocated to normally running Spark tasks, thereby realizing dynamic adjustment of system resources and making resource allocation more reasonable.

[0115] In addition, refer to Figure 8 The embodiment of the present invention provides a Spark task testing device. The Spark task testing device 800 includes but is not limited to the following devices:

[0116] The task acquisition unit 810 is used to acquire a target Spark task to be tested, where the target Spark task includes multiple function objects, and each function object has preset verification parameters.

[0117] An object determining unit 820 is configured to determine a function object having a call statement as a function object to be authenticated, and determine a calling function object of the function object to be authenticated according to the call statement;

[0118] The authentication information acquisition unit 830 is used to obtain target authentication information from a preset authentication information table according to the verification parameter of the calling function object;

[0119] A call authentication unit 840 is used to determine a call authentication result of the function object to be authenticated according to the target authentication information;

[0120] The object authentication unit 850 is configured to, when the authentication result indicates that the authentication of the function object to be authenticated is successful, change the function object to be authenticated into a testable function object;

[0121] The testing unit 860 is configured to execute a functional test of a target Spark task when all the function objects to be authenticated are determined to be testable function objects, and obtain a target test result of the target Spark task.

[0122] In addition, refer to Figure 9 An embodiment of the present invention further provides an electronic device. The electronic device 900 includes a memory 910, a processor 920, and a computer program stored in the memory 910 and executable on the processor 920.

[0123] The processor 920 and the memory 910 may be connected via a bus or other means.

[0124] The non-transient software program and instructions required to implement the Spark task test method of the above embodiment are stored in the memory 910. When executed by the processor 920, the Spark task test method of the above embodiment is executed, for example, the above-described Figure 1 Steps S110 to S160 of the method, Figure 2 Steps S210 to S220 of the method, Figure 3 Steps S310 to S320 of the method, Figure 4 Steps S410 to S430 of the method, Figure 5 Steps S510 to S530 of the method, Figure 6 Steps S610 to S640 of the method, Figure 7 Method steps S710 to S730.

[0125] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0126] In addition, an embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program. The computer program is executed by a processor or controller, for example, by a processor in the above electronic device embodiment, which enables the above processor to execute the Spark task test method in the above embodiment, for example, to execute the above-described Figure 1 Steps S110 to S160 of the method, Figure 2 Steps S210 to S220 of the method, Figure 3 Steps S310 to S320 of the method, Figure 4 Steps S410 to S430 of the method, Figure 5Steps S510 to S530 of the method, Figure 6 Steps S610 to S640 of the method, Figure 7 Method steps S710 to S730 in the method. It will be appreciated by those skilled in the art that all or some of the steps and devices in the method disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable storage medium, which can include computer storage media (or non-transitory storage media) and communication storage media (or temporary storage media). As known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable storage media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other storage medium that can be used to store desired information and can be accessed by a computer. Furthermore, as is well known to those skilled in the art, communication storage media generally contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery storage media.

[0127] The present embodiment can be used in many general or special computer device environments or configurations. For example: personal computers, server computers, handheld electronic devices or portable electronic devices, tablet electronic devices, multi-processor devices, microprocessor-based devices, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above devices or electronic devices, etc. The present application can be described in the general context of computer programs executed by computers, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing electronic devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage electronic devices.

[0128] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. In some cases, the names of these units do not constitute limitations on the units themselves.

[0129] It should be noted that, although several modules or units of the electronic device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the application, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0130] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing electronic device (which can be a personal computer, a server, a touch terminal, or a network electronic device, etc.) to execute the method according to the embodiments of the present application.

[0131] The electronic device of this embodiment may include components such as a radio frequency (RF) circuit, a memory, an input unit, a display unit, a sensor, an audio circuit, a wireless fidelity (WiFi) module, a processor, and a power supply. The RF circuit can be used to receive and send signals during information transmission or calls. In particular, it receives downlink information from the base station and sends it to the processor for processing; in addition, it sends uplink data to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF circuit can also communicate with the network and other devices via wireless communication. The above-mentioned wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc. The memory can be used to store software programs and modules, and the processor executes various functional applications and data processing of the electronic device by running the software programs and modules stored in the memory. The memory may mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created according to the use of the electronic device (such as audio data, a phone book, etc.). In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. The input unit can be used to receive input digital or character information, and to generate key signal input related to the settings and function control of the electronic device. Specifically, the input unit may include a touch panel and other input devices. The touch panel, also known as a touch screen, can collect touch operations on or near it (such as operations using any suitable object or accessory such as a finger, a stylus, etc. on or near the touch panel) and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel may include two parts: a touch detection device and a touch controller.Among them, the touch detection device detects the touch direction and detects the signal caused by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the touch point coordinates, and then sends it to the processor, and can receive and execute commands sent by the processor. In addition, the touch panel can be implemented using various types such as resistive, capacitive, infrared and surface acoustic wave. In addition to the touch panel, the input unit can also include other input devices. Specifically, other input devices can include but are not limited to one or more of a physical keyboard, function keys (such as volume control buttons, switch buttons, etc.), a trackball, a mouse, a joystick, etc. The display unit can be used to display input information or provided information and various menus of the electronic device. The display unit may include a display panel, and optionally, the display panel can be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. Furthermore, the touch panel can cover the display panel. When the touch panel detects a touch operation on or near it, it is transmitted to the processor to determine the type of touch event, and the processor then provides corresponding visual output on the display panel based on the type of touch event. The touch panel and the display panel are two independent components to realize the input and output functions of the electronic device, but in some embodiments, the touch panel and the display panel can be integrated to realize the input and output functions of the electronic device. The electronic device may also include at least one sensor, such as a light sensor, a motion sensor and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor can adjust the brightness of the display panel according to the brightness of the ambient light, and the proximity sensor can turn off the display panel and / or backlight when the electronic device is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that recognize the posture of the electronic device (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors that can be configured in electronic devices, they will not be repeated here. The audio circuit, speaker, and microphone can provide an audio interface. The audio circuit can convert the received audio data into an electrical signal and transmit it to the speaker, which converts it into a sound signal for output. On the other hand, the microphone converts the collected sound signal into an electrical signal, which is received by the audio circuit and converted into audio data. The audio data is then output to the processor for processing and then sent to another electronic device through the RF circuit, or the audio data is output to the memory for further processing.

[0132] Those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art that are not disclosed herein.

[0133] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

[0134] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above implementation. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A Spark task testing method, characterized in that: include: Obtain a target Spark task to be tested, wherein the target Spark task includes multiple function objects, and each function object is preset with verification parameters; Determine the function object having the call statement as the function object to be authenticated, and determine the calling function object of the function object to be authenticated according to the call statement; Obtain target authentication information from a preset authentication information table according to the verification parameter of the calling function object; Determine the call authentication result of the function object to be authenticated according to the target authentication information; When the call authentication result indicates that the authentication of the function object to be authenticated is passed, the function object to be authenticated is set as a testable function object; When all the functional objects to be authenticated are determined to be the testable functional objects, executing the functional test of the target Spark task to obtain the target test result of the target Spark task; The target authentication information includes a Spark identifier and a verification identifier, and determining the call authentication result of the function object to be authenticated according to the target authentication information includes: Determine the task identifier of the target Spark task; When the task identifier matches the Spark identifier, and the verification identifier indicates that the verification is passed, the call authentication result of the function object to be authenticated is determined to be authentication passed.

2. A Spark task testing method according to claim 1, characterized in that: The target authentication information also includes target user information. Before obtaining the target Spark task to be tested, the method further includes: Acquire update information for the functional object, the update information carrying operation user information and object modification information; When the operating user information matches the target user information, and the corresponding verification identifier indicates that the verification is passed, the functional object is updated according to the object modification information.

3. A Spark task testing method according to claim 1, characterized in that: The target Spark task includes a plurality of task files, each of which records a program set corresponding to at least one functional object. The executing the functional test of the target Spark task to obtain the target test result of the target Spark task includes: Saving all of the task files to a target storage disk; Parsing and functionally testing the task file to obtain functional test data and functional test results; The functional test data and the functional test results corresponding to each task file are saved to the target storage disk.

4. A Spark task testing method according to claim 3, characterized in that: Saving all the task files to the target storage disk includes: determining a file format of each of the task files; When the file format of the task file belongs to a preset format, saving the task file to the target storage disk; or, When the file format of the task file does not belong to the preset format, converting the file format of the task file into the preset format and saving the converted file to the target storage disk; The preset format is a file format that the Spark program supports parsing.

5. A Spark task testing method according to claim 1, characterized in that: The functional test of executing the target Spark task includes: Obtain a target test script corresponding to the target Spark task, and determine the resource requirements of the target test script; Input the target test script into a script execution queue, and determine the number of available resources in the script execution queue; When the number of available resources meets the resource requirement, allocating target test resources to the target test script according to the resource requirement, or when the number of available resources does not meet the resource requirement, allocating the target test resources according to the number of available resources; The target test script is run according to the target test resource.

6. A Spark task testing method according to claim 5, characterized in that: When the number of available resources meets the resource requirement, after allocating target test resources to the target test script according to the resource requirement, the method further includes: Determine the number of test resources and production resources of the script execution queue, wherein the number of test resources is the number of resources in the script execution queue used for running functional tests, and the number of production resources is the number of resources in the script execution queue used for system operation; Determine the ratio of the number of test resources to the number of production resources as a target ratio; The target proportion is reduced according to a preset ratio so that the script execution queue increases the number of production resources.

7. A Spark task testing device, characterized in that: include: A task acquisition unit, configured to acquire a target Spark task to be tested, wherein the target Spark task includes a plurality of function objects, and each function object is preset with verification parameters; an object determining unit, configured to determine a function object having a call statement recorded therein as a function object to be authenticated, and determine a calling function object of the function object to be authenticated according to the call statement; An authentication information acquisition unit, configured to acquire target authentication information from a preset authentication information table according to the verification parameter of the calling function object; A calling authentication unit, configured to determine a calling authentication result of the function object to be authenticated according to the target authentication information; An object authentication unit, configured to, when the call authentication result indicates that the authentication of the functional object to be authenticated is passed, set the functional object to be authenticated as a testable functional object; A testing unit, configured to, when all the functional objects to be authenticated are determined to be the testable functional objects, execute a functional test of the target Spark task and obtain a target test result of the target Spark task; The target authentication information includes a Spark identifier and a verification identifier, and determining the call authentication result of the function object to be authenticated according to the target authentication information includes: Determine the task identifier of the target Spark task; When the task identifier matches the Spark identifier, and the verification identifier indicates that the verification is passed, the call authentication result of the function object to be authenticated is determined to be authentication passed.

8. An electronic device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the Spark task testing method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium storing a computer program, characterized in that: The computer program is used to execute the Spark task testing method according to any one of claims 1 to 6.

Images