A method and device for processing small data volume jobs based on big data engine

Abstract

The present application proposes a method and device for processing small data volume jobs based on a big data engine, which method includes the following steps: S1, obtaining a data file to be processed, and determining whether the data file to be processed is a small data volume job. If so, executing step S2; S2, determining whether the resource pool has an idle runtime environment. If so, the business code is uniformly encapsulated, and the runtime environment interprets and executes the encapsulated code. If not, executing step S3; S3, determining whether the resource pool has reached the resource capacity upper limit. If so, the data file is processed and enters the job queue to wait. If not, a runtime environment is created and put into the resource pool, and the business code is uniformly encapsulated, and the runtime environment interprets and executes the encapsulated code. S4, detecting whether the idle time of the runtime environment in the resource pool exceeds a preset threshold. If so, the runtime resources are released.

Description

Technical Field

[0001] The present application relates to the technical field of big data task processing, and specifically to a method and device for processing small data volume jobs based on a big data engine. Background Art

[0002] Big data processing includes data collection, processing, analysis, and presentation. This includes various data processing engines (Spark and Flink). Users customize business code based on their scenarios and process and store data using these engines.

[0003] Existing big data engines (Spark / Flink) offer two task execution modes: interactive execution (spark-shell / flink-shell) and submission execution (spark-submit / flink-submit). Taking Spark as an example, after starting interactive execution, a Spark execution environment (spark-session) is created. Users directly use the built-in context to write task code in the Spark-shell window for data processing. However, interactive execution is inconvenient for complex tasks. With the Spark-submit submission mode, users write and package their code for execution. Each submission requires requesting resources from the Spark cluster and creating a Spark-session. Resources are then reclaimed after the task is completed.

[0004] The existing big data task submission and processing method (spark-submit / flink-submit) requires a certain amount of resources from the processing cluster for each task submission, and the cluster needs to reclaim resources after the task is completed. This method has the following problems:

[0005] 1) Submitting a large number of small tasks consumes a significant amount of cluster resources. For example, in a small Flink cluster (128GB of memory and 32 cores), the minimum execution unit for a task requires 1GB of executor memory and 4 executor cores. Consecutive submission of 50 small tasks by different users and businesses consumes a significant amount of cluster resources, forcing other tasks to wait for execution due to inability to apply for resources.

[0006] 2) For small tasks, the amount of data is relatively small, the runtime environment creation time is much longer than the task running time, and the resources required to create the runtime environment are also much greater than the resources required for data processing itself, resulting in a waste of cluster resources.

[0007] In view of this, the present invention proposes a method and device for processing small data volume jobs based on a big data engine. By creating a runtime environment resource pool and reusing the created runtime environment to perform data processing, the problem of frequent submission of small batch data jobs in daily business is solved. Summary of the Invention

[0008] In order to solve the problems of users frequently submitting small data volume task processing, the need to create and destroy the runtime environment consumes a large amount of cluster resources, and the time to create the runtime environment is longer than the data processing time, and the response time is long. This application provides a small data volume job processing method and device based on a big data engine to solve the above technical defects.

[0009] According to one aspect of the present invention, a method for processing small data volume jobs based on a big data engine is proposed, the method comprising the following steps:

[0010] S1. Obtain a data file to be processed and determine whether the data file to be processed is a small data volume job. If yes, execute step S2.

[0011] S2. Determine whether there is an idle runtime environment in the resource pool. If so, encapsulate the business code uniformly, and the runtime environment interprets and executes the encapsulated code. If not, execute step S3.

[0012] S3. Determine whether the resource pool has reached its upper limit. If so, process the data file and put it into the job queue. If not, create a runtime environment and put it into the resource pool. Then, encapsulate the business code uniformly. The runtime environment interprets and executes the encapsulated code.

[0013] S4. Detect whether the idle time of the runtime environment in the resource pool exceeds a preset threshold. If so, release the runtime resources.

[0014] The above technical solution can achieve the purpose of quickly processing small batch data jobs by creating a runtime environment resource pool and reusing idle runtime environments.

[0015] In a specific embodiment, step S1 includes the following sub-steps:

[0016] S11, obtaining the data file to be processed;

[0017] S12, dynamically calculating the number of data records of the data file to be processed / the size of the user-processed file, and reading the maximum number of records / maximum processing storage capacity configured in the database to determine whether the job type of the data file to be processed is a small data volume job;

[0018] S13. If not, enter the normal task submission process; if yes, push the data file to be processed to the job queue and then execute the operation of step S2.

[0019] The above technical solution dynamically evaluates user data resources and first determines small data volume operations for subsequent processing.

[0020] In a specific embodiment, step S2 includes the following sub-steps:

[0021] S21. If it is the first execution, initialize the resource pool and create a runtime environment and put it into the resource pool;

[0022] S22. Determine whether the resource pool has an idle runtime environment. If so, reuse the runtime environment for data processing, uniformly encapsulate the business code, and the runtime environment interprets and executes the encapsulated code. If not, execute step S3.

[0023] The above technical solution is used to determine whether there is an idle runtime environment and decide whether to recreate the runtime environment.

[0024] In a specific embodiment, step S3 includes the following sub-steps:

[0025] S31. Compare the cluster running core / running memory usage with the set resource upper limit to determine whether the resource pool has reached the resource capacity upper limit;

[0026] S32. If the requirement is met, the runtime environment will no longer be created, and the data file will be processed and placed in the job queue to wait until there is an idle runtime environment. If the requirement is not met, a runtime environment will be created and placed in the resource pool, and the business code will be uniformly packaged. The runtime environment will interpret and execute the packaged code.

[0027] The above technical solution sets a capacity limit for the resource pool and performs unified resource management, which can free up more resources for executing other tasks and improve the utilization efficiency of cluster resources.

[0028] In a specific embodiment, step S4 includes the following sub-steps:

[0029] S41, regularly detecting whether the idle time of the runtime environment in the resource pool exceeds the maximum idle time according to the configured idle detection cycle;

[0030] S42. If it exceeds, the runtime environments are destroyed in chronological order to release idle cluster resources. If it does not exceed, step S41 is executed again.

[0031] Through the above technical solution, the runtime environment resource pool regularly detects the idle connection holding time, releases resources for idle connections that exceed the specified time, and avoids resource waste caused by idle periods.

[0032] In a specific embodiment, the business code is uniformly encapsulated in step S2 and step S3, and the runtime environment interprets and executes the encapsulated code, specifically including:

[0033] Unify the business code and execute different operations based on the different parameters passed in;

[0034] The runtime environment interprets and executes the encapsulated code, builds a data processing flow graph for data processing, and provides feedback on the job execution status.

[0035] Through the above technical solution, it is possible to submit a processing task based on the context environment retrieved from the runtime resource pool.

[0036] In a specific embodiment, during the execution of the packaged code by the runtime environment, the runtime environment state changes from idle to busy until the execution is completed and the idle state is restored.

[0037] Through the above technical solution, during the operation of the runtime resource pool, the user interprets and executes the business code by obtaining the idle runtime environment (getIdleSession), and returns the runtime environment (returnSession) after the execution is completed.

[0038] In a specific embodiment, the method further includes creating a resource pool and configuring the following parameters:

[0039] a. Runtime environment parameters: running memory, number of running cores, cluster connection information, and user service library;

[0040] b. Resource pool capacity parameters: maximum operating core occupancy, maximum memory occupancy;

[0041] c. Resource pool detection parameters: maximum idle time, idle detection cycle.

[0042] In a second aspect, the present application provides a small data volume job processing device based on a big data engine, characterized in that the device includes:

[0043] An evaluation module is used to obtain a data file to be processed, determine whether the data file to be processed is a small data volume job, and if so, execute the operations in the idle runtime environment judgment module;

[0044] The idle runtime environment judgment module is used to determine whether the resource pool has an idle runtime environment. If so, the business code is uniformly encapsulated and the runtime environment interprets and executes the encapsulated code. If not, the operations in the resource capacity judgment module are executed;

[0045] The resource capacity judgment module is used to determine whether the resource pool has reached the resource capacity limit. If so, the data file is processed and put into the job queue for waiting. If not, a runtime environment is created and placed into the resource pool. The business code is then uniformly packaged, and the runtime environment interprets and executes the packaged code.

[0046] The resource release module is used to detect whether the idle time of the runtime environment in the resource pool exceeds a preset threshold. If so, the runtime resources are released.

[0047] In a third aspect, the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, any of the above methods is implemented.

[0048] Compared with the prior art, the present invention has the following advantages:

[0049] By creating a runtime environment resource pool, this invention avoids cluster resource consumption caused by repeated creation and destruction of runtime resources. Submitting multiple batches of small data tasks improves data processing speed by reusing idle runtime environments. By setting a resource pool capacity cap and implementing unified resource management, more resources can be freed up for other tasks, improving cluster resource utilization efficiency.

[0050] As big data technologies are increasingly applied to various business scenarios, data presentation forms are becoming increasingly diverse. Optimizing cluster resource utilization has become a key concern. This invention can be applied to the processing of small batches of data in daily business operations, significantly improving cluster resource utilization efficiency while increasing data processing response speed, making it well-suited for daily data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Other features, objects and advantages of the present application will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0052] Figure 1 It is a flowchart of a method for processing small data volume jobs based on a big data engine according to the present application;

[0053] Figure 2 This is a schematic diagram of the main framework of the method for processing small data volume jobs based on a big data engine according to the present application;

[0054] Figure 3 This is a schematic diagram of the runtime environment session creation principle according to an embodiment of the present application;

[0055] Figure 4 This is a schematic diagram of the life cycle of a runtime resource pool SessionPool according to an embodiment of the present application;

[0056] Figure 5 is a schematic diagram of a small data volume job processing device based on a big data engine according to the present application;

[0057] Figure 6 It is a structural diagram of a computer system suitable for implementing the electronic device of the embodiment of the present application. DETAILED DESCRIPTION

[0058] The present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the relevant invention and are not intended to limit the invention. It should also be noted that, for ease of description, only portions relevant to the relevant invention are shown in the accompanying drawings.

[0059] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0060] Figure 1 The flowchart of the method for processing small data volume jobs based on a big data engine of the present application is shown; Figure 2 The main framework diagram of the method for processing small data volume jobs based on a big data engine according to the present application is shown. Figure 1 and Figure 2 , the method comprises the following steps:

[0061] S1. Obtain a data file to be processed and determine whether the data file to be processed is a small data volume job. If yes, execute step S2.

[0062] In this embodiment, the following sub-steps are specifically included:

[0063] S11, obtaining the data file to be processed;

[0064] S12, dynamically calculating the number of data records of the data file to be processed / the size of the user-processed file, and reading the maximum number of records / maximum processing storage capacity configured in the database to determine whether the job type of the data file to be processed is a small data volume job;

[0065] S13. If not, enter the normal task submission process; if yes, push the data file to be processed to the job queue and then execute the operation of step S2.

[0066] S2. Determine whether there is an idle runtime environment in the resource pool. If so, encapsulate the business code uniformly, and the runtime environment interprets and executes the encapsulated code. If not, execute step S3.

[0067] In this embodiment, the following sub-steps are specifically included:

[0068] S21. If it is the first execution, initialize the resource pool and create a runtime environment and put it into the resource pool;

[0069] S22. Determine whether the resource pool has an idle runtime environment. If so, reuse the runtime environment for data processing, uniformly encapsulate the business code, and the runtime environment interprets and executes the encapsulated code. If not, execute step S3.

[0070] S3. Determine whether the resource pool has reached the resource capacity limit. If so, process the data file and put it into the job queue to wait. If not, create a runtime environment and put it into the resource pool. Then, uniformly encapsulate the business code, and the runtime environment interprets and executes the encapsulated code.

[0071] In this embodiment, the following sub-steps are specifically included:

[0072] S31. Compare the cluster running core / running memory usage with the set resource upper limit to determine whether the resource pool has reached the resource capacity upper limit;

[0073] S32. If the requirement is met, the runtime environment will no longer be created, and the data file will be processed and placed in the job queue to wait until there is an idle runtime environment. If the requirement is not met, a runtime environment will be created and placed in the resource pool, and the business code will be uniformly packaged. The runtime environment will interpret and execute the packaged code.

[0074] Among them, in step S2 and step S3, the business code is uniformly encapsulated, and the runtime environment interprets and executes the encapsulated code, specifically including

[0075] Unify the business code and execute different operations based on the different parameters passed in;

[0076] The runtime environment interprets and executes the encapsulated code, builds a data processing flow graph for data processing, and provides feedback on the job execution status.

[0077] During the execution of the packaged code by the runtime environment, the runtime environment state changes from idle to busy until the execution is completed and the idle state is restored.

[0078] S4. Detect whether the idle time of the runtime environment in the resource pool exceeds a preset threshold. If so, release the runtime resources.

[0079] In this embodiment, the following sub-steps are specifically included:

[0080] S41, regularly detecting whether the idle time of the runtime environment in the resource pool exceeds the maximum idle time according to the configured idle detection cycle;

[0081] S42. If it exceeds, the runtime environments are destroyed in chronological order to release idle cluster resources. If it does not exceed, step S41 is executed again.

[0082] By creating a runtime environment resource pool, this invention avoids cluster resource consumption caused by repeated creation and destruction of runtime resources. Submitting tasks in small batches can improve data processing speed by reusing idle runtime environments. By setting a resource pool capacity cap and implementing unified resource management, more resources can be freed up for other tasks, improving cluster resource utilization efficiency.

[0083] As big data technologies are increasingly applied to various business scenarios, data presentation forms are becoming increasingly diverse. Optimizing cluster resource utilization has become a key concern. This invention, applied to the processing of small batches of data jobs in daily business operations, improves data processing response speed while significantly increasing cluster resource utilization efficiency, making it well-suited for daily data processing.

[0084] The following is a detailed description of the runtime environment resource pool design method and specific job processing flow.

[0085] 1. Runtime Environment Resource Pool Design Method

[0086] The lifecycle of a session pool includes the creation (init), run (run), and destruction (destory) states of the pool. The pool includes the creation time (createTime), run time (runTime), and state of a single session.

[0087] When creating a runtime resource pool, users need to configure the following parameters:

[0088] 1) Runtime environment parameters: executor-memory, executor-cores, cluster connection information (cluster-config), and user business library (bussiness-lib).

[0089] 2) Resource pool capacity parameters: Maximum CPU usage (MaxCpuCapicity), Maximum memory usage (MaxMemoryCapicity)

[0090] 3) Resource pool detection parameters: maximum idle time MaxIdleTime, idle detection period CheckTimeInteval.

[0091] To create a runtime environment, it is necessary to repackage the computing engine used by the business. Figure 3 A schematic diagram of the runtime environment session creation principle according to an embodiment of the present application is shown below. Figure 3 .

[0092] Taking Spark as an example, the core of program operation is to build the corresponding Interpreter, which is used to interpret and execute business code segments.

[0093] This implementation follows the core REPL (ReadEvalPrintLoop) class org.apache.spark.repl.sparkIloop in spark-shell. Spark REPL accepts user input, compiles and executes it, and returns the results to the user. Each line of input is encapsulated and compiled. The compiled Java bytecode is loaded by the classloader, and the execution engine then copies and executes the loaded bytecode into memory.

[0094] sparkInterpreter provides functional packaging for sparkILoop, mainly including the initialization method init for initializing spark-contetxt, and the inteprte method for parsing user code into Request class and loading and executing Request class.

[0095] The program creates a session based on the runtime environment parameters provided by the user. The session then creates an interpreter based on the specific engine being used. The created runtime environment session is added to the SessionPool resource pool and its status is set to Idle.

[0096] Figure 4FIG. 4 shows a schematic diagram of a runtime resource pool SessionPool resource pool life cycle according to an embodiment of the present application, as shown in FIG. Figure 4 As shown in the figure, during the operation of the runtime resource pool, the user interprets and executes the business code by obtaining an idle runtime environment (getIdleSession) and returning the runtime environment (returnSession) after execution. During the execution process, the runtime environment state changes from idle to busy (idle->busy) until the execution is completed and the idle state is restored.

[0097] Each runtime environment is periodically checked to see if its idle time has reached the maximum idle time (runTime > MaxIdleTime) based on the configured idle detection period CheckTimeInteval. If this exceeds the maximum idle time, the runtime environment is destroyed. During idle periods for small batch data services, runtime environments are destroyed in chronological order to release idle cluster resources.

[0098] 2. Specific operation processing flow

[0099] The small batch job submission process includes four steps: (1) dynamic evaluation of user data resources (2) acquisition of runtime environment (3) business encapsulation and data processing (4) runtime resource pool resource recovery.

[0100] (1) Dynamic evaluation of user data resources

[0101] The program dynamically calculates the number of data records (dataRecords) and the file size (fileSize) of the data file to be processed. It also reads the database settings (MaxRecordsLimit and MaxFileSizeLimit) to determine whether the job is a small-volume job. Small-volume jobs are pushed to the processing queue; large-volume jobs follow the normal task submission process.

[0102] (2) Obtaining the runtime environment

[0103] 1) If it is the first execution and the resource pool has not been created, the resource pool is initialized and the runtime environment is created and placed in the resource pool.

[0104] 2) If there is an idle runtime environment in the resource pool at this time, it will be reused for data processing.

[0105] 3) If the resource pool has been created and there is no idle runtime environment in the current pool, recreate it.

[0106] 4) If the resource pool exceeds the resource capacity limit (based on the comparison of the cluster's core CPU / memory usage with the set resource limits (MaxCpuCapicity / MaxMemoryCapicity)), no runtime environment will be created and the task will be queued until a free runtime environment becomes available.

[0107] (3) Business encapsulation and data processing

[0108] Based on the context environment retrieved from the runtime resource pool, the processing task is submitted. Users need to uniformly encapsulate different businesses, use a unified entry for tasks, and execute different jobs based on the input parameters (business type, business parameters).

[0109] The runtime environment interprets and executes the encapsulated processing code, builds a data processing flow graph for data processing, and provides feedback on the job execution status.

[0110] (4) Runtime resource pool resource recovery

[0111] The runtime environment resource pool regularly checks the idle connection holding time and releases resources for idle connections that exceed the specified time to avoid resource waste during idle periods.

[0112] Further references Figure 5 As an implementation of the above method, this application provides an embodiment of a small data volume job processing device based on a big data engine. Figure 1 Corresponding to the method embodiment shown, the system can be specifically applied to various electronic devices. The apparatus 200 includes the following modules:

[0113] The evaluation module 210 is used to obtain the data file to be processed and determine whether the data file to be processed is a small data volume job. If so, the operation in the idle runtime environment determination module 220 is executed;

[0114] The idle runtime environment determination module 220 is used to determine whether there is an idle runtime environment in the resource pool. If so, the business code is uniformly encapsulated and the runtime environment interprets and executes the encapsulated code. If not, the operation in the resource capacity determination module 230 is executed;

[0115] The resource capacity determination module 230 is used to determine whether the resource pool has reached its upper limit. If so, the data file is processed and placed in the job queue. If not, a runtime environment is created and placed in the resource pool. The business code is then uniformly packaged, and the runtime environment interprets and executes the packaged code.

[0116] The resource release module 240 is used to detect whether the idle time of the runtime environment in the resource pool exceeds a preset threshold, and if so, release the runtime resources.

[0117] The present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it implements any of the above methods.

[0118] Reference below Figure 6 , which shows a structural diagram of a computer system 600 suitable for implementing a terminal device or server of an embodiment of the present application. Figure 6 The terminal device or server shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0119] like Figure 6 As shown, computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage unit 608 into a random access memory (RAM) 603. Various programs and data required for the operation of system 600 are also stored in RAM 603. CPU 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to bus 604.

[0120] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, and the like; an output section 607 including a liquid crystal display (LCD), a speaker, and the like; a storage section 608 including a hard disk and the like; and a communication section 609 including a network interface card such as a LAN card or a modem. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. Removable media 611, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 610 as needed, so that computer programs read therefrom can be installed into the storage section 608 as needed.

[0121] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the method of the present application are executed. It should be noted that the computer-readable medium described in the present application can be a computer-readable signal medium or a computer-readable medium or any combination of the above two. The computer-readable medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this application, a computer-readable medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including, but not limited to, wireless, wire, optical cable, RF, or any suitable combination thereof.

[0122] Computer program code for performing the operations of the present application can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0123] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the prescribed logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart, can be implemented by a dedicated hardware-based system that performs the prescribed function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0124] The modules described in the embodiments of this application may be implemented in software or hardware. The units described may also be provided in a processor. For example, they may be described as comprising a receiving module, an acquisition module, a determination module, a calculation module, and a generation module. The names of these units do not, in some cases, limit the units themselves. For example, the receiving unit may also be described as "a module that obtains preset target user configuration information in response to determining that the verification request information includes a user name, a request time, a user signature code, and a client application code."

[0125] As another aspect, the present application also provides a computer-readable medium, which may be included in the server described in the above embodiment; or may exist independently and not be assembled into the server. The computer-readable medium carries one or more programs, which, when executed by the server, causes the server to: receive a verification request message sent by a client of a target user; in response to determining that the verification request message includes a user name, a request time, a user signature code, and a client application code, obtain preset configuration information of the target user, wherein the configuration information includes a preset user password corresponding to the user name; determine whether the verification request message is valid based on the request time; in response to determining that it is valid, determine whether a preset storage area includes a user signature code; in response to determining that it is not included, store the user signature code in a preset storage area, and calculate a server application code based on the user password, the request time, and the user signature code; in response to determining that the server application code matches the client application code, generate verification success information for indicating that the verification request is a legitimate request.

[0126] In addition, the computer-readable medium may be included in the terminal device described in the above embodiment, or may exist independently and not be incorporated into the terminal device. The computer-readable medium carries one or more programs. When executed by the terminal device, the one or more programs cause the terminal device to: obtain user information input by a target user, wherein the user information includes a user name and a user password; generate a user signature code representing the target user based on the user information; determine a request time; calculate a client application code based on the user password, the request time, and the user signature code; generate verification request information including the user name, the request time, the user signature code, and the client application code; and send the verification request information to the server.

[0127] The above description is merely a preferred embodiment of the present application and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also encompasses other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned inventive concept. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A method for processing small data volume jobs based on a big data engine, characterized in that: The following steps are involved: S1. Obtain a data file to be processed and determine whether the data file to be processed is a small data volume job. If so, execute step S2. Step S1 includes the following sub-steps: S11. Obtain a data file to be processed; S12. Dynamically calculate the number of data records in the data file to be processed or the size of the user-processed file, and read the maximum number of records or maximum processing storage capacity configured in the database to determine whether the job type of the data file to be processed is a small data volume job; S13. If not, enter the normal task submission process. If so, push the data file to be processed to the job queue and then execute the operation of step S2. S2. Determine whether there is an idle runtime environment in the resource pool. If so, encapsulate the business code uniformly, and the runtime environment interprets and executes the encapsulated code. If not, execute step S3. S3. Determine whether the resource pool has reached the resource capacity upper limit. If so, the processed data file enters the job queue and waits. If not, create the runtime environment and put it into the resource pool, then uniformly encapsulate the business code, and the runtime environment interprets and executes the encapsulated code. Step S3 includes the following sub-steps: S31. Compare the cluster running core or running memory usage with the set resource upper limit to determine whether the resource pool has reached the resource capacity upper limit. S32. If so, no longer create the runtime environment, and the processed data file enters the job queue and waits until there is an idle runtime environment. If not, create the runtime environment and put it into the resource pool, then uniformly encapsulate the business code, and the runtime environment interprets and executes the encapsulated code. S4. Detect whether the idle time of the runtime environment in the resource pool exceeds a preset threshold. If so, release the runtime resources.

2. The method for processing small data volume jobs based on a big data engine according to claim 1, characterized in that: Step S2 includes the following sub-steps: S21. If it is the first execution, initialize the resource pool and create a runtime environment and put it into the resource pool; S22. Determine whether the resource pool has an idle runtime environment. If so, reuse the runtime environment for data processing, uniformly encapsulate the business code, and the runtime environment interprets and executes the encapsulated code. If not, execute step S3.

3. The method for processing small data volume jobs based on a big data engine according to claim 1, characterized in that: Step S4 includes the following sub-steps: S41. Regularly detecting whether the idle time of the runtime environment in the resource pool exceeds the maximum idle time according to the configured idle detection period; S42: If it exceeds, the runtime environments are destroyed in chronological order to release idle cluster resources. If it does not exceed, step S41 is executed again.

4. The method for processing small data volume jobs based on a big data engine according to claim 1, characterized in that: In steps S2 and S3, the business code is uniformly encapsulated, and the runtime environment interprets and executes the encapsulated code, specifically including: Unify the business code and execute different operations based on the different parameters passed in; The runtime environment interprets and executes the encapsulated code, constructs a data processing flow graph to process data, and provides feedback on the job execution status.

5. The method for processing small data volume jobs based on a big data engine according to claim 4, characterized in that: During the execution process of the runtime environment interpreting and executing the encapsulated code, the runtime environment state changes from idle to busy until the execution is completed and the idle state is restored.

6. The method for processing small data volume jobs based on a big data engine according to claim 1, characterized in that: It also includes creating the resource pool and configuring the following parameters: a. Runtime environment parameters: running memory, number of running cores, cluster connection information, and user service library; b. Resource pool capacity parameters: maximum operating core occupancy, maximum memory occupancy; c. Resource pool detection parameters: maximum idle time, idle detection cycle.

7. A small data volume job processing device based on a big data engine, characterized in that: The method according to any one of claims 1 to 6, wherein the device comprises: An evaluation module is used to obtain a data file to be processed, determine whether the data file to be processed is a small data volume job, and if so, execute the operations in the idle runtime environment determination module; An idle runtime environment judgment module is used to judge whether there is an idle runtime environment in the resource pool. If so, the business code is uniformly encapsulated, and the runtime environment interprets and executes the encapsulated code. If not, the operation in the resource capacity judgment module is executed; A resource capacity determination module is configured to determine whether the resource pool has reached its upper limit. If so, the processed data file enters a job queue and waits. If not, the runtime environment is created and placed in the resource pool, and the business code is uniformly packaged. The runtime environment interprets and executes the packaged code. The resource release module is used to detect whether the idle time of the runtime environment in the resource pool exceeds a preset threshold, and if so, release the runtime resources.

8. A computer-readable storage medium, wherein a computer program is stored in the medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images