Method, device and electronic device for accelerating processing of elasticsearch data in big data

Abstract

The embodiment of the present invention provides a method, device and electronic device for accelerating the processing of elasticsearch data in big data, belonging to the field of data processing technology, the method comprising: extracting features from elasticsearch data to obtain a data feature set K corresponding to the elasticsearch data; using a main process to initialize a running resource Z matching the task requirements; after n subprocesses are started, the main process divides the elasticsearch data into a segmented data set containing n data segments; the processing performance of the current device processing the elasticsearch data is evaluated to obtain a performance evaluation result; based on the performance evaluation result P, the computing power evaluation result S and the balancing function J, the thread number set to be allocated in the n subprocesses is calculated. The present invention can improve the processing efficiency of elasticsearch data.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device and electronic device for accelerating the processing of elasticsearch data in big data. Background Art

[0002] Elasticsearch is an open source search engine and analysis engine designed for large-scale data sets, providing real-time, distributed, RESTful-style search and analysis capabilities. Its main uses include:

[0003] 1. Full-text search: Elasticsearch supports advanced full-text search functions, which can search large amounts of text data quickly and efficiently, including fuzzy matching, synonym processing, spelling correction, etc. It is often used to build website search engines, document search, log analysis and other scenarios.

[0004] 2. Log and event data management: Elasticsearch is often used to centrally collect, store, and analyze log data. It can process logs from various sources, such as application logs, system logs, security logs, etc., to facilitate troubleshooting, performance monitoring, and compliance auditing.

[0005] 3. Index storage and aggregation: As a time series data storage and analysis platform, elasticsearch can store time series data and perform complex aggregation operations such as statistical counting, average calculation, percentile analysis, etc. It is suitable for Internet of Things (IoT) data, performance monitoring indicators, etc.

[0006] The existing technology has the following problems in processing elasticsearch data:

[0007] 1. Low resource utilization: A single multi-threaded or multi-process approach fails to fully utilize hardware resources.

[0008] 2. Slow processing speed: When processing large-scale data, a single parallel method has limited efficiency.

[0009] 3. Poor scalability: Faced with the ever-increasing amount of data and complex computing tasks, traditional methods are difficult to expand.

[0010] The above problems become technical problems that need to be solved. Summary of the invention

[0011] In view of this, an embodiment of the present invention provides a method, device and electronic device for accelerating the processing of elasticsearch data in big data, which at least partially solves the problems existing in the prior art.

[0012] In a first aspect, an embodiment of the present invention provides a method for accelerating the processing of elasticsearch data in big data, comprising:

[0013] After obtaining the elasticsearch data to be processed, feature extraction is performed on the elasticsearch data to obtain a data feature set K corresponding to the elasticsearch data;

[0014] Calculate the task requirements required for elasticsearch data processing according to the data feature set K, and use the main process to initialize the running resources Z that match the task requirements;

[0015] Based on the running resource Z, the number n of sub-processes to be started is determined. The n sub-processes are started by the main process. After the n sub-processes are started, the main process divides the elasticsearch data into a segmented data set D={D1, D2, ...Dn} containing n data segments, and injects the n data segments into each independently running sub-process;

[0016] Evaluate the processing performance of the current device that processes elasticsearch data, and obtain the performance evaluation result P={Pu,Pm,Pn,Pr}. Evaluate the computational complexity of the segmented data set D={D1,D2,…Dn}, and obtain the computing power evaluation result S={S1,S2,…Sn}. Pu represents the process computing performance of the current device, Pm represents the storage response performance of the current device, Pu represents the network transmission performance of the current device, and Pu represents the process switching performance of the current device.

[0017] Based on the performance evaluation result P, the computing power evaluation result S and the balancing function J, the set of thread numbers X={X1,X2,…Xn} that need to be allocated in the n sub-processes is calculated, and multiple threads are created in each sub-process based on the thread number set X. After each sub-process completes the data calculation task, the calculation result is returned to the main process, and the main process integrates and outputs the calculation results.

[0018] According to a specific implementation of the embodiment of the present disclosure, the feature extraction of elasticsearch data to obtain a data feature set K corresponding to the elasticsearch data includes:

[0019] The data volume, document complexity, query frequency, index update rate, query complexity and number of concurrent users of elasticsearch data are calculated respectively to obtain the data volume feature Kvo, document complexity feature Kco, query frequency feature Kqu, index update rate feature Kud, query complexity feature Kqc and concurrent user number feature Kcr, and then the data feature set K={Kvo,Kco,Kqu,Kud,Kqc,Kcr} corresponding to the elasticsearch data is obtained.

[0020] According to a specific implementation of the embodiment of the present disclosure, the step of calculating the task requirements required for elasticsearch data processing according to the data feature set K and initializing the running resources Z matching the task requirements using the main process includes:

[0021] Construct and run the resource calculation function Z = f(Kvo, Kco, Kqu, Kud, Kqc, Kcr) to calculate the storage requirement Se, computing resource requirement Ce, and memory requirement Me required for elasticsearch data processing, where: , Rr is the replication factor, and Bf is the index buffer size.

[0022] According to a specific implementation of the embodiment of the present disclosure, the determining the number n of sub-processes that need to be started based on the running resource Z includes:

[0023] The calculation formula for building the number of subprocesses n is:

[0024]

[0025] Indicates the number of CPU cores available in the system. Indicates the number of CPU cores that each child process is expected to occupy, taking into account the CPU intensity of the task. Indicates the amount of memory available in the system. Indicates the expected memory size of each child process. represents the total time budget to complete all tasks, It represents the expected time for a single subprocess to process a single task unit, and NL represents the theoretical maximum parallelism of the task. Indicates the system reserved resource factor.

[0026] According to a specific implementation of an embodiment of the present disclosure, the processing performance of the current device processing elasticsearch data is evaluated to obtain a performance evaluation result P={Pu, Pm, Pn, Pr}, and the computational complexity of the segmented data set D={D1, D2, ...Dn} is evaluated to obtain a computing power evaluation result S={S1, S2, ...Sn}, including:

[0027] Build performance evaluation result calculation formula:

[0028]

[0029] and are the first adjustment factor and the second adjustment factor respectively.

[0030] According to a specific implementation of an embodiment of the present disclosure, the processing performance of the current device processing elasticsearch data is evaluated to obtain a performance evaluation result P={Pu, Pm, Pn, Pr}, and the computational complexity of the segmented data set D={D1, D2, ...Dn} is evaluated to obtain a computing power evaluation result S={S1, S2, ...Sn}, and further includes:

[0031] Obtain the time complexity and space complexity Ci of the algorithm used for data Di, as well as the hardware performance H of the current device, to form an evaluation result Si for data Di:

[0032]

[0033] and is the adjustment coefficient, γ is the index of the impact of data scale on complexity, and β is the index of the relief of hardware performance on computing requirements.

[0034] According to a specific implementation of the embodiment of the present disclosure, the calculation of the number of threads X={X1, X2, ...Xn} to be allocated in n sub-processes based on the performance evaluation result P, the computing power evaluation result S and the balancing function J includes:

[0035] Construct a calculation function to calculate the number of calculation threads Xi:

[0036]

[0037] Qi is an adjustment factor for the i-th child process.

[0038] According to a specific implementation of the embodiment of the present disclosure, the calculation of the set of thread numbers X={X1, X2, ...Xn} to be allocated in n sub-processes based on the performance evaluation result P, the computing power evaluation result S and the balancing function J also includes:

[0039] Construct equilibrium function J:

[0040]

[0041] is the weight parameter, τ and λ are adjustment parameters.

[0042] In a second aspect, an embodiment of the present invention provides a device for accelerating the processing of elasticsearch data in big data, including:

[0043] The extraction module obtains the elasticsearch data to be processed, and then extracts features from the elasticsearch data to obtain a data feature set K corresponding to the elasticsearch data;

[0044] The calculation module calculates the task requirements required for elasticsearch data processing according to the data feature set K, and uses the main process to initialize the running resources Z that match the task requirements;

[0045] A determination module determines the number n of sub-processes to be started based on the running resource Z, wherein the n sub-processes are started by the main process. After the n sub-processes are started, the main process divides the elasticsearch data into a segmented data set D={D1, D2, ...Dn} containing n data segments, and injects the n data segments into each independently running sub-process;

[0046] The evaluation module evaluates the processing performance of the current device that processes elasticsearch data, and obtains the performance evaluation result P={Pu,Pm,Pn,Pr}. It also evaluates the computational complexity of the segmented data set D={D1,D2,…Dn}, and obtains the computing power evaluation result S={S1,S2,…Sn}, where Pu represents the process computing performance of the current device, Pm represents the storage response performance of the current device, Pu represents the network transmission performance of the current device, and Pu represents the process switching performance of the current device.

[0047] The execution module calculates the set of thread numbers X={X1,X2,…Xn} that need to be allocated in n sub-processes based on the performance evaluation result P, the computing power evaluation result S and the balancing function J, and creates multiple threads in each sub-process based on the thread number set X. After each sub-process completes the data calculation task, the calculation result is returned to the main process, and the main process integrates and outputs the calculation results.

[0048] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0049] at least one processor; and,

[0050] a memory communicatively connected to the at least one processor; wherein,

[0051] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for accelerating the processing of elasticsearch data in big data in any of the first aspects or any of the implementations of the first aspect.

[0052] In a fourth aspect, an embodiment of the present invention further provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the method for accelerating the processing of elasticsearch data in big data in the aforementioned first aspect or any implementation of the first aspect.

[0053] In a fifth aspect, an embodiment of the present invention further provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the method for accelerating the processing of elasticsearch data in big data in the aforementioned first aspect or any implementation of the first aspect.

[0054] The scheme for accelerating the processing of elasticsearch data in big data in an embodiment of the present invention includes: after obtaining the elasticsearch data to be processed, extracting features from the elasticsearch data to obtain a data feature set K corresponding to the elasticsearch data; calculating the task requirements required for elasticsearch data processing according to the data feature set K, and using a main process to initialize a running resource Z matching the task requirements; based on the running resource Z, determining the number n of subprocesses to be started, the n subprocesses are started by the main process, and after the n subprocesses are started, the main process divides the elasticsearch data into a segmented data set D={D1, D2, ... Dn} containing n data segments, and injects the n data segments into each independently running subprocess; During the process; the processing performance of the current device processing elasticsearch data is evaluated to obtain the performance evaluation result P={Pu, Pm, Pn, Pr}, and the computational complexity of the segmented data set D={D1, D2, ...Dn} is evaluated to obtain the computing power evaluation result S={S1, S2, ...Sn}, where Pu represents the process computing performance of the current device, Pm represents the storage response performance of the current device, Pu represents the network transmission performance of the current device, and Pu represents the process switching performance of the current device; based on the performance evaluation result P, the computing power evaluation result S and the balancing function J, the set of threads X={X1, X2, ...Xn} that need to be allocated in the n sub-processes is calculated, and multiple threads are created in each sub-process based on the thread number set X. After each sub-process completes the data calculation task, the calculation result is returned to the main process, and the main process integrates and outputs the calculation result. The scheme of this application has the following beneficial effects:

[0055] (1) By combining multi-process and multi-threading technologies, we can make full use of multi-core CPUs and large memories to improve the efficiency of big data processing.

[0056] (2) Maximize the utilization of hardware resources by reasonably allocating tasks.

[0057] (3) In the face of growing data volumes and more complex tasks, the system can expand its processing capabilities by increasing the number of processes and threads.

[0058] (4) By shortening the request events to elasticsearch, the performance of elasticsearch's external services is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0060] Figure 1 A schematic diagram of a method for accelerating the processing of elasticsearch data in big data provided by an embodiment of the present invention;

[0061] Figure 2 A schematic flow chart of another method for accelerating the processing of elasticsearch data in big data provided by an embodiment of the present invention;

[0062] Figure 3 A schematic diagram of the structure of a device for accelerating the processing of elasticsearch data in big data provided by an embodiment of the present invention;

[0063] Figure 4 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0065] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0066] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0067] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The drawings only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0068] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0069] The embodiment of the present disclosure provides a method for accelerating the processing of elasticsearch data in big data. The method for accelerating the processing of elasticsearch data in big data provided in this embodiment can be performed by a computing device, which can be implemented as software, or as a combination of software and hardware, and the computing device can be integrated in a server, a terminal device, etc.

[0070] See also Figure 1 and Figure 2 , the embodiment of the present disclosure provides a method for accelerating the processing of elasticsearch data in big data, including:

[0071] S101, after obtaining the elasticsearch data to be processed, feature extraction is performed on the elasticsearch data to obtain a data feature set K corresponding to the elasticsearch data.

[0072] In order to extract features from Elasticsearch data, we can calculate the data volume feature Kvo, document complexity feature Kco, query frequency feature Kqu, index update rate feature Kud, query complexity feature Kqc and concurrent user number feature Kcr respectively, and finally summarize them to obtain the data feature set K.

[0073] For the data volume feature KVO, the total number of documents in all indexes in the Elasticsearch data can be counted. Document complexity can be measured based on the number of fields, field types (such as text, numbers, objects, arrays, etc.), and the depth of nested fields. By sampling a certain number of documents, the average number of fields per document, the distribution of different field types, and the average nesting depth of fields can be calculated.

[0074] The query frequency feature Kqu can be calculated by collecting query requests within a period of time through the monitoring system and counting the number of queries per second (QPS). The index update rate feature Kud can be obtained by monitoring the index write operations, including document creation, update and deletion operations, and counting the number of index updates per second (UPS).

[0075] The query complexity feature Kqc can be obtained by analyzing the complexity of the query request, such as the logical operators (AND, OR) used in the query, the number of filters, the depth of nested queries, the number of sorted fields, the type and number of aggregations, etc. The concurrent user number feature Kcr can be obtained by monitoring the concurrent user activities recorded by the system and identifying the number of users performing queries or data operations simultaneously during peak hours.

[0076] By integrating the six features calculated above, we can obtain the data feature set K = {Kvo, Kco, Kqu, Kud, Kqc, Kcr} corresponding to the Elasticsearch data.

[0077] S102, calculating the task requirements required for elasticsearch data processing according to the data feature set K, and using the main process to initialize the running resources Z that match the task requirements.

[0078] Based on the previously defined data feature set K = {Kvo, Kco, Kqu, Kud, Kqc, Kcr}, the task requirements for Elasticsearch data processing can be further calculated, including the required computing resources, memory resources, storage resources, and network resources. Then, based on these requirements, the main process can initialize the corresponding running resources Z to ensure that the resources match the task requirements.

[0079] When calculating resource requirements Zcpu, we can estimate the CPU cycles required to process the query based on the query frequency feature Kqu and the query complexity feature Kqc. Consider the index update rate feature Kud to evaluate the CPU requirements for index write operations. Combined with the concurrent user number feature Kcr, we can adjust CPU resources to cope with high concurrency scenarios.

[0080] For the memory resource requirement Zmem, the size setting of the index cache, query cache, and field data cache can be guided by the data volume feature Kvo, and the consumption of field parsing and query processing in memory can be affected by the document complexity feature Kco.

[0081] For the network resource demand Znet, the network bandwidth demand can be evaluated according to the query frequency Kqu and the number of concurrent users Kcr, and the network throughput requirement for data writing can be determined according to the index update rate Kud.

[0082] The main process can perform initialization operations based on the task requirements calculated above:

[0083] Allocate CPU cores: Dynamically adjust or statically allocate the number of CPU cores to child processes based on computing resource demand Zcpu, taking into account hyperthreading technology and CPU affinity settings.

[0084] Memory configuration: Allocate sufficient memory resources Zmem to the Elasticsearch process, including heap memory, direct memory, and file system cache, to ensure stable system operation.

[0085] Storage optimization: Configure storage resources Zstorage, including SSD or HDD selection, RAID configuration, and hot spare disk strategy.

[0086] Network settings: According to the network resource requirements Znet optimizes the network configuration, including bandwidth allocation, port settings and network protocol optimization.

[0087] Finally, the master process is set up according to the above requirements to ensure that the resource configuration of the entire Elasticsearch cluster or a single node matches the task requirements to achieve optimal processing performance and resource utilization.

[0088] S103, based on the running resource Z, determine the number n of sub-processes that need to be started, and the n sub-processes are started by the main process. After the n sub-processes are started, the main process divides the elasticsearch data into a split data set D={D1, D2, ...Dn} containing n data segments, and injects the n data segments into each independently running sub-process.

[0089] Assume that the resource consumption of each child process is a unit resource set, including a CPU core, a certain amount of memory, and a fair share of network and storage resources.

[0090] According to the total amount of resources in Z, calculate the maximum number of subprocesses n that can be supported max For example, if there are C physical cores in total, after hyperthreading, n max= C. The actual number of sub-processes n that are started can be selected as n≤C considering the efficient use of resources and system stability. The specific value may need to be fine-tuned based on historical operation data and performance testing.

[0091] As an example, construct the formula for calculating the number of subprocesses n:

[0092]

[0093] Indicates the number of CPU cores available in the system. Indicates the number of CPU cores that each child process is expected to occupy, taking into account the CPU intensity of the task. Indicates the amount of memory available in the system. Indicates the expected memory size of each child process. represents the total time budget to complete all tasks, It represents the expected time for a single subprocess to process a single task unit, and NL represents the theoretical maximum parallelism of the task. Indicates the system reserved resource factor.

[0094] Once n is determined, the main process divides the Elasticsearch data set into n data segments D = {D1, D2,… Dn} according to its size and structure. Each data segment is as balanced as possible to ensure uniform data distribution and reduce performance bottlenecks caused by uneven processing. The main process injects the divided data segments Di into the corresponding child processes Pi through inter-process communication (such as shared memory, message queues, or network communication).

[0095] Through the above steps, the system can effectively allocate and manage subprocesses based on the current running resource Z and efficiently process Elasticsearch data.

[0096] S104, evaluate the processing performance of the current device that processes elasticsearch data to obtain a performance evaluation result P={Pu, Pm, Pn, Pr}, and perform computational complexity evaluation on the segmented data set D={D1, D2, …Dn} to obtain a computing power evaluation result S={S1, S2, …Sn}, where Pu represents the process computing performance of the current device, Pm represents the storage response performance of the current device, Pu represents the network transmission performance of the current device, and Pu represents the process switching performance of the current device.

[0097] The performance evaluation result calculation formula can be constructed:

[0098]

[0099] and are the first adjustment factor and the second adjustment factor respectively, and the performance evaluation result P is obtained by the performance evaluation result calculation formula.

[0100] You can also obtain the time complexity and space complexity Ci of the algorithm used by the data Di, as well as the hardware performance H of the current device, to form an evaluation result Si for the data Di:

[0101]

[0102] and is the adjustment coefficient, γ is the index of the impact of data scale on complexity, and β is the index of the relief of hardware performance on computing requirements.

[0103] S105, based on the performance evaluation result P, the computing power evaluation result S and the balancing function J, calculate the set of thread numbers X={X1,X2,…Xn} that need to be allocated in the n sub-processes, and create multiple threads in each sub-process based on the thread number set X. After each sub-process completes the data calculation task, the calculation result is returned to the main process, and the main process integrates and outputs the calculation results.

[0104] Specifically, we can construct a calculation function to calculate the number of threads Xi:

[0105]

[0106] Qi is an adjustment factor for the i-th child process.

[0107] Construct equilibrium function J:

[0108]

[0109] is the weight parameter, τ and λ are adjustment parameters.

[0110] The main process allocates Xi threads to each child process Pi according to the calculated thread number set X = {X1, X2, ..., Xn}, and injects the corresponding data segment Di. After each child process receives its data segment and thread number, it creates Xi threads internally. These threads process their respective data segments in parallel and perform data computing tasks.

[0111] When the threads in all subprocesses complete the data calculation tasks, they return their respective calculation results to the corresponding subprocesses. Each subprocess summarizes the calculation results of all its internal threads and returns the results to the main process through inter-process communication mechanisms such as shared memory, message queues, or network communication. The main process receives the calculation results of all subprocesses and performs integration operations, which may include data merging, result verification, exception handling, and other steps. After the integration is completed, the main process outputs the final calculation results, completing the entire data processing flow.

[0112] Through such a design, the system can effectively utilize multi-level parallel processing at the process level and thread level to improve the speed and efficiency of data processing while ensuring resource balance.

[0113] See also Figure 2 The main process is the core of the entire big data processing system and is responsible for initializing resources, task allocation, and process management. The detailed steps for starting the main process are as follows:

[0114] S21 Initialize resources

[0115] When the main process starts, a series of resource initialization operations need to be performed, including but not limited to the following steps:

[0116] Load configuration file: Read the configuration file to obtain parameter information required for data processing, such as data source path, number of processes and threads, log configuration, etc.

[0117] Log initialization: Set up a log system to record important events and error information during system operation to facilitate subsequent debugging and maintenance.

[0118] Data source connection: Establish a connection with a data source, such as a database connection, file system connection, etc., to ensure that data can be read and written smoothly.

[0119] Memory allocation: pre-allocate the memory space required for data processing according to the parameters in the configuration file to ensure the rational use of memory resources.

[0120] S22 Determine the number of processes and threads

[0121] The main process needs to reasonably determine the number of subprocesses to be created and the number of threads in each subprocess based on actual task requirements and system resource conditions:

[0122] System resource detection: detect the current system's CPU core number, memory size and other resource information.

[0123] Task complexity assessment: Evaluate the required computing resources based on the size and complexity of the data to be processed.

[0124] Determine the number of processes: Determine the number of subprocesses to be created based on the number of CPU cores and task complexity. This number is usually 1 to 2 times the number of CPU cores to fully utilize the computing power of multi-core CPUs.

[0125] Determine the number of threads: Determine the number of threads that need to be created in each subprocess based on the parallelism requirements of the task and the system memory size. Usually, each subprocess creates multiple threads to achieve high concurrent processing.

[0126] S23 Create a process pool

[0127] The main process uses the process pool technology to manage multiple sub-processes:

[0128] Process pool initialization: Create a process pool object and specify the required number of child processes.

[0129] Process pool configuration: Configure the parameters of the process pool, such as the maximum number of processes, process restart strategy, etc., to ensure that the process pool can run efficiently and stably.

[0130] Start subprocess: Add tasks to be executed to the process pool and start multiple subprocesses. Each subprocess will run independently and wait for tasks assigned by the main process.

[0131] S24 Data Division and Task Allocation

[0132] The main process divides the data to be processed and assigns it to each subprocess:

[0133] Data partitioning: According to the data volume and task complexity, the data to be processed is divided into multiple data blocks, and each data block will serve as an independent processing unit.

[0134] Task queue: Add the divided data blocks to the task queue. Each task in the task queue contains a data block and its processing method.

[0135] Task allocation: The main process takes tasks from the task queue and assigns them to child processes in the process pool for processing.

[0136] S25 Monitoring and Scheduling

[0137] During the task execution, the main process needs to monitor the running status of the subprocess in real time and schedule it:

[0138] Process monitoring: Real-time monitoring of the running status of each sub-process, including whether the process is running normally and whether there are any abnormalities.

[0139] Load balancing: Dynamically adjust the task allocation strategy according to the load of the sub-process to ensure load balancing of each sub-process and avoid overloading or idleness of some processes.

[0140] Fault recovery: When a sub-process is detected to have a fault or abnormal exit, the main process should restart the sub-process in time and reallocate unfinished tasks to ensure the stability and continuity of the system.

[0141] S26 Results Collection and Merging

[0142] After the subprocess completes its task, the main process needs to collect the processing results of each subprocess and merge them:

[0143] Result collection: The main process obtains the processing results from the child process, usually through inter-process communication (such as pipes, queues, etc.).

[0144] Result merging: Integrate the results returned by each sub-process to generate the final processing result.

[0145] Result output: Output the integrated results to the specified storage medium, such as writing to a database, saving to a file, etc.

[0146] Through the above detailed main process startup steps, we can ensure that the system can run efficiently and stably, make full use of the advantages of multi-core CPU and multi-threading, and accelerate the big data processing process.

[0147] In big data processing systems, multi-process creation is a key step. By creating multiple independent sub-processes, the computing power of multi-core CPUs can be effectively utilized, avoiding Python's global interpreter lock (GIL) restrictions and significantly improving the parallelism of data processing. The specific steps are as follows:

[0148] S31 process pool initialization

[0149] The main process first needs to initialize a process pool to manage and schedule the creation and running of multiple subprocesses:

[0150] 1. Create a process pool object: Use the `multiprocessing.Pool` class in Python or other multiprocessing libraries to create a process pool object. For example:

[0151] from multiprocessing import Pool

[0152] pool = Pool(processes=number_of_processes)

[0153] Among them, `number_of_processes` is the number of child processes to be created, usually 1 to 2 times the number of CPU cores.

[0154] 2. Configure process pool parameters: According to task requirements and system resources, configure relevant parameters of the process pool, such as the maximum number of processes, task queue length, etc., to ensure that the process pool can run efficiently and stably.

[0155] S32 process start

[0156] After the process pool is initialized, the main process starts multiple subprocesses and assigns pending tasks to these subprocesses:

[0157] 1. Define a task function: Define a task function that contains the data processing logic that the subprocess needs to execute. For example:

[0158] def process_data(data_chunk):

[0159] # Data processing logic

[0160] result = perform_computation(data_chunk)

[0161] return result

[0162] 2. Task allocation: Assign the data blocks to be processed to the child processes in the process pool. You can use the `apply_async` method to assign tasks asynchronously, or use the `map` method to assign tasks in batches. For example:

[0163] results = pool.map(process_data, data_chunks)

[0164] or:

[0165] for data_chunk in data_chunks:

[0166] pool.apply_async(process_data, args=(data_chunk,), callback=collect_results)

[0167] 3. Start subprocess: The process pool will automatically create and start multiple subprocesses. Each subprocess runs independently and performs the assigned tasks.

[0168] S33 Inter-process communication

[0169] In a multi-process environment, subprocesses need to communicate to coordinate work and share data. Common inter-process communication mechanisms include pipes, queues, and shared memory:

[0170] 1. Pipe: Pipes enable two-way communication between processes. For example:

[0171] from multiprocessing import Pipe

[0172] parent_conn, child_conn = Pipe()

[0173] The main process and child process can send and receive data through `parent_conn` and `child_conn`.

[0174] 2. Queue: Use queues to implement data transfer and task scheduling between multiple processes. For example:

[0175] from multiprocessing import Queue

[0176] queue = Queue()

[0177] queue.put(task)

[0178] result = queue.get()

[0179] 3. Shared Memory: Shared memory is used to share data between processes. For example:

[0180] from multiprocessing import Value, Array

[0181] shared_value = Value('i', 0)

[0182] shared_array = Array('i', range(10))

[0183] S34 Process Monitoring and Management

[0184] The main process needs to monitor the running status of the subprocess in real time and perform necessary management operations:

[0185] 1. Status monitoring: Real-time monitoring of the running status of each subprocess, including CPU and memory usage, whether the process is running normally, etc. You can use the `Process` object in the `multiprocessing` module to obtain the information of the subprocess.

[0186] for process in pool._pool:

[0187] print(process.is_alive())

[0188] 2. Fault handling: When a subprocess is detected to have exited abnormally or failed, the main process should restart the subprocess in time and reallocate the unfinished tasks. For example:

[0189] if not process.is_alive():

[0190] process.terminate()

[0191] new_process = Process(target=process_data, args=(data_chunk,))

[0192] new_process.start()

[0193] 3. Load balancing: Dynamically adjust the task allocation strategy according to the load of the sub-process to ensure load balancing of each sub-process and avoid overloading or idleness of some processes.

[0194] S35 Results Collection and Merging

[0195] After the subprocess completes the task, the main process needs to collect the processing results of each subprocess and merge them:

[0196] 1. Result collection: The main process obtains the processing results from the child process through an inter-process communication mechanism (such as a queue or pipe).

[0197] result = queue.get()

[0198] results.append(result)

[0199] 2. Result merging: Integrate the results returned by each sub-process to generate the final processing result.

[0200] final_result = merge_results(results)

[0201] 3. Close the process pool: After all tasks are completed, the main process closes the process pool to release system resources.

[0202] pool.close()

[0203] pool.join()

[0204] Through detailed multi-process creation steps, efficient parallel data processing can be achieved, the computing power of multi-core CPUs can be fully utilized, and the big data processing process can be accelerated.

[0205] In each child process, by using the thread pool technology to create multiple threads, you can further refine the task allocation and improve the parallelism and efficiency of data processing. The detailed steps are as follows:

[0206] S41 thread pool initialization

[0207] After each child process is started, a thread pool needs to be initialized first to manage and schedule the creation and running of multiple threads:

[0208] 1. Import the thread pool library: You can use `concurrent.futures.ThreadPoolExecutor` in the Python standard library to create and manage thread pools. For example:

[0209] from concurrent.futures import ThreadPoolExecutor

[0210] 2. Create a thread pool object: Create a thread pool object in the child process and specify the number of threads required. For example:

[0211] thread_pool = ThreadPoolExecutor(max_workers=number_of_threads)

[0212] Among them, `number_of_threads` is the number of threads to be created, which is usually determined by the task complexity of the child process and system resources.

[0213] S42 Define thread tasks

[0214] Define the task function that each thread needs to execute, including the specific logic of data processing:

[0215] 1. Task function definition: Write a task function that will be executed in each thread. For example:

[0216] def thread_task(data_segment):

[0217] # Specific data processing logic

[0218] processed_data = perform_computation(data_segment)

[0219] return processed_data

[0220] S43 Task Allocation

[0221] In the child process, the data block assigned to the process is further divided into multiple small segments and assigned to each thread in the thread pool:

[0222] 1. Data partitioning: Divide the data blocks received by the child process into smaller segments to facilitate parallel processing by multiple threads. For example:

[0223] data_segments = split_data(data_block, number_of_threads)

[0224] 2. Submit tasks: assign data segments to each thread in the thread pool and use the `submit` method to submit tasks asynchronously. For example:

[0225] futures = [thread_pool.submit(thread_task, segment) for segment indata_segments]

[0226] S44 Resource sharing between threads

[0227] Threads can share the memory resources of the subprocess, such as shared variables and data structures, to facilitate collaborative processing of tasks:

[0228] 1. Shared memory: Shared data structures can be defined in the child process for each thread to access. For example:

[0229] shared_data = []

[0230] 2. Thread safety: When accessing shared resources, use thread synchronization mechanisms (such as locks) to ensure thread safety and avoid race conditions. For example:

[0231] import threading

[0232] lock = threading.Lock()

[0233] def thread_task(data_segment):

[0234] global shared_data

[0235] processed_data = perform_computation(data_segment)

[0236] with lock:

[0237] shared_data.append(processed_data)

[0238] return processed_data

[0239] S45 Results collection and merging

[0240] After the thread completes its task, it is necessary to collect the processing results of each thread and merge them:

[0241] 1. Result collection: Collect the processing results of each thread through the `futures` object. For example:

[0242] results = [future.result() for future in futures]

[0243] 2. Result merging: Integrate the results returned by each thread to generate the processing results of the subprocess. For example:

[0244] final_result = merge_results(results)

[0245] S46 Close thread pool

[0246] After all tasks are completed, close the thread pool and release system resources:

[0247] 1. Shut down the thread pool: call the `shutdown` method of the thread pool and wait for all thread tasks to complete before shutting down. For example:

[0248] thread_pool.shutdown(wait=True)

[0249] Example

[0250] The following is an example of using a thread pool to process data in a complete child process:

[0251] from concurrent.futures import ThreadPoolExecutor

[0252] import threading

[0253] # Define shared resources

[0254] shared_data = []

[0255] lock = threading.Lock()

[0256] #Thread task function

[0257] def thread_task(data_segment):

[0258] global shared_data

[0259] # Specific data processing logic

[0260] processed_data = perform_computation(data_segment)

[0261] # Thread-safe access to shared resources

[0262] with lock:

[0263] shared_data.append(processed_data)

[0264] return processed_data

[0265] # Subprocess task function

[0266] def process_data(data_block):

[0267] # Initialize the thread pool

[0268] number_of_threads = 10

[0269] thread_pool = ThreadPoolExecutor(max_workers=number_of_threads)

[0270] # Data partitioning

[0271] data_segments = split_data(data_block, number_of_threads)

[0272] # Submit a task

[0273] futures = [thread_pool.submit(thread_task, segment) for segmentin data_segments]

[0274] # Collect results

[0275] results = [future.result() for future in futures]

[0276] # Merge results

[0277] final_result = merge_results(results)

[0278] # Close the thread pool

[0279] thread_pool.shutdown(wait=True)

[0280] return final_result

[0281] Through detailed multi-thread creation steps, task allocation can be further refined in each sub-process, improving the parallelism and efficiency of data processing, making full use of system resources, and accelerating the big data processing process.

[0282] In big data processing systems, task allocation and scheduling are key steps to ensure efficient data processing. Through a reasonable task allocation and scheduling mechanism, the parallel processing capabilities of multi-process and multi-thread can be fully utilized to significantly improve data processing speed. The detailed steps are as follows:

[0283] S51 data block division

[0284] The main process first needs to divide the large data set to be processed into blocks so that it can be assigned to each subprocess for processing:

[0285] 1. Data loading: Load the data set to be processed from the data source to ensure that the data is complete and can be read smoothly. For example:

[0286] data = load_data(source)

[0287] 2. Data preprocessing: Perform necessary preprocessing on the loaded data, such as deduplication and cleaning, to ensure data quality.

[0288] processed_data = preprocess_data(data)

[0289] 3. Data partitioning: Divide the data into several blocks according to the data volume and the number of sub-processes. Each data block will be used as an independent processing unit.

[0290] data_blocks = split_data(processed_data, num_processes)

[0291] S52 Task Queue Creation

[0292] The main process creates a task queue to store and schedule pending tasks:

[0293] 1. Task queue initialization: Create a task queue object to store data blocks and their processing methods.

[0294] from multiprocessing import Queue

[0295] task_queue = Queue()

[0296] 2. Task Enqueue: Add the divided data blocks as tasks to the task queue. Each task contains a data block and its processing function.

[0297] for block in data_blocks:

[0298] task_queue.put((process_data_block, block))

[0299] S53 Subprocess task allocation

[0300] The main process takes tasks from the task queue and assigns them to each subprocess for processing:

[0301] 1. Process pool creation: Create a process pool object to manage multiple child processes.

[0302] from multiprocessing import Pool

[0303] process_pool = Pool(processes=num_processes)

[0304] 2. Task allocation: Assign tasks in the task queue to child processes in the process pool. The child processes take tasks from the task queue and execute them.

[0305] results = [process_pool.apply_async(execute_task, (task_queue,))for _ in range(num_processes)]

[0306] 3. Execute task function: define a function, and the child process takes the task from the task queue and executes it.

[0307] def execute_task(task_queue):

[0308] while not task_queue.empty():

[0309] task, data_block = task_queue.get()

[0310] task(data_block)

[0311] S54 Task Scheduling and Load Balancing

[0312] The main process needs to monitor the running status of the child process and perform task scheduling and load balancing:

[0313] 1. Monitor sub-process status: monitor the running status of each sub-process in real time, including whether the process is running normally, task execution progress, etc.

[0314] import time

[0315] while any([result.ready() for result in results]):

[0316] time.sleep(1)

[0317] 2. Dynamic task scheduling: dynamically adjust the task allocation strategy according to the load of the sub-process to ensure balanced distribution of tasks

[0318] def dynamic_task_allocation(task_queue, process_pool):

[0319] while not task_queue.empty():

[0320] for process in process_pool._pool:

[0321] if process.ready():

[0322] task, data_block = task_queue.get()

[0323] process_pool.apply_async(task, (data_block,))

[0324] 3. Fault handling: When a subprocess is detected to have exited abnormally or a task has failed, the main process should restart the subprocess in a timely manner and reallocate the unfinished tasks.

[0325] for result in results:

[0326] if not result.successful():

[0327] failed_task, failed_data_block = result.get()

[0328] task_queue.put((failed_task, failed_data_block))

[0329] S55 thread task allocation

[0330] After receiving the data block, the child process further divides it into smaller data segments and assigns them to each thread in the thread pool:

[0331] 1. Thread pool creation: Create a thread pool object in the child process to manage multiple threads.

[0332] from concurrent.futures import ThreadPoolExecutor

[0333] thread_pool = ThreadPoolExecutor(max_workers=num_threads)

[0334] 2. Data segmentation: Divide the data blocks received by the child process into smaller data segments so that threads can process them in parallel.

[0335] data_segments = split_data(data_block, num_threads)

[0336] 3. Thread task allocation: Allocate data segments to each thread in the thread pool and use the `submit` method to submit tasks asynchronously.

[0337] futures = [thread_pool.submit(process_segment, segment) forsegment in data_segments]

[0338] 4. Thread task execution function: define a function, and the thread executes specific data processing logic.

[0339] def process_segment(data_segment):

[0340] # Specific data processing logic

[0341] processed_result = compute(data_segment)

[0342] return processed_result

[0343] S56 Results Collection and Merging

[0344] After the subprocesses and threads complete their tasks, the main process needs to collect the processing results of each subprocess and merge them:

[0345] 1. Thread result collection: Collect the processing results of each thread through the `futures` object.

[0346] thread_results = [future.result() for future in futures]

[0347] 2. Subprocess result collection: The main process obtains the processing results from the subprocess through an inter-process communication mechanism (such as a queue or pipe).

[0348] process_results = [result.get() for result in results]

[0349] 3. Result merging: Integrate the results returned by each sub-process and thread to generate the final processing result.

[0350] final_result = merge_all_results(process_results)

[0351] Through detailed task allocation and scheduling steps, the main process and sub-processes can work together efficiently, fully utilize system resources, accelerate the big data processing process, and ensure processing speed and efficiency.

[0352] Result merging is a key step in the big data processing system, ensuring that after all subprocesses and threads complete their tasks, the main process can effectively collect, integrate and output the final processing results. The detailed steps are as follows:

[0353] S61 Subprocess result collection

[0354] After completing the task, each subprocess needs to return the processing result to the main process:

[0355] 1. Inter-process communication mechanism: Use an inter-process communication mechanism (such as queues, pipes, or shared memory) to transfer the processing results of the child process. For example, using a queue:

[0356] from multiprocessing import Queue

[0357] result_queue = Queue()

[0358] 2. Subprocess result storage: After the subprocess completes the task, the processing result is stored in the shared queue:

[0359] def process_data_block(data_block, result_queue):

[0360] processed_result = perform_computation(data_block)

[0361] result_queue.put(processed_result)

[0362] 3. The main process collects the results: The main process collects the processing results of each subprocess from the shared queue:

[0363] process_results = []

[0364] while not result_queue.empty():

[0365] process_results.append(result_queue.get())

[0366] S62 thread result collection

[0367] After the thread inside the child process completes the task, the processing results need to be summarized in the child process:

[0368] 1. Thread pool result collection: collect the processing results of each thread through the `futures` object:

[0369] 2. Subprocess result return: The subprocess returns the thread summary results to the main process through the inter-process communication mechanism:

[0370] def process_data_block(data_block, result_queue):

[0371] thread_results = process_data_block(data_block)

[0372] result_queue.put(thread_results)

[0373] S63 Result merging strategy

[0374] The main process needs to integrate the collected results of each sub-process and adopt a reasonable merging strategy:

[0375] 1. Sequential merging: Merge the results in the order in which the tasks are executed. For example, if the data processing task is performed in blocks, the results can be merged by block:

[0376] final_result = []

[0377] for result in process_results:

[0378] final_result.extend(result)

[0379] 2. Deduplication and merging: Deduplication is performed on the results to ensure that the final output does not contain duplicate data:

[0380] unique_results = list(set(final_result))

[0381] 3. Aggregate calculation: perform aggregate calculations on the results, such as average and sum, which is suitable for tasks that require statistical analysis:

[0382] aggregated_result = sum(final_result) / len(final_result)

[0383] 4. Sorting and merging: Sort and merge the results, suitable for tasks that require sorted output:

[0384] sorted_result = sorted(final_result)

[0385] S64 Result Output

[0386] Output the final processing results to the specified storage medium or system:

[0387] 1. Store to file: Write the final result to a file and save it to disk:

[0388] with open('output.txt', 'w') as f:

[0389] for item in final_result:

[0390] f.write("%s\n" % item)

[0391] 2. Write to database: insert the results into the database for subsequent query and analysis:

[0392] import sqlite3

[0393] conn = sqlite3.connect('results.db')

[0394] cursor = conn.cursor()

[0395] cursor.executemany('INSERT INTO results (data) VALUES (?)',[(item,) for item in final_result])

[0396] conn.commit()

[0397] conn.close()

[0398] S65 Error Handling and Logging

[0399] During the result merging process, the main process needs to handle possible errors and record relevant logs:

[0400] 1. Error handling: Capture and handle exceptions that may occur during the merging process, such as data format errors, connection failures, etc.:

[0401] try:

[0402] final_result = merge_all_results(process_results)

[0403] except Exception as e:

[0404] print(f"Error occurred during result merging: {e}")

[0405] 2. Logging: Record important events and error information during processing to facilitate subsequent debugging and maintenance: import logging

[0406] logging.basicConfig(filename='process.log', level=logging.INFO)

[0407] logging.info('Results merged successfully')

[0408] Through detailed result merging steps, the main process can effectively collect and integrate the processing results of each subprocess and thread, ensuring the accuracy and completeness of the final output results, thereby improving the overall efficiency and reliability of the big data processing system. It also increases the external service capabilities of elasticsearch.

[0409] According to a specific implementation of the embodiment of the present disclosure, the feature extraction of elasticsearch data to obtain a data feature set K corresponding to the elasticsearch data includes:

[0410] The data volume, document complexity, query frequency, index update rate, query complexity and number of concurrent users of elasticsearch data are calculated respectively to obtain the data volume feature Kvo, document complexity feature Kco, query frequency feature Kqu, index update rate feature Kud, query complexity feature Kqc and concurrent user number feature Kcr, and then the data feature set K={Kvo,Kco,Kqu,Kud,Kqc,Kcr} corresponding to the elasticsearch data is obtained.

[0411] According to a specific implementation of the embodiment of the present disclosure, the step of calculating the task requirements required for elasticsearch data processing according to the data feature set K and initializing the running resources Z matching the task requirements using the main process includes:

[0412] Construct and run the resource calculation function Z = f(Kvo, Kco, Kqu, Kud, Kqc, Kcr) to calculate the storage requirement Se, computing resource requirement Ce, and memory requirement Me required for elasticsearch data processing, where: , Rr is the replication factor, and Bf is the index buffer size.

[0413] According to a specific implementation of the embodiment of the present disclosure, the determining the number n of sub-processes that need to be started based on the running resource Z includes:

[0414] The calculation formula for building the number of subprocesses n is:

[0415]

[0416] Indicates the number of CPU cores available in the system. Indicates the number of CPU cores that each child process is expected to occupy, taking into account the CPU intensity of the task. Indicates the amount of memory available in the system. Indicates the expected memory size of each child process. represents the total time budget to complete all tasks, It represents the expected time for a single subprocess to process a single task unit, and NL represents the theoretical maximum parallelism of the task. Indicates the system reserved resource factor.

[0417] According to a specific implementation of an embodiment of the present disclosure, the processing performance of the current device processing elasticsearch data is evaluated to obtain a performance evaluation result P={Pu, Pm, Pn, Pr}, and the computational complexity of the segmented data set D={D1, D2, ...Dn} is evaluated to obtain a computing power evaluation result S={S1, S2, ...Sn}, including:

[0418] Build performance evaluation result calculation formula:

[0419]

[0420] and are the first adjustment factor and the second adjustment factor respectively.

[0421] According to a specific implementation of an embodiment of the present disclosure, the processing performance of the current device processing elasticsearch data is evaluated to obtain a performance evaluation result P={Pu, Pm, Pn, Pr}, and the computational complexity of the segmented data set D={D1, D2, ...Dn} is evaluated to obtain a computing power evaluation result S={S1, S2, ...Sn}, and further includes:

[0422] Obtain the time complexity and space complexity Ci of the algorithm used for data Di, as well as the hardware performance H of the current device, to form an evaluation result Si for data Di:

[0423]

[0424] and is the adjustment coefficient, γ is the index of the impact of data scale on complexity, and β is the index of the relief of hardware performance on computing requirements.

[0425] According to a specific implementation of the embodiment of the present disclosure, the calculation of the number of threads X={X1, X2, ...Xn} to be allocated in n sub-processes based on the performance evaluation result P, the computing power evaluation result S and the balancing function J includes:

[0426] Construct a calculation function to calculate the number of calculation threads Xi:

[0427]

[0428] Qi is an adjustment factor for the i-th child process.

[0429] According to a specific implementation of the embodiment of the present disclosure, the calculation of the set of thread numbers X={X1, X2, ...Xn} to be allocated in n sub-processes based on the performance evaluation result P, the computing power evaluation result S and the balancing function J also includes:

[0430] Construct equilibrium function J:

[0431]

[0432] is the weight parameter, τ and λ are adjustment parameters.

[0433] Corresponding to the above method embodiment, see Figure 3 The embodiment of the present invention further discloses a device 30 for accelerating the processing of elasticsearch data in big data, comprising:

[0434] The extraction module 301 obtains the elasticsearch data to be processed, and then performs feature extraction on the elasticsearch data to obtain a data feature set K corresponding to the elasticsearch data;

[0435] The calculation module 302 calculates the task requirements required for elasticsearch data processing according to the data feature set K, and uses the main process to initialize the running resources Z that match the task requirements;

[0436] The determination module 303 determines the number n of sub-processes to be started based on the running resource Z. The n sub-processes are started by the main process. After the n sub-processes are started, the main process divides the elasticsearch data into a segmented data set D={D1, D2, ...Dn} containing n data segments, and injects the n data segments into each independently running sub-process.

[0437] Evaluation module 304 evaluates the processing performance of the current device processing elasticsearch data to obtain a performance evaluation result P={Pu, Pm, Pn, Pr}, and performs computational complexity evaluation on the segmented data set D={D1, D2, ...Dn} to obtain a computing power evaluation result S={S1, S2, ...Sn}, where Pu represents the process computing performance of the current device, Pm represents the storage response performance of the current device, Pu represents the network transmission performance of the current device, and Pu represents the process switching performance of the current device;

[0438] The execution module 305 calculates the set of thread numbers X={X1,X2,…Xn} that need to be allocated in the n sub-processes based on the performance evaluation result P, the computing power evaluation result S and the balancing function J, and creates multiple threads in each sub-process based on the thread number set X. After each sub-process completes the data calculation task, the calculation result is returned to the main process, and the main process integrates and outputs the calculation results.

[0439] See also Figure 4 The embodiment of the present invention further provides an electronic device 60, the electronic device comprising:

[0440] at least one processor; and,

[0441] a memory communicatively connected to the at least one processor; wherein,

[0442] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for accelerating the processing of elasticsearch data in big data in the aforementioned method embodiment.

[0443] An embodiment of the present invention further provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the aforementioned method embodiment.

[0444] An embodiment of the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the method for accelerating the processing of elasticsearch data in big data in the aforementioned method embodiment.

[0445] Reference below Figure 4 , which shows a schematic diagram of the structure of an electronic device 60 suitable for implementing the embodiment of the present disclosure. The electronic device in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 4 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0446] like Figure 4 As shown, the electronic device 60 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 60 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0447] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, a gage, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 60 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 The electronic device 60 is shown with various devices, but it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have instead.

[0448] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0449] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for accelerating the processing of elasticsearch data in big data, characterized in that: include: After obtaining the elasticsearch data to be processed, feature extraction is performed on the elasticsearch data to obtain a data feature set K corresponding to the elasticsearch data; Calculate the task requirements required for elasticsearch data processing according to the data feature set K, and use the main process to initialize the running resources Z that match the task requirements; Based on the running resource Z, the number n of sub-processes to be started is determined. The n sub-processes are started by the main process. After the n sub-processes are started, the main process divides the elasticsearch data into a segmented data set D={D1, D2, ...Dn} containing n data segments, and injects the n data segments into each independently running sub-process; Evaluate the processing performance of the current device that processes elasticsearch data, and obtain the performance evaluation result P = {Pu, Pm, Pn, Pr}. Evaluate the computational complexity of the segmented data set D = {D1, D2, ... Dn}, and obtain the computing power evaluation result S = {S1, S2, ... Sn}, where Pu represents the process computing performance of the current device, Pm represents the storage response performance of the current device, Pn represents the network transmission performance of the current device, and Pr represents the process switching performance of the current device. Based on the performance evaluation result P, the computing power evaluation result S and the balancing function J, the set of thread numbers X = {X1, X2, ...Xn} that need to be allocated in the n sub-processes is calculated, and multiple threads are created in each sub-process based on the thread number set X. After each sub-process completes the data calculation task, the calculation result is returned to the main process, and the main process integrates and outputs the calculation results.

2. The method according to claim 1, characterized in that The feature extraction of elasticsearch data to obtain a data feature set K corresponding to the elasticsearch data includes: The data volume, document complexity, query frequency, index update rate, query complexity and number of concurrent users of elasticsearch data are calculated respectively to obtain the data volume feature Kvo, document complexity feature Kco, query frequency feature Kqu, index update rate feature Kud, query complexity feature Kqc and concurrent user number feature Kcr, and then the data feature set K = {Kvo, Kco, Kqu, Kud, Kqc, Kcr} corresponding to the elasticsearch data is obtained.

3. The method according to claim 2, characterized in that The step of calculating the task requirements required for elasticsearch data processing according to the data feature set K, and initializing the running resources Z matching the task requirements using the main process, includes: Construct the running resource calculation function Z = f(Kvo, Kco, Kqu, Kud, Kqu, Kqc, Kcr) to calculate the storage requirement Se, computing resource requirement Ce and memory requirement Me required for elasticsearch data processing, where Se = Kvo × (1 + R r )+B f , Rr is the replication factor, and Bf is the index buffer size.

4. The method according to claim 3, characterized in that The determining the number n of sub-processes to be started based on the running resource Z includes: The calculation formula for building the number of subprocesses n is: Cores avail Indicates the number of CPU cores available in the system, C cpu Indicates the number of CPU cores that each child process is expected to occupy, taking into account the CPU intensity of the task, Memory avail Indicates the available memory size of the system, M mem Indicates the expected memory size of each child process, T total represents the total time budget to complete all tasks, T task It represents the expected time for a single subprocess to process a single task unit, NL represents the theoretical maximum parallelism of the task, and Z reserve Indicates the system reserved resource factor.

5. The method according to claim 4, characterized in that The processing performance of the current device processing elasticsearch data is evaluated to obtain a performance evaluation result P = {Pu, Pm, Pn, Pr}, and the computational complexity of the segmented data set D = {D1, D2, ...Dn} is evaluated to obtain a computing power evaluation result S = {S1, S2, ...Sn}, including: Build performance evaluation result calculation formula: α1 and α2 are the first regulating factor and the second regulating factor, respectively.

6. The method according to claim 5, characterized in that The processing performance of the current device processing elasticsearch data is evaluated to obtain a performance evaluation result P = {Pu, Pm, Pn, Pr}, and the computational complexity of the segmented data set D = {D1, D2, ... Dn} is evaluated to obtain a computing power evaluation result S = {S1, S2, ... Sn}, and further includes: Obtain the time complexity and space complexity Ci of the algorithm used for data Di, as well as the hardware performance H of the current device, to form an evaluation result Si for data Di: Si=k1·|Di| γ ·C i (H)+k2·H -β k1 and k2 are adjustment coefficients, γ is the index of the impact of data scale on complexity, and β is the index of the relief of hardware performance on computing requirements.

7. The method according to claim 6, characterized in that The method of calculating the number of threads X={X1, X2, ...Xn} to be allocated in n sub-processes based on the performance evaluation result P, the computing power evaluation result S and the balancing function J includes: Construct a calculation function to calculate the number of calculation threads Xi: Qi is an adjustment factor for the i-th child process.

8. The method according to claim 7, characterized in that The method of calculating the set of thread numbers X={X1, X2, ...Xn} to be allocated in n sub-processes based on the performance evaluation result P, the computing power evaluation result S and the balancing function J also includes: Construct equilibrium function J: ω1, ω2, ω3, ω4 are weight parameters, and τ and λ are adjustment parameters.

9. A device for accelerating the processing of elasticsearch data in big data, characterized in that: include: The extraction module obtains the elasticsearch data to be processed, and then extracts features from the elasticsearch data to obtain a data feature set K corresponding to the elasticsearch data; The calculation module calculates the task requirements required for elasticsearch data processing according to the data feature set K, and uses the main process to initialize the running resources Z that match the task requirements; A determination module determines the number n of sub-processes to be started based on the running resource Z, wherein the n sub-processes are started by the main process. After the n sub-processes are started, the main process divides the elasticsearch data into a segmented data set D={D1, D2, ...Dn} containing n data segments, and injects the n data segments into each independently running sub-process; The evaluation module evaluates the processing performance of the current device that processes elasticsearch data, and obtains the performance evaluation result P = {Pu, Pm, Pn, Pr}. It also evaluates the computational complexity of the segmented data set D = {D1, D2, ...Dn}, and obtains the computing power evaluation result S = {S1, S2, ...Sn}, where Pu represents the process computing performance of the current device, Pm represents the storage response performance of the current device, Pn represents the network transmission performance of the current device, and Pr represents the process switching performance of the current device. The computing module calculates the set of thread numbers X={X1,X2,…Xn} that need to be allocated in n sub-processes based on the performance evaluation result P, the computing power evaluation result S and the balancing function J, and creates multiple threads in each sub-process based on the thread number set X. After each sub-process completes the data computing task, the computing result is returned to the main process, which integrates and outputs the computing result.

10. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for accelerating the processing of elasticsearch data in big data as described in any one of claims 1 to 8.

Images