Data processing method and apparatus

Abstract

The application provides a data processing method and device, wherein the data processing method comprises: creating a data segment sequence corresponding to a target task in an execution state; loading a first data segment in the data segment sequence to a first circular array; reading and processing the first data segment in the first circular array according to a loading result, and loading a second data segment adjacent to the first data segment to a second circular array, the second data segment in the second circular array being processed after the first data segment in the first circular array is processed; loading a third data segment adjacent to the second data segment in the data segment sequence to the first circular array when the second data segment in the second circular array is in a processing state; and switching the target task to a completion state until all data segments in the data segment sequence are processed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a data processing method. The present application also relates to a data processing device, a computing device and a computer readable storage medium. BACKGROUND

[0002] In the prior art, in order to improve the processing efficiency, the data in the file is loaded into the memory queue in advance during the process of reading and processing the file, and then the data is read from the queue in sequence for processing. If the data amount of the file is large, a large-capacity memory space needs to be opened at one time, which may easily cause the problem of insufficient memory. Therefore, there is an urgent need to provide a solution to the above problem. SUMMARY

[0003] Therefore, the embodiments of the present application provide a data processing method. The present application also relates to a data processing device, a computing device and a computer readable storage medium to solve the problem of insufficient memory capacity in the prior art.

[0004] According to a first aspect of the embodiments of the present application, a data processing method is provided, comprising:

[0005] creating a data segment sequence corresponding to a target task in an execution state;

[0006] loading a first data segment in the data segment sequence to a first circular array;

[0007] reading and processing the first data segment in the first circular array according to the loading result, and loading a second data segment adjacent to the first data segment to a second circular array, wherein the second data segment in the second circular array is processed after the first data segment in the first circular array is processed;

[0008] loading a third data segment adjacent to the second data segment in the data segment sequence to the first circular array when the second data segment in the second circular array is in a processing state;

[0009] switching the target task to a completion state until the data segments contained in the data segment sequence are all processed.

[0010] According to a second aspect of the embodiments of the present application, a data processing device is provided, comprising:

[0011] a creating module configured to create a data segment sequence corresponding to a target task in an execution state;

[0012] a first loading module configured to load a first data segment in the data segment sequence to a first circular array;

[0013] a reading module configured to read and process the first data segment in the first circular array according to a loading result, and load a second data segment adjacent to the first data segment to a second circular array, wherein the second data segment in the second circular array is processed after the first data segment in the first circular array is processed;

[0014] a second loading module configured to load a third data segment adjacent to the second data segment in the data segment sequence to the first circular array when the second data segment in the second circular array is in a processing state;

[0015] a switching module configured to switch the target task to a completed state until all data segments in the data segment sequence are processed.

[0016] According to a third aspect of the embodiments of the present application, a computing device is provided, which includes a memory, a processor, and computer instructions stored in the memory and executable on the processor, and the processor implements the steps of the data processing method when executing the computer instructions.

[0017] According to a fourth aspect of the embodiments of the present application, a computer readable storage medium is provided, which stores computer instructions, and the computer instructions implement the steps of the data processing method when executed by a processor.

[0018] The data processing method provided by the present application, the data processing method provided by the present application, by creating a data segment sequence corresponding to a target task in an execution state, loading a first data segment in the data segment sequence to a first circular array, that is, by data segments, batch processing of data is realized, then according to the loading result, the first data segment in the first circular array is read and processed, and the second data segment adjacent to the first data segment is loaded to the second circular array, wherein the second data segment in the second circular array is processed after the first data segment in the first circular array is processed, and the third data segment adjacent to the second data segment in the data segment sequence is loaded to the first circular array when the second data segment in the second circular array is in a processing state, that is, the first circular array and the second circular array are alternately loaded with data segments, and alternately provide data that needs to be read; until all data segments in the data segment sequence are processed, the target task is switched to a completed state, which realizes that a large amount of memory space does not need to be opened at one time in the data processing process, and the occupation of memory space is reduced. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 is a schematic diagram of a data processing method provided by an embodiment of the present application;

[0020] Figure 2 is a flowchart of a data processing method provided by an embodiment of the present application;

[0021] Figure 3 is a schematic diagram of a loop array and a stress testing scenario in a data processing method provided by an embodiment of the present application;

[0022] Figure 4 is a processing flowchart of a data processing method applied to a stress testing scenario provided by an embodiment of the present application;

[0023] Figure 5 is a structural schematic diagram of a data processing apparatus provided by an embodiment of the present application;

[0024] Figure 6 is a structural block diagram of a computing device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0025] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, the present application can be practiced without the specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to obscure the present application. Some portions of the detailed description which follow are presented in terms of algorithms, symbolic representations of operations on data bits or binary digital signals stored within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art.

[0026] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of one or more embodiments of the present application. As used in one or more embodiments of the present application and the accompanying claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in one or more embodiments of the present application, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0027] It will be understood that, although the terms first, second, etc. can be used herein to describe various information, these terms are not intended to denote a temporal or chronological order. Rather, these terms are used solely to distinguish one from another. For example, a first entity discussed below could be termed a second entity, and, similarly, a second entity could be termed a first entity without departing from the scope of one or more embodiments of the present application. As used herein, the term "if' can be construed to mean "when" or "in response to determining" or "in response to a determination" or "in response to the fact that...."

[0028] First, the noun terms related to one or more embodiments of the present application are explained.

[0029] Thread: The smallest unit of operation that a system can schedule. It is contained within a process and is the actual unit of execution of a process. A thread is a single sequence of control within a process, and a process may have multiple threads of control that act in parallel.

[0030] Array: An array is a linear data structure that stores data sequentially.

[0031] Circular array: An array is considered circular when the next element of the last element of the array is the first element of the array.

[0032] Alternating double circular array: An alternating double circular array is an array that is internally composed of two circular arrays that work alternately. When one of them is providing data externally, the other loads data and exchanges roles with the previous array at the appropriate time, and so on.

[0033] Queue: A queue is a data structure that follows the first-in, first-out model.

[0034] Thread safety: Thread safety is a concept in multi-threaded programming. In a program with multiple threads that share data and execute in parallel, thread-safe code ensures that each thread can execute normally and correctly through synchronization mechanisms, without unexpected situations such as data pollution.

[0035] Stress: Stress is a concept in stress testing, which refers to sending requests to a specified service according to a certain target pressure (such as the number of requests per second) and obtaining a return.

[0036] Traffic: Refers to the various forms of network requests received by a service.

[0037] QPS (Queries-Per-Second): A metric for measuring the size of traffic.

[0038] Container: A container is a running instance created from an image. A container can be understood as an independent running environment that includes code, dependencies required for running, etc.

[0039] Memory: Memory (Memory) is also known as internal memory and main memory, and its function is to temporarily store operation data in the CPU and exchange data with external storage such as hard disks.

[0040] In the present application, a data processing method is provided, and the present application also relates to a data processing device, a computing device, and a computer readable storage medium, which are described in detail one by one in the following embodiments.

[0041] Reference Figure 1As shown in the schematic diagram, the data processing method provided in the application creates a data segment sequence corresponding to a target task in an execution state, the data segment sequence contains m data segments, loads a first data segment in the data segment sequence to a first circular array, that is, the data is processed in batches through data segments, then reads and processes the first data segment in the first circular array according to the loading result, and loads a second data segment adjacent to the first data segment to a second circular array, wherein the second data segment in the second circular array is processed after the first data segment in the first circular array is processed, under the condition that the second data segment in the second circular array is in a processing state, a third data segment adjacent to the second data segment in the data segment sequence is loaded to the first circular array, that is, the first circular array and the second circular array are used to alternately load data segments and alternately provide data to be read; until the data segments contained in the data segment sequence are all processed, the target task is switched to a completion state, which realizes that a large amount of memory space does not need to be opened at one time in the data processing process, and the occupation of the memory space is reduced.

[0042] Figure 2 A flowchart of a data processing method according to an embodiment of the application is shown, and specifically includes the following steps:

[0043] Step 202: Create a data segment sequence corresponding to a target task in an execution state.

[0044] Specifically, the target task refers to any task that needs to be processed, which can be a stress testing task, a data transmission task, a program execution task, etc., which is not limited here. The data segment sequence refers to a segment sequence created by combining data segments according to their arrangement order in the data file. The data segment refers to the data obtained by segmenting the data in the data file according to the preset segmentation rule.

[0045] It should be noted that the core of the data processing method provided in the application is data processing, and the process of data processing is basically the same for different target tasks. This embodiment is described by taking the target task as a stress testing task for convenience, and the process of data processing of other target tasks can be referred to the same or corresponding description content of this embodiment, which will not be described in detail here.

[0046] In view of the target task when task processing is performed on the data file, if the data in the entire data file is directly loaded into the memory, a queue with a large enough capacity is usually needed to load the content in the data file into the memory to ensure that the data can be obtained when the target task is executed. Therefore, in order to reduce the occupation of the memory space, before the data segment sequence corresponding to the target task in the execution state is created, the data file is divided into at least one data segment, which is specifically as follows:

[0047] The target task instruction is received, and the target task is created and started according to the target task information carried in the target task instruction.

[0048] Correspondingly, the data segment sequence corresponding to the target task in the execution state comprises:

[0049] The file to be processed associated with the target task is obtained.

[0050] The data contained in the file to be processed is segmented and processed according to the number of elements corresponding to the first and second cyclic arrays respectively, to obtain a data segment sequence composed of data segments.

[0051] Specifically, the target task instruction is an instruction for instructing to create a target task. Correspondingly, the target task information is information carried in the target task instruction, which can be task parameters, identification information and / or task program code of the target task. The file to be processed is a file to be processed by the target task, that is, a data file to which the data segment belongs. The first and second cyclic arrays are cyclic arrays created in advance. Specifically, as shown in (a) of Figure 3 The cyclic array includes k elements, the first element is element 1, and the last element is element k. After the last element k is read, the first element is returned to start the next round of reading. The first and second cyclic arrays are used to load the data in the file to be processed and provide the loaded data to the outside. The number of elements refers to the number of array elements in the cyclic array. For example, if the cyclic array includes 50 array elements, the number of elements is 50.

[0052] Based on this, when the target task instruction is received, it means that the target task needs to be created. The target task is created according to the target task information carried in the target task instruction. In the specific implementation, when the target task information carried is the task identification, the execution file corresponding to the target task can be found according to the identification information of the target task, and then the program code in the execution file is called to create the target task and start it. In addition, when the target task information carried is the task program code, the task program code of the target task can also be directly called to create the target task and start it.

[0053] On the basis of creating and starting the target task, the to-be-processed file can be located and obtained according to the file information contained in the target task, such as file identification, file path, etc. Further, the element quantity corresponding to the first loop array and the element quantity corresponding to the second loop array are taken as the segmentation basis to segment the to-be-processed file.

[0054] In the specific segmentation process, each row in the to-be-processed file can correspond to an array element. Assuming that the element quantity of the first loop array is 50 and the element quantity corresponding to the second loop array is 60, the first 50 rows in the to-be-processed file are divided into a data segment, and the 51st-110th rows in the to-be-processed file are divided into a data segment, and the 111th-160th rows in the to-be-processed file are divided into another data segment, and so on. The to-be-processed file is segmented alternately according to the element quantity corresponding to the first loop array and the element quantity corresponding to the second loop array. In addition, an equal number of characters in the to-be-processed file can correspond to an array element, such as 100 characters corresponding to an array element. In the case of the element quantity of the first loop array being 50, the first 100*50=5000 characters in the to-be-processed file are divided into a data segment; in the case of the element quantity of the first loop array being 60, the 5001st-100*60=6000th characters in the to-be-processed file are divided into a data segment, and so on. The to-be-processed file is segmented alternately according to the element quantity corresponding to the first loop array and the element quantity corresponding to the second loop array.

[0055] The segmented data segments are then arranged in the order of the to-be-processed file to form a data segment sequence. In actual application, in order to simplify the task execution process, the element quantity corresponding to the first loop array and the element quantity corresponding to the second loop array are equal.

[0056] Taking the target task as a stress test task as an example, the stress test task instruction is received, the stress test task is created and started according to the task parameters carried in the stress test task instruction, and the stress test file D associated with the stress test task is obtained by executing the stress test task. The number of rows of the stress test file D is 50,000, the element quantity corresponding to the loop array A and the loop array B is 10,000, the data contained in the to-be-processed file D is segmented to obtain 5 data segments: data segment 1, data segment 2, data segment 3, …, data segment m, and the number of rows of each data segment is 10,000. The m data segments are arranged in the order of the to-be-processed file D to form a data segment sequence, as shown in (b) of FIG. 1. Figure 3

[0057] ​In summary, in the case of receiving a target task instruction, the target task is created and started, which increases the flexibility of creating the target task. According to the number of elements corresponding to the first loop array and the second loop array, the to-be-processed file is segmented and processed, so that the obtained data segment can match the storage capacity of the first loop array or the second loop array, avoiding the situation of insufficient capacity or excessive capacity when the data segment is loaded into the first loop array or the second loop array, and improving the utilization rate of resources.

[0058] Step 204: loading the first data segment in the data segment sequence into the first loop array.

[0059] Specifically, based on the above-mentioned creation of the data segment sequence, in order to start data processing on the data segments in the data segment sequence from the beginning, the first data segment in the data segment sequence is loaded into the first loop array. The first data segment refers to the data segment arranged in the first position in the data segment sequence.

[0060] Based on this, loading the first data segment into the first loop array is based on the pre-set corresponding rule of the data in the first data segment and the data elements in the first loop array, and the data in the first data segment is loaded into the corresponding position in the first loop array.

[0061] Considering that each row in the to-be-processed file may have a separate processing value, the data row is taken as the basic unit of data loading and corresponds to the data element in the loop array. In this embodiment, the loading of the first data segment in the data segment sequence into the first loop array is specifically implemented as follows:

[0062] Obtain the data row in the first data segment in the data segment sequence, and load the obtained data row as a data element into the first loop array.

[0063] Specifically, the data row refers to the data corresponding to each row in the data segment. In specific implementation, the data row can be obtained from the first data segment in sequence, and then the obtained data row is loaded into the first loop array according to the arrangement order of the data elements in the first loop array.

[0064] Taking the above-mentioned data segment 1 as an example, the data segment 1 contains 10,000 data rows, and each data row is loaded as a data element into the loop array A. The completed loop array A contains 10,000 non-empty data elements.

[0065] In summary, loading the data row as a data element into the first loop array ensures the accuracy and rationality of data loading on the first data segment.

[0066] Step 206: reading and processing the first data segment in the first circular array according to the loading result, and loading a second data segment adjacent to the first data segment to a second circular array, wherein the second data segment in the second circular array is processed after the first data segment in the first circular array is processed.

[0067] Specifically, on the basis of loading the first data segment to the first circular array, considering that the data segment sequence also contains other data segments to be processed, in order to make the data segments in the data segment sequence be processed in order, in the embodiment, the data segments are alternately loaded and provided by the first circular array and the second circular array.

[0068] The second data segment refers to a data segment in the data segment sequence arranged after the first data segment and adjacent to the first data segment.

[0069] Based on this, on the basis of loading the first data segment, in order to improve the processing efficiency of the data, the loaded data can be directly read and processed. The specific processing method corresponds to the task target of the target task, which is not limited here. In order to improve the processing efficiency of the data, the second data segment can also be loaded to the second circular array in parallel during the process of reading and processing the first data segment. Then, after the first data segment is processed, the second data segment in the second circular array can be directly read and processed.

[0070] In addition, since the first circular array and the second circular array are empty before the first data segment is loaded to the first circular array, when the data is loaded for the first time, the second data segment can be loaded to the second circular array at the same time in the process of loading the first data segment to the first circular array.

[0071] Considering that if the number of data elements in the circular array is large, sequentially processing multiple data elements in the circular array will take a long time, in order to improve the efficiency of reading and processing the data elements in the circular array, in the embodiment, multiple threads can be started concurrently to process the data elements, which is specifically implemented as follows:

[0072] Starting at least two task threads corresponding to the target task;

[0073] Reading and processing the data elements in the first circular array by executing each task thread.

[0074] Specifically, a task thread refers to a thread used to read data elements and perform task processing based on the read data elements. After loading the first data segment into the first circular array, at least two task threads are started, and each task thread reads and processes the data elements in the first circular array. Since at least two task threads have been started, after loading the second data segment into the second circular array, the started task threads can continue to be executed to read and process the data elements in the second circular array.

[0075] Using the previous example, start the n load testing threads corresponding to the load testing task, and execute each load testing thread to read and process data elements in the loop array A until all data elements contained in the loop array A have been read and used to complete the load testing.

[0076] In summary, by concurrently starting at least two task threads to read and process the data elements in the first circular array, the processing efficiency of the first data segment is improved.

[0077] Furthermore, since the data elements in the circular array can be read repeatedly, in order to prevent the task thread from endlessly reading the data elements in the first circular array, this embodiment can record the number of data elements that the task thread has already read. The specific implementation is as follows:

[0078] Record the number of data elements read in the first circular array, wherein the number of read elements is used to detect the reading status of data elements in the first circular array.

[0079] Specifically, the number of elements read refers to the number of data elements read from the circular array. Correspondingly, the read status refers to whether any data element in the first circular array has been read. When the number of elements read equals the number of elements in the first circular array, it means that the data elements in the first circular array have been completely consumed (read). In practice, after each data element is read by the task thread, the number of elements read is incremented by one, indicating that a new data element has been read. Similarly, the number of data elements read in the second circular array can also be recorded.

[0080] Suppose that after the first pressure-generating thread reads a data element from the circular array A, it changes the number of elements read from the initial value of 0 to 1. After the second pressure-generating thread reads a data element from the circular array A, it changes the number of elements read from 1 to 2, and so on, until all 10,000 data elements of the circular array A have been read, at which point the number of elements read becomes 10,000.

[0081] In summary, by recording the read element number of the data elements read in the circular array, it can be determined not only whether all data elements in the circular array have been read, but also whether each data element has been read.

[0082] On the basis of the data elements in the first circular array being read, similar to the above-mentioned obtaining data rows in the first data segment and loading the obtained data rows as data elements into the first circular array, in the embodiment, the second data segment adjacent to the first data segment is loaded into the second circular array, and the specific implementation is as follows:

[0083] The second data segment adjacent to the first data segment in the data segment sequence is determined.

[0084] Obtaining data rows in the second data segment and loading the obtained data rows as data elements into the second circular array.

[0085] Specifically, the specific implementation of obtaining data rows in the second data segment and loading the obtained data rows as data elements into the second circular array is similar to the specific implementation of obtaining data rows in the first data segment and loading the obtained data rows as data elements into the first circular array. Reference can be made to the above-mentioned embodiment, which is not limited herein. It should be noted that the same loading method can be used in the loading process of each data segment.

[0086] Following the above example, the data segment 2 also contains 10,000 data rows, and each data row is loaded as a data element into the circular array B. The completed circular array B also contains 10,000 non-empty data elements.

[0087] In summary, loading data rows as data elements into the second circular array ensures the accuracy and rationality of data loading of the second data segment.

[0088] In order to clearly determine whether the data segment has been loaded, in the embodiment, after the first data segment in the data segment sequence is loaded into the first circular array, the completed circular array can be marked, and the specific implementation is as follows:

[0089] The first circular array is marked as a loading completion state;

[0090] Correspondingly, after the second data segment adjacent to the first data segment is loaded into the second circular array, it further includes:

[0091] The second circular array is marked as the loading completion state.

[0092] Correspondingly, the second data segment in the second circular array is processed after the first data segment in the first circular array is processed, comprising:

[0093] In a case where it is determined that the read element quantity corresponding to the first circular array is equal to the element quantity corresponding to the first circular array, and the second circular array is in the loading completion state, the data elements in the second circular array are read and processed by executing each task thread.

[0094] Specifically, the loading completion state refers to a state used to indicate that the circular array has completed loading of the data segment. By detecting whether the circular array is in the loading completion state, it can be determined whether the circular array has completed loading of the data.

[0095] Since the first circular array and the second circular array need to be switched to alternately provide data for the task threads, before switching to the second circular array, it is necessary to determine that the read element quantity corresponding to the first circular array is equal to the element quantity corresponding to the first circular array, i.e., to determine whether the data elements in the first circular array have been completely read. In addition, it is also necessary to determine that the second circular array is in the loading completion state, i.e., to determine that the second circular array has completed data loading. In both cases, the data elements in the second circular array are read and processed by executing each task thread.

[0096] Following the above example, the read element quantity is 10,000, which is equal to the element quantity of the data elements contained in the circular array A, so it is determined that the 10,000 data elements of the circular array A are all read, and it is determined that the circular array B is in the loading completion state, indicating that the circular array B can be switched to provide data for the task threads. Then, the data elements in the circular array B are read and processed by executing n compression threads.

[0097] In summary, by marking the circular array that has completed data loading as the loading completion state, and on the basis of determining that the data elements in the first circular array have all been read and the second circular array has completed data loading, the data elements in the second circular array are read and processed, which realizes safe switching of the first circular array and the second circular array to provide data for the task threads.

[0098] In addition, considering that there is also a scenario where the data elements in the first circular array have all been read, but the second circular array has not completed loading, in order to avoid the compression threads being in an idle state, in this embodiment, before the data elements in the second circular array are read and processed by executing each task thread, the data elements in the first circular array can be read and processed again, which is specifically implemented as follows:

[0099] In a case where it is determined that the read element quantity corresponding to the first circular array is equal to the element quantity corresponding to the first circular array, and the second circular array is in an unfinished loading state, the data elements in the first circular array are re-read and processed by each task thread until the second circular array is marked as the loading completed state.

[0100] The read element quantity corresponding to the first circular array is determined to be equal to the element quantity corresponding to the first circular array, that is, it is determined whether the data elements in the first circular array have been completely read. In addition, it is determined that the second circular array is in an unfinished loading state, indicating that the second circular array has not completed data loading, and then the task thread is continuously provided with data to be read by the first circular array, and after each data element is read, it is determined whether the second circular array is marked as the loading completed state; if not, the task thread continues to read the data provided by the first circular array; if yes, the task thread stops being provided with data to be read by the first circular array, and then switches to being provided with data to be read by the second circular array.

[0101] In the above example, on the basis of determining that all 10,000 data elements of circular array A have been read, and in a case where it is determined that circular array B is in an unfinished loading state, indicating that circular array B is not accurately ready, the task thread continues to be provided with data to be read from the first data element by circular array A until circular array B is marked as the loading completed state.

[0102] In summary, by using the recyclable characteristics of the circular array, in a case where the second circular array is not ready to provide data reading service, the first circular array continues to provide data reading service from the beginning, avoiding the task thread being in an idle state, that is, avoiding resource waste.

[0103] Step 208: In a case where the second data segment in the second circular array is in a processed state, a third data segment adjacent to the second data segment in the data segment sequence is loaded to the first circular array.

[0104] Specifically, on the basis of loading the second data segment to the second circular array, the second data segment in the second circular array is read and processed. Since the first data element in the first circular array has been processed when this process is started, in order to avoid the first circular array being idle, the third data segment can be loaded to the first circular array in the process of reading and processing the second data segment in the second circular array. The third data segment refers to the data segment arranged after the second data segment in the data segment sequence.

[0105] Before the third data segment is loaded into the first circular array, since the first data segment has been stored in the first circular array, in order to guarantee the accuracy and stability of loading the third data segment, in the embodiment, before the third data segment adjacent to the second data segment in the data segment sequence is loaded into the first circular array, the method further comprises:

[0106] Clearing the first data segment loaded in the first circular array.

[0107] Specifically, clearing the first data segment loaded in the first circular array means clearing the data value of each data element contained in the first circular array.

[0108] Continuing with the above example, after the n pressure threads are executed to continue reading the data elements in the circular array B and processing them, the loaded data segment 1 in the circular array A is cleared, and then the data segment 3 is loaded into the circular array A.

[0109] In summary, before the third data segment is loaded into the first circular array, the first data segment in the first circular array is cleared, which avoids data confusion and guarantees the reliable loading of the third data segment.

[0110] Step 210: Until the data segments contained in the data segment sequence are all processed, the target task is switched to a completed state.

[0111] Specifically, on the basis of loading the third data segment into the first circular array, it is indicated that the first circular array and the second circular array alternately perform data loading and provide data that needs to be read. That is, the first circular array and the second circular array form an alternating double circular array to perform data loading and provide data that needs to be read. Until all the data segments contained in the data segment sequence are processed, it is indicated that the target task is completed, and then the target task is switched to a completed state.

[0112] Continuing with the above example, on the basis of loading the data segment 3 into the circular array A, and determining that the data segment 2 in the circular array B is processed, then the n pressure threads are executed to read the data segment 3 in the circular array A, and the task processing is performed based on the read data segment 3, and the data segment 4 is loaded into the circular array B. In the case where it is determined that all the data segments 3 in the circular array A are read, and it is determined that the circular array B is in a loaded state, the n pressure threads are executed to read the data segment 4 in the circular array B, and the task processing is performed based on the read data segment 4. In this way, the circular array A and the circular array B are alternately used to load the data segments in the data segment sequence and provide data that needs to be read, until all the m data segments contained in the data segment sequence are processed. The specific pressure testing scene is as follows:Figure 3 (b) of the scenario, the stress release processing can be understood as sending a stress test request to the stress test service, and in the case that the m data segments contained in the data segment sequence are all processed, the stress test task is switched to the completion state.

[0113] In addition to the case that the data segment sequence contains at least three data segments, in the embodiment, in the case that the number of data segments contained in the data segment sequence is a first set number, the method further comprises:

[0114] loading the first data segment in the data segment sequence to the first circular array;

[0115] reading the first data segment in the first circular array according to the loading result and processing;

[0116] in the case that the first data segment is processed, the target task is switched to the completion state.

[0117] Specifically, the number of segments refers to the number of data segments contained in the data segment sequence. The first set number refers to the number of data segments that are set in advance, which can be 1. In the case that the number of segments is 1, the first data segment can be loaded and the data to be read can be provided through the first circular array. Therefore, after loading the first data segment to the first circular array, the first data segment is read and processed in the first circular array. After the first data segment is processed, the target task is completed, and the target task is switched to the completion state.

[0118] In the stress release scenario, by using the characteristics of the circular array, when the number of rows of the stress test file is less than the capacity (number of elements) of the circular array, the content of the stress test file only needs to be loaded into the circular array once to reside in the memory for subsequent circular use, and no longer needs to be repeatedly read from the file, improving the efficiency. In contrast, the queue needs to constantly read the content from the file to supplement the elements during the entire stress release process. In addition, repeated data no longer needs to be copied and loaded into the memory, but only needs to have one copy filled into the circular array, which can be continuously read, and the preparation of the stress test file can be de-duplicated, thereby reducing the preparation time of the stress test file.

[0119] For example, there is only one data segment 1 in the data segment sequence, the data segment 1 is loaded into the circular array A, then the data segment 1 is read in the circular array A, and the stress release processing is performed based on the read data segment 1, and in the case that the data segment 1 is processed, the stress test task is switched to the completion state.

[0120] In summary, when the number of data segments is the first set number, the first data segment in the data segment sequence is directly loaded into the first circular array, and the first data segment is read and processed in the first circular array according to the loading result to complete the target task, that is, without using the second circular array in the case of small data volume, the data processing efficiency is further improved.

[0121] In addition to the above cases where the data segment sequence contains at least three data segments and the data segment sequence contains a first set number of data segments, in the present embodiment, when the number of data segments contained in the data segment sequence is a second set number, the following cases are also included:

[0122] The first data segment in the data segment sequence is loaded into the first circular array, and the second data segment adjacent to the first data segment is loaded into the second circular array;

[0123] The first data segment is read and processed in the first circular array;

[0124] After the first data segment in the first circular array is processed, the second data segment is read and processed in the second circular array;

[0125] When the second data segment is processed, the target task is switched to a completed state.

[0126] Specifically, the second set number refers to the number of data segments that are set in advance, and the second set number can be 2. In the case of two segments, the first data segment is loaded and the data needed to be read is provided by the first circular array. The second data segment is loaded and the data needed to be read is provided by the second circular array, so that the processing of the data segment sequence can be completed. Therefore, after the first data segment is loaded into the first circular array and the second data segment is loaded into the second circular array, the first data segment is read and processed in the first circular array. After the first data segment is processed, the second data segment is read and processed in the second circular array. After the second data segment is processed, it indicates that the target task is completed, and the target task is switched to a completed state.

[0127] For example, if there are only data segment 1 and data segment 2 in the data segment sequence, data segment 1 is loaded into the circular array A, and data segment 2 is loaded into the circular array B, then data segment 1 in the circular array A is read, and the pressure processing is performed based on the read data segment 1, in the case of data segment 1 being processed, data segment 2 in the circular array B is read, and the pressure processing is performed based on the read data segment 2, in the case of data segment 2 being processed, the pressure measurement task is switched to the completion state.

[0128] In summary, when the number of data segments is the second set number, the task processing can be completed by switching between the first circular array and the second circular array once, which improves the task processing efficiency

[0129] In some tasks that need to be repeated, the target task may need to be executed multiple times. Therefore, in this embodiment, before the step of creating the data segment sequence corresponding to the target task in the execution state, the following steps are further included:

[0130] Creating a target total task, wherein the target total task includes: a target task and the number of executions of the target task;

[0131] Starting the target task in the target total task and executing the step of creating the data segment sequence corresponding to the target task in the execution state;

[0132] Correspondingly, after the step of switching the target task to the completion state, the following steps are further included:

[0133] Recording the number of task executions corresponding to the target task;

[0134] Determining whether the number of task executions is equal to the number of executions;

[0135] If not, switching the target task to the execution state and returning to the step of loading the first data segment in the data segment sequence into the first circular array;

[0136] If yes, determining the task execution information corresponding to the target total task.

[0137] Specifically, the target total task refers to a total task composed of multiple target tasks. The execution times refer to the number of times the target task needs to be executed. Correspondingly, the task execution times refer to the actual execution times of the target task. After the target task is switched to a completed state, the task execution times corresponding to the target task are recorded, that is, the task execution times are incremented by one after each completion of the target task. If the task execution times are less than the execution times, it indicates that the execution times of the target task have been completed, and the target task needs to be executed again, and the execution step 204 is returned; if the task execution times are equal to the execution times, it indicates that the execution times of the target task have been completed, and the target task does not need to be executed any more, and the task execution information (i.e., the task execution result) of the target total task can be determined according to the execution of the target task. In the stress testing scenario, the task execution information can be represented as: stress testing is completed. In addition, the task execution information can also include specific execution information, such as the number of stress testing requests sent per second during stress testing, and the like.

[0138] In summary, by executing the target task multiple times until the preset execution times are met, the execution of the target total task is realized. The need to repeatedly start the target task that needs to be repeatedly executed is avoided, and the execution process is simplified.

[0139] The data processing method provided in the present application loads the first data segment in the data segment sequence to the first circular array by creating the data segment sequence corresponding to the target task in the execution state, that is, the data is processed in batches by data segments, then the first data segment in the first circular array is read and processed according to the loading result, and the second data segment adjacent to the first data segment is loaded to the second circular array, wherein the second data segment in the second circular array is processed after the first data segment in the first circular array is processed, and the third data segment adjacent to the second data segment in the data segment sequence is loaded to the first circular array when the second data segment in the second circular array is in the processing state, that is, the data segments are alternately loaded by the first circular array and the second circular array, and the data to be read is alternately provided externally; until the data segments contained in the data segment sequence are all processed, the target task is switched to a completed state, which realizes that a large amount of memory space does not need to be opened at one time in the data processing process, and the occupation of the memory space is reduced.

[0140] The following describes the data processing method provided in the present application in combination with the accompanying Figure 4 application of the data processing method provided in the present application in the stress testing scenario. Among them, Figure 4 Fig. 1 shows a processing flowchart of a data processing method applied in a stress testing scenario according to an embodiment of the present application, which specifically includes the following steps:

[0141] Step 402: Obtain a stress test file associated with the stress test task.

[0142] During large events or festivals (such as the global finals of a game), some service systems will bear several times or even several orders of magnitude higher traffic pressure than usual, which is a great test of the system's carrying capacity. It is necessary to conduct bottleneck analysis, capacity expansion, verification, and other operations in advance. The stress test platform provides technical support for these special scenarios. It needs to provide large-scale QPS stress testing capabilities, usually reaching tens of thousands, hundreds of thousands, or even higher QPS. Such stress testing capabilities consume a large amount of stress testing resources, and the industry usually uses distributed stress testing to achieve this. However, although distributed stress testing can improve stress testing capabilities by increasing the number of nodes, resources are always limited, which requires continuous improvement in the resource utilization efficiency of individual stress testing nodes.

[0143] To construct large traffic pressure in the stress testing scenario, a large number of requests must be sent in a short period of time. In order to achieve this effect, it is usually not possible to read data from the outside temporarily before sending requests, but a large amount of stress test data needs to be loaded into the memory in advance, and then consumed by multiple threads and sent requests. The industry usually maintains a thread-safe queue, starts a part of threads to continuously fill data into the queue, and starts another part of threads as stress testing threads to continuously consume data from the queue and perform stress testing operations.

[0144] Therefore, the above processing method may cause insufficient memory. It is usually necessary to create a queue with sufficient capacity to load stress test file content into memory to ensure that data can always be obtained during stress testing. However, when the stress testing scenario becomes complex, such as multiple requests being concatenated to form an interface group, and the request body being large, the queue will occupy a large amount of memory of the stress testing container, eventually causing insufficient memory and affecting the stress testing capability, or even causing the stress testing node to crash. In addition, there is a problem of repeated loading. Because elements are consumed from the queue, in order to ensure continuous stress testing, there is a process of repeatedly reading data from the file and filling it into the queue, which wastes the resources of the stress testing container, and this problem is particularly evident when the number of rows in the stress test file is not large. In summary, the above processing method can also provide continuous stress testing capability, but it has the disadvantages of possibly causing insufficient resources of the stress testing container, or low resource utilization, which ultimately affects the stress testing performance of individual stress testing nodes.

[0145] Step 404: According to the number of elements corresponding to the first loop array and the second loop array respectively, segment the data in the stress test file to obtain a data segment sequence composed of data segments.

[0146] Step 406: Obtain the data rows in the first data segment in the data segment sequence, and load the obtained data rows as data elements into the first loop array.

[0147] Specifically, in each data row, there can be a single stress test request, i.e., a stress test interface (function), or a plurality of stress test requests concatenated to form an interface group.

[0148] Step 408: Mark the first loop array as a loading completion state.

[0149] Specifically, in the case where the first loop array completes loading the first data segment, the first loop array is marked as a loading completion state.

[0150] Step 410: Start at least two stress task threads corresponding to the stress test task, read data elements in the first loop array by executing each stress task thread, perform stress processing on the read data elements, obtain data rows in the second data segment, and load the obtained data rows to the second loop array as data elements.

[0151] Specifically, the stress task thread sends a stress test request to the server to be tested for each data element processed.

[0152] Step 412: Mark the second loop array as a loading completion state.

[0153] Specifically, in the case where the second loop array completes loading the second data segment, the first loop array is marked as a loading completion state. In addition, after step 410, there is a case where the number of read elements corresponding to the first loop array is equal to the number of elements corresponding to the first loop array, but the second loop array is in an unloading completion state. On this basis, the data elements in the first loop array are re-read and processed by executing each task thread until the second loop array is marked as a loading completion state.

[0154] Step 414: In the case where the number of read elements corresponding to the first loop array is equal to the number of elements corresponding to the first loop array, and the second loop array is in a loading completion state, read data elements in the second loop array by executing each stress task thread.

[0155] Step 416: In the case where the second data segment in the second loop array is in a processed state, clear the first data segment loaded in the first loop array.

[0156] Step 418: Load a third data segment adjacent to the second data segment in the data segment sequence to the first loop array.

[0157] Specifically, in a case where the first loop array completes loading the third data segment, the first loop array is marked as a loading completion state. After determining that the second data segment in the second loop array is processed, data elements in the first loop array are read by executing each stress thread, the read data elements are processed, and the second data segment loaded in the second loop array is cleared. The fourth data segment is loaded to the second loop array, and in a case where the second loop array completes loading the fourth data segment, the second loop array is marked as a loading completion state, and so on, until the data segments included in the data segment sequence are all processed.

[0158] Step 420: Until the data segments included in the data segment sequence are all processed, the stress test task is switched to a completion state.

[0159] The above is an implementation process of a stress test node performing a stress test task. In actual application, a plurality of stress test nodes can send stress test requests to the stress test service in parallel, so as to meet the required number of concurrent requests of the stress test service.

[0160] The data processing method provided in the application creates a data segment sequence corresponding to a stress test task in an execution state, loads a first data segment in the data segment sequence to a first loop array, that is, data is processed in batches through data segments, reads the first data segment in the first loop array according to a loading result and processes it, and loads a second data segment adjacent to the first data segment to a second loop array, wherein the second data segment in the second loop array is processed after the first data segment in the first loop array is processed. In a case where the second data segment in the second loop array is in a processed state, a third data segment adjacent to the second data segment in the data segment sequence is loaded to the first loop array, that is, the first loop array and the second loop array are alternately loaded with data segments, and alternately provide data to be read to the outside. Until the data segments included in the data segment sequence are all processed, the stress test task is switched to a completion state, which realizes that a large amount of memory space does not need to be opened at one time in the data processing process, and the occupation of the memory space is reduced.

[0161] Corresponding to the method embodiments, the application also provides data processing device embodiments, Figure 5 A structural schematic diagram of a data processing device provided by an embodiment of the application is shown. As shown in the figure, Figure 5 The device includes:

[0162] The creating module 502 is configured to create a data segment sequence corresponding to a target task in an execution state;

[0163] The first loading module 504 is configured to load a first data segment in the data segment sequence to a first circular array;

[0164] The reading module 506 is configured to read and process the first data segment in the first circular array according to a loading result, and load a second data segment adjacent to the first data segment to a second circular array, wherein the second data segment in the second circular array is processed after the first data segment in the first circular array is processed.

[0165] The second loading module 508 is configured to load a third data segment adjacent to the second data segment in the data segment sequence to the first circular array when the second data segment in the second circular array is in a processing state.

[0166] The switching module 510 is configured to switch the target task to a completed state until all data segments contained in the data segment sequence are processed.

[0167] Optionally, the data processing apparatus further comprises:

[0168] The receiving module is configured to receive a target task instruction, create a target task according to target task information carried in the target task instruction, and start the target task.

[0169] Correspondingly, the creating module 502 comprises:

[0170] The obtaining submodule is configured to obtain a to-be-processed file associated with the target task.

[0171] The segmenting submodule is configured to perform segmenting processing on data contained in the to-be-processed file according to a number of elements corresponding to the first circular array and the second circular array respectively, to obtain a data segment sequence composed of data segments.

[0172] Optionally, the data processing apparatus further comprises, when the number of data segments contained in the data segment sequence is a first set number:

[0173] The third loading module is configured to load a first data segment in the data segment sequence to a first circular array;

[0174] The second reading module is configured to read and process the first data segment in the first circular array according to a loading result.

[0175] The second switching module is configured to switch the target task to a completed state when the first data segment is processed.

[0176] Optionally, the data processing apparatus, when the number of data segments included in the data segment sequence is a second set number, further comprises:

[0177] a fourth loading module configured to load a first data segment in the data segment sequence to a first circular array, and load a second data segment adjacent to the first data segment to a second circular array;

[0178] a third reading module configured to read and process the first data segment in the first circular array;

[0179] after the first data segment in the first circular array is processed, reading and processing the second data segment in the second circular array;

[0180] a third switching module configured to switch the target task to a completed state when the second data segment is processed.

[0181] Optionally, the data processing apparatus further comprises:

[0182] a creating task module configured to create a target total task, wherein the target total task comprises: a target task and an execution number of the target task;

[0183] a starting module configured to start the target task in the target total task, and execute a step of creating a data segment sequence corresponding to the target task in an execution state;

[0184] Correspondingly, the data processing apparatus further comprises:

[0185] a recording number module configured to record a task execution number corresponding to the target task;

[0186] a judging module configured to judge whether the task execution number is equal to the execution number;

[0187] if not, a third switching module is executed, and the third switching module is configured to switch the target task to an execution state, and return to execute the first loading module;

[0188] if yes, a determining module is executed, and the determining module is configured to determine task execution information corresponding to the target total task.

[0189] Optionally, the first loading module is further configured to:

[0190] obtain a data row in the first data segment in the data segment sequence, and load the obtained data row as a data element to the first circular array;

[0191] Correspondingly, the reading module is further configured to:

[0192] The determining sub-module is configured to determine a second data segment adjacent to the first data segment in the data segment sequence.

[0193] The loading sub-module is configured to acquire a data row in the second data segment and load the acquired data row as a data element into the second circular array.

[0194] Optionally, the reading module 506 is further configured to:

[0195] The starting thread sub-module is configured to start at least two task threads corresponding to the target task.

[0196] The executing thread sub-module is configured to read data elements in the first circular array and process by executing each task thread.

[0197] Optionally, the data processing apparatus further comprises:

[0198] The record number module is configured to record a read element number of the data elements in the first circular array, wherein the read element number is used to detect a read state of the data elements in the first circular array.

[0199] Optionally, the data processing apparatus further comprises:

[0200] The first marking module is configured to mark the first circular array as a loading completion state.

[0201] Correspondingly, the reading module 506 is further configured to:

[0202] The second marking module is configured to mark the second circular array as the loading completion state.

[0203] Correspondingly, the reading module 506 is further configured to:

[0204] In a case where the read element number corresponding to the first circular array is equal to the element number corresponding to the first circular array and the second circular array is in the loading completion state, reading data elements in the second circular array and processing by executing each task thread.

[0205] Optionally, the data processing apparatus further comprises:

[0206] The re-reading module is configured to, in a case where the first loop array is determined to correspond to the read element quantity equal to the element quantity corresponding to the first loop array and the second loop array is in an incomplete loading state, re-read data elements in the first loop array and process by executing each task thread until the second loop array is marked as the loading completion state.

[0207] Optionally, the data processing apparatus further comprises:

[0208] The clearing module is configured to clear the loaded first data segment in the first loop array.

[0209] The data processing apparatus provided in the present application loads the first data segment in the data segment sequence to the first loop array by creating the data segment sequence corresponding to the target task in the execution state, that is, the data is processed in batches by the data segment, then reads the first data segment in the first loop array according to the loading result and processes, and loads the second data segment adjacent to the first data segment to the second loop array, wherein the second data segment in the second loop array is processed after the first data segment in the first loop array is processed, and the third data segment adjacent to the second data segment in the data segment sequence is loaded to the first loop array in a case where the second data segment in the second loop array is in the processing state, that is, the first loop array and the second loop array are alternately loaded with data segments, and alternately provide data to be read to the outside; until the data segments contained in the data segment sequence are all processed, the target task is switched to the completion state, which realizes that a large amount of memory space does not need to be opened at one time in the data processing process, and the occupation of the memory space is reduced.

[0210] The above is a schematic scheme of the data processing apparatus of the embodiment. It should be noted that the technical scheme of the data processing apparatus belongs to the same concept as the technical scheme of the data processing method described above, and the details of the technical scheme of the data processing apparatus which are not described in detail can be referred to the description of the technical scheme of the data processing method.

[0211] Figure 6 A structural block diagram of a computing device 600 according to an embodiment of the present application is shown. The components of the computing device 600 include but are not limited to a memory 610 and a processor 620. The processor 620 is connected with the memory 610 through a bus 630, and a database 650 is used to save data.

[0212] The computing device 600 also includes an access device 640 that enables the computing device 600 to communicate via one or more networks 660. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or combinations of such networks, such as the Internet. The access device 640 can include one or more of any type of network interface (for example, a network interface card (NIC)), such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, or the like.

[0213] In one embodiment of the present application, the above-mentioned components of the computing device 600, as well as other components not shown in FIG. 6, can be connected to each other by a bus. It should be understood that Figure 6 the computing device structure diagram shown is only for the purpose of example, and is not a limitation on the scope of the present application. Those skilled in the art can add or replace other components as needed. Figure 6 the computing device structure diagram shown is only for the purpose of example, and is not a limitation on the scope of the present application. Those skilled in the art can add or replace other components as needed.

[0214] The computing device 600 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (for example, a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, and the like), a mobile phone (for example, a smartphone), a wearable computing device (for example, a smart watch, smart glasses, and the like), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 600 can also be a mobile or stationary server.

[0215] wherein the processor 620 implements the steps of the data processing method when executing the computer instructions.

[0216] The above is a schematic scheme of the computing device of the embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the data processing method described above belong to the same concept, and the details of the technical scheme of the computing device that are not described in detail can be referred to the description of the technical scheme of the data processing method.

[0217] An embodiment of the present application further provides a computer readable storage medium, which stores computer instructions, and the computer instructions are executed by a processor to implement the steps of the data processing method.

[0218] The above is a schematic scheme of the computer readable storage medium of the embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the data processing method described above belong to the same concept, and the details of the technical scheme of the storage medium that are not described in detail can be referred to the description of the technical scheme of the data processing method.

[0219] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in a different order and still achieve desirable results. Additionally, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

[0220] The computer instructions include computer program codes, which can be in the form of source code, object code, executable files, or some intermediate forms, etc. The computer readable medium can include any entity or device capable of carrying the computer program codes, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0221] It should be noted that, for the aforementioned method embodiments, the sequences of the described actions are not the only ones that can be performed to implement the present application. In some embodiments, the sequences of actions can be performed in different order or simultaneously. In some embodiments, other sequences of actions can be performed, which should be apparent to a person of ordinary skill in the art in light of the teachings of the present application.

[0222] In the above embodiments, the description of each embodiment focuses on different aspects, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0223] The preferred embodiments of the present application disclosed above are only used to explain the present application. The optional embodiments do not describe all the details and limit the present application to the specific embodiments described. Obviously, many modifications and changes can be made according to the content of the present application. The present application selects and describes these embodiments in order to better explain the principles and practical applications of the present application, so that those skilled in the art can well understand and use the present application. The present application is limited by the claims and their full scope and equivalents.

Claims

1. A data processing method, characterized in that, include: Create a sequence of data segments corresponding to the target task that is in the execution state; The first data segment in the data segment sequence is loaded into the first circular array; Based on the loading result, the first data segment is read from the first circular array and processed, and the second data segment adjacent to the first data segment is loaded into the second circular array. The second data segment in the second circular array is processed after the first data segment in the first circular array is processed. While the second data segment in the second circular array is being processed, the third data segment adjacent to the second data segment in the data segment sequence is loaded into the first circular array; The target task will be switched to the completed state once all data segments contained in the data segment sequence have been processed.

2. The data processing method according to claim 1, characterized in that, Before creating the data segment sequence corresponding to the target task in the execution state, the process also includes: Receive the target task instruction, create the target task based on the target task information carried in the target task instruction, and start it. Accordingly, the creation of the data segment sequence corresponding to the target task in the execution state includes: Obtain the files to be processed associated with the target task; Based on the number of elements corresponding to the pre-created first and second circular arrays, the data contained in the file to be processed is segmented to obtain a data segment sequence composed of data segments.

3. The data processing method according to claim 1 or 2, characterized in that, When the number of data segments included in the data segment sequence is a first predetermined number, the method further includes: The first data segment in the data segment sequence is loaded into the first circular array; Based on the loading result, read the first data segment from the first loop array and process it; Once the first data segment has been processed, the target task is switched to a completed state.

4. The data processing method according to claim 1 or 2, characterized in that, When the number of data segments included in the data segment sequence is a second predetermined number, the method further includes: The first data segment in the data segment sequence is loaded into the first circular array, and the second data segment adjacent to the first data segment is loaded into the second circular array; Read the first data segment from the first circular array and process it; After the first data segment in the first circular array is processed, the second data segment is read from the second circular array and processed. Once the second data segment has been processed, the target task is switched to a completed state.

5. The data processing method according to claim 1, characterized in that, Before creating the data segment sequence corresponding to the target task in the execution state, the process also includes: Create a target total task, wherein the target total task includes: target tasks and the number of times the target tasks are executed; Start the target task in the overall target task and execute the step of creating the data segment sequence corresponding to the target task in the execution state; Accordingly, after switching the target task to the completed state, the process also includes: Record the number of times the target task is executed; Determine whether the number of times the task is executed is equal to the number of executions. If not, switch the target task to the execution state and return to the execution of loading the first data segment in the data segment sequence into the first loop array; If so, determine the task execution information corresponding to the target total task.

6. The data processing method according to claim 1 or 2, characterized in that, The step of loading the first data segment from the data segment sequence into the first circular array includes: Obtain the data row from the first data segment in the data segment sequence, and load the obtained data row as a data element into the first circular array; Accordingly, loading the adjacent second data segments of the first data segment into the second circular array includes: In the data segmentation sequence, determine the second data segment adjacent to the first data segment; Get the data rows from the second data segment and load the data rows as data elements into the second circular array.

7. The data processing method according to claim 6, characterized in that, The step of reading and processing the first data segment from the first circular array based on the loading result includes: Start at least two task threads corresponding to the target task; Each task thread reads and processes the data elements in the first loop array.

8. The data processing method according to claim 7, characterized in that, Also includes: Record the number of data elements read in the first circular array, wherein the number of read elements is used to detect the reading status of data elements in the first circular array.

9. The data processing method according to claim 8, characterized in that, After loading the first data segment from the data segmentation sequence into the first circular array, the method further includes: Mark the first loop array as loaded; Accordingly, after loading the second data segments adjacent to the first data segments into the second circular array, the method further includes: Mark the second loop array as the loading complete state; Accordingly, the second data segment in the second circular array is processed after the first data segment in the first circular array has been processed, including: If it is determined that the number of elements to be read in the first circular array is equal to the number of elements in the first circular array, and the second circular array is in the loaded state, the data elements in the second circular array are read and processed by each task thread.

10. The data processing method according to claim 9, characterized in that, Before the step of reading and processing data elements in the second circular array by executing each task thread, the process further includes: If it is determined that the number of elements read corresponding to the first circular array is equal to the number of elements corresponding to the first circular array, and the second circular array is in an unloaded state, the data elements in the first circular array are reread and processed by each task thread until the second circular array is marked as the loaded state.

11. The data processing method according to claim 1, characterized in that, Before loading the third data segment adjacent to the second data segment in the data segment sequence into the first circular array, the method further includes: Clear the first data segment loaded in the first loop array.

12. A data processing apparatus, characterized in that, include: The creation module is configured to create a sequence of data segments corresponding to a target task that is in the execution state. The first loading module is configured to load the first data segment in the data segment sequence into the first circular array; The reading module is configured to read and process the first data segment in the first circular array according to the loading result, and load the second data segment adjacent to the first data segment into the second circular array, wherein the second data segment in the second circular array is processed after the first data segment in the first circular array is processed; The second loading module is configured to load a third data segment adjacent to the second data segment in the data segment sequence into the first circular array while the second data segment in the second circular array is being processed. The switching module is configured to switch the target task to a completed state until all data segments contained in the data segment sequence have been processed.

13. A computing device, comprising a memory, a processor, and computer instructions stored in the memory and executable on the processor, characterized in that, When the processor executes the computer instructions, it implements claim 1.

11. The steps of any of the methods described in the above.

14. A computer-readable storage medium storing computer instructions, characterized in that, When the computer instruction is executed by the processor, it implements claim 1.

11. The steps of any of the methods described in the above.

15. A computer program product comprising computer instructions, characterized in that, When executed by a processor, the computer instructions implement the steps of the method according to any one of claims 1-11.

Images