Data processing method and device, electronic equipment and readable storage medium

Abstract

The embodiment of the invention provides a data processing method and device, electronic equipment and a readable storage medium. The method comprises the following steps: in response to a data processing request, determining a target address indicated by the data processing request; acquiring data of a preset data size from the target address as to-be-processed data until the data in the target address is the to-be-processed data, and executing target processing operation on the to-be-processed data to obtain processing sub-results corresponding to the to-be-processed data; the target processing operation is a data processing operation indicated by the data processing request; and generating a data processing result based on the processing sub-results corresponding to the to-be-processed data. The data can be subjected to block processing, so that the data processing efficiency can be improved to a certain extent.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data processing method, device, electronic device and readable storage medium. Background Art

[0002] With the development of computer technology, there is a large amount of data that needs to be processed in various fields, such as image data, video data, network monitoring data, etc.

[0003] At present, in the process of processing these data, the data to be processed is usually read into the memory first, and then the data is processed in the memory. However, this method has poor processing efficiency when the data to be processed is large. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a data processing method, device, electronic device and readable storage medium to improve data processing efficiency. The specific technical solution is as follows:

[0005] In a first aspect of the present invention, a data processing method is provided, the method comprising:

[0006] In response to a data processing request, determining a target address indicated by the data processing request;

[0007] Acquire data of a preset data size from the target address as the data to be processed, until all the data in the target address are the data to be processed, and respectively perform target processing operations on the data to be processed to obtain processing sub-results corresponding to the data to be processed; the target processing operations are the data processing operations indicated by the data processing request;

[0008] A data processing result is generated based on the processing sub-results corresponding to each of the data to be processed.

[0009] Optionally, performing target processing operations on the data to be processed respectively includes:

[0010] For any of the data to be processed, selecting a thread from the threads to be used included in the thread pool as a target thread;

[0011] Determine the target thread as a used thread, and call the target thread to perform the target processing operation on the data to be processed;

[0012] The target thread is determined as a thread to be used.

[0013] Optionally, the calling the target thread to perform the target processing operation on the data to be processed includes:

[0014] Calling the target thread to determine a target position in the data to be processed; the target position is a position where a line break character is located in the data to be processed and where there is no line break character between the data to be processed and the end position of the data to be processed;

[0015] Taking the data from the starting position of the to-be-processed data to the target position as new to-be-processed data, and calling the target thread to perform the target processing operation on the new to-be-processed data;

[0016] And, the data from the target position to the end position is used as data to be spliced, and the data to be spliced ​​is spliced ​​with the next data to be processed of the data to be processed.

[0017] Optionally, before generating the data processing result based on the processing sub-results corresponding to the data to be processed, the method further includes:

[0018] In response to a data write request, a preset data write module is used to write each of the processing sub-results to the write address indicated by the data write request.

[0019] Optionally, generating a data processing result based on the processing sub-results corresponding to each of the to-be-processed data includes:

[0020] The processing sub-results corresponding to the data to be processed are spliced ​​in a target order to obtain a data processing result; the target order is the order in which the data to be processed are obtained.

[0021] Optionally, the method is applied to a Prometheus database.

[0022] In a second aspect of the present invention, there is also provided a data processing device, the device comprising:

[0023] A first determining module, configured to determine, in response to a data processing request, a target address indicated by the data processing request;

[0024] an acquisition module, configured to acquire data of a preset data size from the target address as data to be processed, until all the data in the target address are data to be processed, and to perform target processing operations on the data to be processed respectively to obtain processing sub-results corresponding to the data to be processed; the target processing operations are the data processing operations indicated by the data processing request;

[0025] A generating module is used to generate a data processing result based on the processing sub-results corresponding to each of the data to be processed.

[0026] Optionally, the acquisition module includes:

[0027] A selection submodule, for selecting a thread from the threads to be used included in the thread pool as a target thread for any of the to-be-processed data;

[0028] A calling submodule, used for determining the target thread as a used thread, and calling the target thread to perform the target processing operation on the data to be processed;

[0029] The second determining submodule is used to determine the target thread as the thread to be used.

[0030] Optionally, the calling submodule is specifically used to:

[0031] Calling the target thread to determine a target position in the data to be processed; the target position is a position where a line break character is located in the data to be processed and where there is no line break character between the data to be processed and the end position of the data to be processed;

[0032] Taking the data from the starting position of the to-be-processed data to the target position as new to-be-processed data, and calling the target thread to perform the target processing operation on the new to-be-processed data;

[0033] And, the data from the target position to the end position is used as data to be spliced, and the data to be spliced ​​is spliced ​​with the next data to be processed of the data to be processed.

[0034] Optionally, the device further comprises:

[0035] The writing module is used to respond to the data writing request and adopt the preset data writing module to write each of the processing sub-results to the writing address indicated by the data writing request.

[0036] Optionally, the generating module is specifically used to:

[0037] The processing sub-results corresponding to the data to be processed are spliced ​​in a target order to obtain a data processing result; the target order is the order in which the data to be processed are obtained.

[0038] Optionally, the device is applied to a Prometheus database.

[0039] In a third aspect of the present invention, there is also provided an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0040] Memory, used to store computer programs;

[0041] The processor is used to implement the method described in the first aspect when executing the program stored in the memory.

[0042] In a fourth aspect of the implementation of the present invention, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the method described in the first aspect above.

[0043] In a fifth aspect of the implementation of the present invention, there is also provided a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method described in the first aspect above.

[0044] The data processing method provided by the embodiment of the present invention determines the target address indicated by the data processing request in response to the data processing request; obtains data of a preset data size from the target address as the data to be processed, until the data in the target address are all the data to be processed, and performs target processing operations on the data to be processed respectively to obtain processing sub-results corresponding to each of the data to be processed; the target processing operations are the data processing operations indicated by the data processing request; and generates data processing results based on the processing sub-results corresponding to each of the data to be processed. In the embodiment of the present invention, by obtaining the data to be processed according to the preset data size and performing target processing operations on the data to be processed respectively, the data to be processed can be obtained from the target address in a block processing mode, which reduces the memory consumption caused by data acquisition and improves the data acquisition performance. At the same time, by performing target processing operations on the data to be processed respectively, the data can be processed in blocks, thereby improving the data processing efficiency to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0046] Figure 1 is a flow chart of steps of a data processing method in an embodiment of the present invention;

[0047] Figure 2 A schematic diagram of a scenario in an embodiment of the present invention;

[0048] Figure 3 An interactive schematic diagram of data processing in an embodiment of the present invention;

[0049] Figure 4 is a structural schematic diagram of a data processing device in an embodiment of the present invention;

[0050] Figure 5 Schematic diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] Figure 1 is a flow chart of steps of a data processing method in an embodiment of the present invention, such as Figure 1 As shown, the method may include:

[0053] Step 101: In response to a data processing request, determine a target address indicated by the data processing request;

[0054] Step 102: obtain data of a preset data size from the target address as data to be processed, until all the data in the target address are data to be processed, and perform target processing operations on the data to be processed respectively to obtain processing sub-results corresponding to each data to be processed; the target processing operation is the data processing operation indicated by the data processing request.

[0055] Step 103: Generate a data processing result based on the processing sub-results corresponding to each of the data to be processed.

[0056] With respect to the above steps 101 to 103, the embodiment of the present invention can be applied to any computer device or database with data computing and processing capabilities. In one case, the embodiment of the present invention can be applied to a time series database, such as Prometheus, and can also be applied to a relational database (RDB), which is not limited by the embodiment of the present invention.

[0057] The data processing request may be actively sent by a third-party device, or may be automatically generated according to a preset monitoring cycle. Accordingly, the data processing request may carry the storage address of the data to be processed, so that the embodiment of the present invention can obtain the storage address of the data to be processed as the target address by parsing the data processing request.

[0058] The target address refers to the storage address where the data to be processed is located, which may be a memory address, a cloud storage address, or an external device storage address, and the embodiment of the present invention does not limit this. Specifically, when the target address is a memory address, the embodiment of the present invention may initiate access to the target address through a memory access operation to obtain the data to be processed. When the target address is a cloud storage address or an external device storage address, the embodiment of the present invention may obtain the data to be processed by sending a Hypertext Transfer Protocol (http) request to the target address.

[0059] Among them, the above-mentioned preset data size can be pre-set, and can be set according to the data processing efficiency requirements and processing performance in the actual scenario, and the embodiment of the present invention does not limit this. Exemplarily, the above-mentioned preset data size can be 64KB, 1M, 10M, etc., and the embodiment of the present invention does not limit this. Specifically, the embodiment of the present invention can obtain data from the target address as the data to be processed according to the preset data size, that is, the data of the target address is divided into blocks, and the size of each data block is the preset data size, so that only one data block needs to be obtained each time. Specifically, the embodiment of the present invention can send a data block acquisition request to the target address, so that the target address returns the data to be processed in sequence according to the preset data size. Alternatively, the embodiment of the present invention can also directly access the target address and obtain data of the preset data size each time, which can be set according to actual needs, and the embodiment of the present invention does not limit this. Exemplarily, taking the preset data size of 64KB and applying it to the Prometheus database as an example, the above-mentioned data acquisition operation can be implemented by calling the data reading method (read) of the Prometheus database, and 64KB of data is obtained each time by calling read. Furthermore, data may be acquired from the target address according to a preset data size until all the data in the target address are data to be processed, indicating that all the data indicated by the data processing request has been acquired and all the data to be processed has been acquired.

[0060] Among them, the above-mentioned target processing operation is the data processing operation indicated by the data processing request, which can be any data parsing operation, data encryption operation or other data processing operation, which can be set according to actual needs, and the embodiment of the present invention does not limit this. Specifically, the embodiment of the present invention can perform the target processing operation on each data to be processed, and obtain the processing sub-result corresponding to each data to be processed. Specifically, the embodiment of the present invention can perform the target processing operation on each data to be processed each time it obtains a data to be processed, or it can perform the target processing operation on each data to be processed after obtaining multiple or all data to be processed. It can be set according to the implementation scenario, and the embodiment of the present invention does not limit this.

[0061] Specifically, the above target processing operation can be implemented by calling a pre-created thread pool, or can also be implemented by a preset data processing module, which is not limited in the embodiment of the present invention.

[0062] Further, after obtaining the processing sub-results corresponding to each of the data to be processed, the final data processing result can be obtained based on each of the processing sub-results. Specifically, each of the processing sub-results can be spliced ​​to obtain the data processing result. Alternatively, each of the processing sub-results can be combined to obtain the data processing result, which can be set according to actual needs, and the embodiment of the present invention does not limit this.

[0063] Optionally, the above operation of generating a data processing result based on the processing sub-results corresponding to each of the data to be processed may specifically include:

[0064] S1031. Concatenate the processing sub-results corresponding to the data to be processed in a target order to obtain a data processing result; the target order is the order in which the data to be processed are obtained.

[0065] Among them, the above-mentioned target order refers to the order of obtaining each data to be processed. Furthermore, after obtaining the processing sub-results corresponding to each data to be processed, the processing sub-results can be spliced ​​according to the order of obtaining the corresponding data to be processed to obtain a complete processing result.

[0066] Optionally, the embodiment of the present invention can be applied to the Prometheus database, which can provide monitoring services, that is, by performing a Hypertext Transfer Protocol (http) service on the monitored object, the data exposed by a certain indicator of the monitored object is obtained. When the amount of indicators or data exposed by the monitored object is particularly large, Prometheus will occupy a large amount of memory during the data pulling process, and even timeouts may occur during processing, resulting in poor data processing performance and inability to effectively monitor the monitored object. Figure 2 is a schematic diagram of a scenario in an embodiment of the present invention, such as Figure 2As shown in the figure, the data capture mode of Prometheus is to initiate an http request to the target address (target) through the transceiver module (http-client), and the target returns the data. After Prometheus obtains the request result, it reads all the results into the memory, and then parses the data row by row in the memory through the data reading instruction (readAll) and the data parsing module (parse). After the parsing is completed, the result is stored locally or remotely. This mode will encounter a problem. When the returned data is particularly large, it will consume a lot of memory, and the processing will time out due to slow processing.

[0067] In the embodiment of the present invention, by responding to a data processing request, the target address indicated by the data processing request is determined; data of a preset data size is obtained from the target address as the data to be processed, until the data in the target address are all the data to be processed, and the target processing operation is performed on the data to be processed respectively to obtain the processing sub-results corresponding to each of the data to be processed; the target processing operation is the data processing operation indicated by the data processing request; and the data processing result is generated based on the processing sub-results corresponding to each of the data to be processed. In the embodiment of the present invention, by obtaining the data to be processed according to the preset data size and performing the target processing operation on the data to be processed respectively, the data to be processed can be obtained from the target address in a block processing mode, which reduces the memory consumption caused by data acquisition and improves the data acquisition performance. At the same time, by performing the target processing operation on the data to be processed respectively, the data can be processed in blocks, thereby improving the data processing efficiency to a certain extent.

[0068] Optionally, the above-mentioned target processing operations are respectively performed on the data to be processed, and the embodiment of the present invention may specifically include:

[0069] S21. For any of the data to be processed, a thread is selected from the threads to be used included in the thread pool as a target thread.

[0070] S22: Determine the target thread as a used thread, and call the target thread to perform the target processing operation on the data to be processed.

[0071] S23: Determine the target thread as a thread to be used.

[0072] The thread pool may be pre-created, such as a fixed thread pool (Executors.newFixedThreadPool), a cached thread pool (newCachedThreadPool), or a scheduling thread pool, etc., which is not limited in the present embodiment. In the Prometheus database, a data parsing thread pool (parser thread pool) may also be created.

[0073] Specifically, the thread pool may contain multiple threads to be used, so that for any data to be processed, the embodiment of the present invention can select a thread from the threads to be used contained in the thread pool as the target thread. Specifically, the selection can be made randomly or according to the order in which the threads are generated, and the embodiment of the present invention does not limit this.

[0074] Furthermore, the embodiment of the present invention can determine the target thread as a used thread, and call the target thread to perform the target processing operation on the data to be processed. Accordingly, for other data to be processed, the thread to be used can continue to be selected from the thread pool as the target thread to perform the target processing operation, so that different threads can synchronously process different data to be processed at the same time. Optionally, after the target thread is used to perform the target processing operation on the data to be processed, the target thread can continue to be determined as the thread to be used to implement thread recycling, so that the target thread can continue to process subsequent data to be processed.

[0075] In the embodiment of the present invention, for any of the data to be processed, a thread is selected from the threads to be used included in the thread pool as the target thread; the target thread is determined as a used thread, and the target thread is called to perform the target processing operation on the data to be processed; the target thread is determined as the thread to be used. In this way, by setting the thread pool, asynchronous processing can be achieved for each data to be processed, thereby further improving data processing efficiency.

[0076] Optionally, the operation of calling the target thread to perform the target processing operation on the data to be processed may specifically include:

[0077] S31, calling the target thread to determine a target position in the data to be processed; the target position is a position in the data to be processed where there is no line break between the data to be processed and the end position of the data to be processed.

[0078] S32: taking the data from the starting position of the data to be processed to the target position as new data to be processed, and calling the target thread to perform the target processing operation on the new data to be processed.

[0079] And, S33, taking the data from the target position to the end position as the data to be spliced, and splicing the data to be spliced ​​with the next data to be processed of the data to be processed.

[0080] Among them, the above-mentioned line break refers to the spacing symbol between data rows, which usually appears at the end of each data row, and the data processing operation is often performed according to the data row, and the preset data size may not be consistent with the size of the data row. As a result, the acquired data to be processed does not end with a line break in some cases. In order to ensure the normal implementation of the data processing operation, the embodiment of the present invention can parse each data to be processed.

[0081] Specifically, the line breaks included in the data to be processed can be traversed, and the position of the line break that does not exist between the end position of the data to be processed and is located is determined as the target position, that is, the position of the last line break in the data to be processed is determined as the target position. Specifically, the embodiment of the present invention can call the target thread to traverse the data to be processed from the end position, and determine the position of the first line break traversed as the target position.

[0082] Furthermore, after obtaining the target position, the data from the starting position to the target position of the data to be processed can be used as new data to be processed, and the target thread can be called to perform the target processing operation on the new data to be processed, so that it can be determined that the new data to be processed ends with a newline character.

[0083] The next data to be processed of the above-mentioned data to be processed refers to the first data to be processed obtained after the data to be processed is obtained, that is, the data to be processed and the next data to be processed are continuous in the target address. Further, the data from the target position to the end position of the data to be processed can be used as the data to be spliced, and the data to be spliced ​​and the next data to be processed are spliced ​​to obtain the new next data to be processed, so that data omission can be avoided and the continuity of the new data to be processed is also guaranteed.

[0084] For example, if the position of the last line break in the data to be processed with a data size of m is n, the target thread can process the data at positions 0 to n as new data to be processed. At the same time, the data n to m are used as the data to be spliced ​​and spliced ​​with the next data to be processed. Specifically, the splicing operation can be splicing the data to be spliced ​​to the head of the next data to be processed.

[0085] In an embodiment of the present invention, the target thread is called to determine a target position in the data to be processed; the target position is a position in the data to be processed where there is no line break between the end position of the data to be processed and the end position of the data to be processed; the data from the start position of the data to be processed to the target position is used as new data to be processed, and the target thread is called to perform the target processing operation on the new data to be processed; and the data from the target position to the end position is used as data to be spliced, and the data to be spliced ​​is spliced ​​with the next data to be processed of the data to be processed. In this way, by processing the data to be processed through the line break position, it can be determined that the processed new data to be processed ends with a line break, thereby ensuring the normal implementation of subsequent data processing operations and ensuring the data processing effect.

[0086] Optionally, before the above operation of generating a data processing result based on the processing sub-results corresponding to each of the data to be processed, the embodiment of the present invention may further include:

[0087] S41 . In response to a data write request, using a preset data write module, write each of the processing sub-results to a write address indicated by the data write request.

[0088] The write address may be a cloud storage address or a local storage address, which is not limited in the embodiment of the present invention. The data write request may be automatically generated according to a certain period or may be actively sent by an external device, which is not limited in the embodiment of the present invention.

[0089] Specifically, the data write request may carry a write address, and then the write address may be obtained from the data write request, and the processing sub-results corresponding to each piece of data to be processed may be written to the write address. Specifically, the write address may be configured as a corresponding address parameter in the data write module, and then the data may be written through the data write module.

[0090] The data writing module may be pre-set and used to perform data writing operations. Taking the Prometheus database as an example, the data writing module may be a remote writing module (remote-write).

[0091] For example, taking the monitoring service of the Prometheus database as an example, Prometheus monitoring is performed by requesting the http service exposed by the monitored object, parsing the request result into indicators, and finally storing the indicators. When the amount of indicators exposed by the monitored object is particularly large, Prometheus will occupy a large amount of memory during the pulling process, and even timeouts may occur during processing, resulting in the inability to monitor the target object. Figure 3is a schematic diagram of an interaction of data processing in an embodiment of the present invention, such as Figure 3 As shown, targets is the monitoring object, that is, the target address, http-client is the sending and receiving module in the Prometheus database, http-read is the data reading module, which is used to call the read method to read data, parser is the thread used to perform target processing operations, and remote-write is the data writing module.

[0092] Specifically, when the data to be processed (response body) is particularly large, the Prometheus crawling task can configure the data block parameter to be valid (i.e., make blockParse = true), and then the data can be read according to the data block. After Prometheus obtains the target address (target), it can periodically initiate http requests, and targets can return the corresponding data. At the same time, the embodiment of the present invention can call the read method to read data from the response body in a loop until the data is read (until read returns io.EOF to indicate that the reading is complete). The size of the read method is 64KB, so 64KB is read at a time by default, and the read result is sent to the parser thread pool for processing.

[0093] Furthermore, the parser thread pool can be pre-created and initialized, and can create 2 threads by default, equal to the number of CPU cores. Furthermore, after receiving a result sent by a reader, the thread pool immediately schedules a thread to process the result and sends the result to remote-write. After all results are processed, the processing time is recorded. Furthermore, after each parser thread processes the data, it sends it to the remote storage queue to write the processed result to the remote storage address.

[0094] In the embodiment of the present invention, the block processing mode is used to send data to the thread pool for asynchronous processing, so that the capture of large-scale data targets can be completed within the specified capture cycle, and the large-scale data exposed by a single indicator can be processed, while reducing the memory usage of the Prometheus service. At the same time, by setting the data processing request, it can be configured for a specific capture task. In addition, the problem of large memory consumption and long data processing time when Prometheus pulls a single large data can be solved. The block processing mode is used to read the data returned by the target address, and the asynchronous thread pool mode is used for data processing, so that the block processing mode is used to send the data to the thread pool for asynchronous processing, which can improve the processing performance and reduce memory consumption. And through further testing, the above method can greatly reduce the memory consumption. When there is 100M data at the target address, the default processing method of Prometheus needs to occupy at least 100M memory. The data processing method of the embodiment of the present invention uses less than 1M memory, and the processing performance is improved by 2 times the number of CPU cores, and the processing performance is greatly improved. In addition, it avoids the situation where the data at the target address is too large to be captured, resulting in the inability to monitor the monitored object, and can realize the monitoring or pulling of large-scale data.

[0095] The data processing method provided by the embodiment of the present invention determines the target address indicated by the data processing request in response to the data processing request; obtains data of a preset data size from the target address as the data to be processed, until the data in the target address are all the data to be processed, and performs target processing operations on the data to be processed respectively to obtain processing sub-results corresponding to each of the data to be processed; the target processing operations are the data processing operations indicated by the data processing request; and generates data processing results based on the processing sub-results corresponding to each of the data to be processed. In the embodiment of the present invention, by obtaining the data to be processed according to the preset data size and performing target processing operations on the data to be processed respectively, the data to be processed can be obtained from the target address in a block processing mode, which reduces the memory consumption caused by data acquisition and improves the data acquisition performance. At the same time, by performing target processing operations on the data to be processed respectively, the data can be processed in blocks, thereby improving the data processing efficiency to a certain extent.

[0096] Figure 4 FIG. 1 is a schematic diagram of the structure of a data processing device in an embodiment of the present invention. Figure 4 As shown, the device 20 may include:

[0097] A first determining module 201, configured to determine, in response to a data processing request, a target address indicated by the data processing request;

[0098] The acquisition module 202 is used to acquire data of a preset data size from the target address as the data to be processed, until all the data in the target address are the data to be processed, and respectively perform target processing operations on the data to be processed to obtain processing sub-results corresponding to each of the data to be processed; the target processing operation is the data processing operation indicated by the data processing request;

[0099] The generating module 203 is used to generate a data processing result based on the processing sub-result corresponding to each of the data to be processed.

[0100] Optionally, the acquisition module includes:

[0101] A selection submodule, for selecting a thread from the threads to be used included in the thread pool as a target thread for any of the to-be-processed data;

[0102] A calling submodule, used for determining the target thread as a used thread, and calling the target thread to perform the target processing operation on the data to be processed;

[0103] The second determining submodule is used to determine the target thread as the thread to be used.

[0104] Optionally, the calling submodule is specifically used to:

[0105] Calling the target thread to determine a target position in the data to be processed; the target position is a position where a line break character is located in the data to be processed and where there is no line break character between the data to be processed and the end position of the data to be processed;

[0106] Taking the data from the starting position of the to-be-processed data to the target position as new to-be-processed data, and calling the target thread to perform the target processing operation on the new to-be-processed data;

[0107] And, the data from the target position to the end position is used as data to be spliced, and the data to be spliced ​​is spliced ​​with the next data to be processed of the data to be processed.

[0108] Optionally, the device further comprises:

[0109] The writing module is used to respond to the data writing request and adopt the preset data writing module to write each of the processing sub-results to the writing address indicated by the data writing request.

[0110] Optionally, the generating module is specifically used to:

[0111] The processing sub-results corresponding to the data to be processed are spliced ​​in a target order to obtain a data processing result; the target order is the order in which the data to be processed are obtained.

[0112] Optionally, the device is applied to a Prometheus database.

[0113] In summary, the data processing device provided in the embodiment of the present invention determines the target address indicated by the data processing request by responding to the data processing request; obtains data of a preset data size from the target address as the data to be processed, until the data in the target address are all the data to be processed, and performs target processing operations on the data to be processed respectively to obtain processing sub-results corresponding to each of the data to be processed; the target processing operations are the data processing operations indicated by the data processing request; and generates data processing results based on the processing sub-results corresponding to each of the data to be processed. In the embodiment of the present invention, by obtaining the data to be processed according to the preset data size and performing target processing operations on the data to be processed respectively, the data to be processed can be obtained from the target address in a block processing mode, which reduces the memory consumption caused by data acquisition and improves the data acquisition performance. At the same time, by performing target processing operations on the data to be processed respectively, the data can be processed in blocks, thereby improving the data processing efficiency to a certain extent.

[0114] The embodiment of the present invention further provides an electronic device, such as Figure 5 As shown, it includes a processor 9001, a communication interface 9002, a memory 9003 and a communication bus 9004, wherein the processor 9001, the communication interface 9002, and the memory 9003 communicate with each other through the communication bus 9004.

[0115] Memory 9003, used for storing computer programs;

[0116] The processor 9001 is used to implement the data processing method provided in the above embodiment when executing the program stored in the memory 9003.

[0117] The data processing method provided by the embodiment of the present invention determines the target address indicated by the data processing request in response to the data processing request; obtains data of a preset data size from the target address as the data to be processed, until the data in the target address are all the data to be processed, and performs target processing operations on the data to be processed respectively to obtain processing sub-results corresponding to each of the data to be processed; the target processing operations are the data processing operations indicated by the data processing request; and generates data processing results based on the processing sub-results corresponding to each of the data to be processed. In the embodiment of the present invention, by obtaining the data to be processed according to the preset data size and performing target processing operations on the data to be processed respectively, the data to be processed can be obtained from the target address in a block processing mode, which reduces the memory consumption caused by data acquisition and improves the data acquisition performance. At the same time, by performing target processing operations on the data to be processed respectively, the data can be processed in blocks, thereby improving the data processing efficiency to a certain extent.

[0118] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0119] The communication interface is used for communication between the above terminal and other devices.

[0120] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0121] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0122] In another embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores instructions. When the instructions are executed on a computer, the computer executes the data processing method described in any one of the above embodiments.

[0123] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes the data processing method described in any one of the above embodiments.

[0124] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0125] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0126] It should be noted that the various data-related processes in the embodiments of the present application are all carried out in compliance with the corresponding data protection laws and policies of the country where the device is located, and with the authorization given by the owner of the corresponding device.

[0127] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0128] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A data processing method, characterized in that: The method comprises: In response to a data processing request, determining a target address indicated by the data processing request; Acquire data of a preset data size from the target address as the data to be processed, until all the data in the target address are the data to be processed, and respectively perform target processing operations on the data to be processed to obtain processing sub-results corresponding to the data to be processed; the target processing operations are the data processing operations indicated by the data processing request; A data processing result is generated based on the processing sub-results corresponding to each of the data to be processed.

2. The method according to claim 1, characterized in that The performing target processing operations on the data to be processed respectively includes: For any of the data to be processed, selecting a thread from the threads to be used included in the thread pool as a target thread; Determine the target thread as a used thread, and call the target thread to perform the target processing operation on the data to be processed; The target thread is determined as a thread to be used.

3. The method according to claim 2, characterized in that The calling of the target thread to perform the target processing operation on the data to be processed includes: Calling the target thread to determine a target position in the data to be processed; the target position is a position where a line break character is located in the data to be processed and where there is no line break character between the data to be processed and the end position of the data to be processed; Taking the data from the starting position of the to-be-processed data to the target position as new to-be-processed data, and calling the target thread to perform the target processing operation on the new to-be-processed data; And, the data from the target position to the end position is used as data to be spliced, and the data to be spliced ​​is spliced ​​with the next data to be processed of the data to be processed.

4. The method according to claim 1, characterized in that Before generating the data processing result based on the processing sub-results corresponding to the data to be processed, the method further includes: In response to a data write request, a preset data write module is used to write each of the processing sub-results to the write address indicated by the data write request.

5. The method according to any one of claims 1 to 4, characterized in that: The generating of the data processing result based on the processing sub-results corresponding to the data to be processed includes: The processing sub-results corresponding to the data to be processed are spliced ​​in a target order to obtain a data processing result; the target order is the order in which the data to be processed are obtained.

6. The method according to any one of claims 1 to 4, characterized in that: The method is applied to the Prometheus database.

7. A data processing device, characterized in that: The device comprises: A first determining module, configured to determine, in response to a data processing request, a target address indicated by the data processing request; an acquisition module, configured to acquire data of a preset data size from the target address as data to be processed, until all the data in the target address are data to be processed, and to perform target processing operations on the data to be processed respectively to obtain processing sub-results corresponding to the data to be processed; the target processing operations are the data processing operations indicated by the data processing request; A generating module is used to generate a data processing result based on the processing sub-results corresponding to each of the data to be processed.

8. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-6 when executing a program stored in a memory.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program, characterized in that When the computer program is executed by a computer, the method according to any one of claims 1 to 6 is implemented.

Images