Data processing method, device, system and server

Abstract

The present disclosure relates to a data processing method, device, system and server, the data processing method comprising: receiving a data query request sent by a client; obtaining large object data corresponding to the data query request; splitting the large object data into dimension data under each data dimension according to a plurality of preset data dimensions, and storing the dimension data on a server; and sending the dimension data under the data dimension corresponding to the data query request to the client. The data processing method, device, system and server according to the present disclosure can solve the problems of low query efficiency and long data loading time in the data query process. By splitting the data and performing local storage, the data loading time of the data query can be reduced, and the data query efficiency can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a data processing method, device, system and server. Background Art

[0002] With the development of information technology and Internet technology, the amount of information has increased rapidly, resulting in larger and larger data storage and more complex data types. In the process of data call, it is usually necessary to find the required data among a large amount of coupled data.

[0003] Specifically, in the related technology, when it is necessary to query data of a specific type / attribute, it is usually necessary to traverse all the stored data to extract the data of the required type / attribute from these data. In such a data query method, the query efficiency is low and the data loading time will be very long, especially when facing large-scale data calls and computing tasks, it may even cause problems such as freezes and query errors. Summary of the invention

[0004] The present disclosure provides a data processing method, device, system and server to at least solve the problems of low query efficiency and long data loading time in the data query process in the related technology. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided a data processing method, which is applied to a server, and the data processing method includes receiving a data query request sent by a client; obtaining large object data corresponding to the data query request, wherein the large object data includes data of multiple dimensions coupled together; splitting the large object data into dimensional data under each data dimension according to multiple preset data dimensions, and storing the dimensional data on the server; and sending the dimensional data under the data dimension corresponding to the data query request to the client.

[0006] Optionally, the step of storing the dimensional data on the server includes: serializing each dimensional data and storing the serialized dimensional data on the server, wherein the step of sending the dimensional data under the data dimension corresponding to the data query request to the client includes: deserializing the serialized dimensional data under the data dimension corresponding to the data query request, and sending the deserialized dimensional data to the client.

[0007] Optionally, the step of obtaining the large object data corresponding to the data query request includes: based on the data query request, searching for dimension data corresponding to the data dimension of the data query request on the server; if the corresponding dimension data is not found, executing the step of obtaining the large object data corresponding to the data query request.

[0008] Optionally, the server includes a first memory and a second memory, wherein the step of storing the serialized dimensional data on the server includes: storing the serialized dimensional data in the second memory, compressing the serialized dimensional data, and storing the compressed dimensional data in the first memory.

[0009] Optionally, the server is communicatively connected to a distributed file system, wherein the step of obtaining large object data corresponding to the data query request includes: if compressed dimension data corresponding to the data dimension of the data query request is not found in the first memory, obtaining the large object data corresponding to the data query request from the distributed file system.

[0010] Optionally, the data query request includes a historical job information query request, and the preset data dimensions include a task data dimension, a job data dimension, an aggregate data dimension, and a job statistical data dimension, wherein, according to the preset data dimension, according to the preset multiple data dimensions, the large object data is split into dimensional data under each data dimension, and the step of storing the dimensional data on the server includes: for the task data dimension, the job data dimension, the aggregate data dimension, and the job statistical data dimension, the large object data is split into task data, job data, aggregate data, and job statistical data, and the task data, the job data, the aggregate data, and the job statistical data are stored in the second memory.

[0011] Optionally, the data processing method also includes: if compressed dimensional data corresponding to the data dimension of the data query request is found in the first memory, decompressing the corresponding compressed dimensional data from the second memory, deserializing the decompressed dimensional data, and sending the deserialized dimensional data to the client.

[0012] According to a second aspect of an embodiment of the present disclosure, there is provided a data processing device, which is applied to a server, and the data processing device includes: a search unit, configured to receive a data query request sent by a client; an acquisition unit, configured to acquire large object data corresponding to the data query request, wherein the large object data includes data of multiple dimensions coupled together; a splitting unit, configured to split the large object data into dimensional data under each data dimension according to multiple preset data dimensions, and store the dimensional data on the server; and a sending unit, configured to send the dimensional data under the data dimension corresponding to the data query request to the client.

[0013] Optionally, the splitting unit is further configured to: serialize each dimension data and store the serialized dimension data on the server, wherein the sending unit is further configured to: deserialize the serialized dimension data under the data dimension corresponding to the data query request, and send the deserialized dimension data to the client.

[0014] Optionally, the acquisition unit is further configured to: based on the data query request, search on the server for dimension data corresponding to the data dimension of the data query request; if no corresponding dimension data is found, execute the step of acquiring large object data corresponding to the data query request.

[0015] Optionally, the server includes a first memory and a second memory, wherein the splitting unit is further configured to: store the serialized dimensional data in the second memory, compress the serialized dimensional data, and store the compressed dimensional data in the first memory.

[0016] Optionally, the server is communicatively connected to a distributed file system, wherein the acquisition unit is further configured to: if compressed dimensional data corresponding to the data dimension of the data query request is not found in the first memory, then acquire large object data corresponding to the data query request from the distributed file system.

[0017] Optionally, the data query request includes a historical job information query request, and the preset data dimensions include a task data dimension, a job data dimension, an aggregate data dimension, and a job statistical data dimension, wherein the splitting unit is further configured to: for the task data dimension, the job data dimension, the aggregate data dimension, and the job statistical data dimension, split the large object data into task data, job data, aggregate data, and job statistical data, and store the task data, the job data, the aggregate data, and the job statistical data in the second memory.

[0018] Optionally, the acquisition unit is also configured to: if compressed dimensional data corresponding to the data dimension of the data query request is found in the first memory, decompress the corresponding compressed dimensional data from the second memory, deserialize the decompressed dimensional data, and send the deserialized dimensional data to the client.

[0019] According to a third aspect of an embodiment of the present disclosure, a server is provided, comprising: a processor; and a memory for storing instructions executable by the processor, wherein the processor executable instructions, when executed by the processor, prompt the processor to execute a data processing method according to an embodiment of the present disclosure.

[0020] According to a fourth aspect of an embodiment of the present disclosure, a data processing system is provided, wherein the data processing system includes a distributed file system and a server according to an embodiment of the present disclosure.

[0021] According to a fifth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of a server, the server is enabled to execute a data processing method according to an embodiment of the present disclosure.

[0022] According to a sixth aspect of an embodiment of the present disclosure, a computer program product is provided, and when the computer instructions are executed by a processor, the data processing method according to the embodiment of the present disclosure is implemented.

[0023] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0024] The large object data corresponding to the data query request can be split according to the data dimensions, and the dimensional data of each dimension can be stored on the server, so that in response to the data query request, the data of the corresponding data dimension in the split data can be sent without searching in the large object data, thereby reducing the data loading time of the data query and improving the data query efficiency.

[0025] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0027] Figure 1 The flowchart of a data processing method in an example application scenario is shown according to an exemplary embodiment.

[0028] Figure 2 The invention is a flowchart of a data processing method according to an exemplary embodiment.

[0029] Figure 3 The present invention is a flowchart showing the steps of acquiring and processing large object data in a data processing method according to an exemplary embodiment.

[0030] Figure 4 is a flow chart showing an existing data query and feedback method.

[0031] Figure 5 It is a block diagram of a data processing device according to an exemplary embodiment.

[0032] Figure 6 It is a block diagram of a server according to an exemplary embodiment.

[0033] Figure 7 It is a block diagram of a data processing system according to an exemplary embodiment. DETAILED DESCRIPTION

[0034] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0035] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0036] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0037] It should be noted that the user information involved in this disclosure (including but not limited to user device information, user personal information, etc.) is all information authorized by the user or fully authorized by all parties.

[0038] It should also be noted that, taking the application scenario of the historical operation information query service described below as an example, the data query can be a query of data related to historical tasks and historical operation information. However, although the application scenario of the historical operation information query service is described below as an example, it should be understood that the application scenario of the data processing method, device, system and server according to the present disclosure is not limited to this, and it can also be applied to any other application scenario involving data query, processing and / or storage.

[0039] It should also be noted that the "server" described in the context of this disclosure may refer to any execution subject that can execute the data processing method described below. In the application scenario of the historical job information query service, the server may be a historical server. However, the above execution subject of this disclosure is not particularly limited, and it may be any type of server that can manage computing resources.

[0040] A data processing method, a data processing apparatus, a server, a data processing system, a computer-readable storage medium, and a computer program product according to exemplary embodiments of the present disclosure will be described below with reference to the accompanying drawings.

[0041] Figure 1 is a flow chart of a data processing method in an example application scenario according to an exemplary embodiment. Specifically, Figure 1 The process of data query and processing in an exemplary application scenario of the historical job information query service is shown. Figure 1 The data processing method of the exemplary embodiment of the present disclosure is generally described as a whole. It should be noted that: Figure 1 The steps / devices in the embodiment are merely exemplary steps / devices in an exemplary embodiment of the present disclosure. In another exemplary embodiment of the present disclosure, one or more of these steps / devices may be omitted or replaced, and / or other steps / devices in addition to these steps / devices may be added.

[0042] like Figure 1 As shown, one or more of the multiple clients (Client) may send a data query request, such as a remote procedure call (RPC) request, to a client service (clientService) (step S1).

[0043] The customer service sends a query instruction to the cached history storage (CachedHistoryStorage) based on the data query request (step S2).

[0044] The cache history storage may search in the memory cache (MemoryCache) whether there is data corresponding to the data query request (step S4).

[0045] If the corresponding data is not found in step S4, on the one hand, the cache history memory can send an instruction to fetch index info to the hs manager (hsManager) (step S3). The hs manager gets data from the job list cache (JobListCache) (step S31), so that the job list cache calls back (fallback to hdfs) data from the Hadoop distributed file system (HDFS) (step S311), and HDFS puts back (put back) the large object data to the job list cache in response to the callback instruction (step S312).

[0046] On the other hand, the memory cache can fallback to leveldb data from server local storage such as LevelDB (step S41). The server local storage can fallback to hdfs data from HDFS (step S411). The large object data from HDFS can be split and serialized (divide and serialize big objects) (step S412), and the split and serialized data can be returned to the server local storage as a whole (serialize whole and put back) (step S413). The server local storage can provide at least a portion of the returned data to the memory cache. For example, according to the job data query request sent by the client, the basic job information split from the large object data can be first deserialized to the memory cache (deserialize job info first) (step S42).

[0047] The memory cache can provide data split from the large object data through corresponding interfaces. For example, it can provide deserialized basic job information (deserialize job info) (step 51) through the GetJobReport interface, it can provide deserialized aggregation information (desrialize counters) (step 52) through the GetCounters interface, and it can provide deserialized task information (desrialize taskMap) (step 53) through the GetTaskCompeletionEvents interface.

[0048] pass Figure 1The data processing method shown can split the large object data corresponding to the data query request according to the data dimensions, and store the dimensional data of each dimension on the server, so that in response to the data query request, the data of the corresponding data dimension in the split data can be sent without having to search in the large object data, thereby reducing the data loading time of the data query and improving the data query efficiency.

[0049] Figure 2 is a flow chart of a data processing method according to an exemplary embodiment. The data processing method is used in a server, such as Figure 2 As shown, the data processing method may include the following steps:

[0050] In step S11, a data query request sent by a client may be received.

[0051] In this step, the data query request is information sent by the client to the server to indicate the required data, which may carry the characteristics of the data to be queried. The data query request may be, for example, an RPC request.

[0052] Here, the "data query request" described in the exemplary embodiment of the present disclosure may vary according to different application scenarios of the present disclosure, wherein "data query" may be a query on data from any feature level such as data type, attribute, annotation, structure or granularity, and the content of the data query may be changed according to the required information content. Taking the application scenario of the historical job information query service as an example, the data query request may include a historical job information query request for querying data related to historical tasks and historical operation information.

[0053] In step S12, large object data corresponding to the data query request may be obtained.

[0054] In this step, the large object data may refer to raw data including data of multiple dimensions coupled together. For example, the large object data may refer to unstructured data having a length exceeding a predetermined length limit, where the predetermined length may depend on the application scenario, and accordingly, the large object data may also represent different data according to the application scenario, for example, in the application scenario of the historical job information query service, the large object data may be Fulljob large object data, however, the present disclosure is not limited thereto, and the large object data described in the present disclosure may be any data that can be split according to any data dimension.

[0055] As an example, in step S12, based on the data query request from the client, dimensional data corresponding to the data dimension of the data query request can be searched locally on the server. If the corresponding dimensional data is not found, the large object data corresponding to the data query request can be obtained.

[0056] Here, data dimension refers to a specific feature of data, such as data type, attribute, annotation, structure or granularity. Data dimension can refer to qualitative dimension, such as data type; it can also refer to quantitative dimension, such as data value. For qualitative dimension data, it can be divided into different data dimensions according to the characteristics of the data; for quantitative dimension data, it can be divided into different data dimensions according to the numerical grouping. Each data dimension has corresponding dimensional data.

[0057] As an example, the operation of searching for corresponding dimension data may be implemented by searching for meta information of the dimension data, and it may be determined whether the corresponding dimension data is stored on the server according to the meta information.

[0058] In step S12, the data dimension of the data requested by the data query request can be determined, and the data of the data dimension can be queried on the server. If the data is not found, it can be considered that the large object data corresponding to the data requested by the data query request has not been stored on the server, and the large object data can be obtained from the outside.

[0059] As an example, the server may be connected to a distributed file system (DFS) for communication to obtain large object data from the distributed file system. Here, the distributed file system may be, for example, a Hadoop distributed file system (HDFS). Although the present disclosure describes that the server obtains large object data from a distributed file system such as HDFS, the present disclosure is not limited thereto, and the large object data may also be obtained from any location.

[0060] Here, if dimension data corresponding to the data dimension of the data query request is found locally on the server, the corresponding dimension data may be sent to the client in response to the data query request, that is, step S14 described below is executed.

[0061] In step S13, the large object data may be split into dimension data under each data dimension according to a plurality of preset data dimensions, and the dimension data may be stored on the server.

[0062] In this step, multiple data dimensions can be pre-set according to the application scenario and / or actual needs. Taking the application scenario of historical job information query service as an example, the preset data dimensions may include task data dimension, job data dimension, counters data dimension and job statistics data dimension.

[0063] After obtaining the large object data, the server may split the large object data according to preset data dimensions to obtain dimensional data under each data dimension.

[0064] The server can store each dimension data after splitting on the server. For example, each dimension data can be serialized and the serialized dimension data can be stored on the server. In this way, by serializing and storing dimension data, the dimension data can be decomposed into a byte stream. When the dimension data is subsequently called, the stored byte stream can be directly read through deserialization and the dimension data can be reconstructed. During the storage process, there is no need to write any information to explain the data content, which improves the storage and reading efficiency of the dimension data.

[0065] As an example, the server may include a first memory and a second memory, the first memory may be a non-persistent memory, such as a memory cache, and the second memory may be a persistent memory, such as a local database, such as a kv database using LevelDB.

[0066] In this case, the serialized dimensional data may be stored in the second memory, and the serialized dimensional data may be compressed, and the compressed dimensional data may be stored in the first memory.

[0067] Here, further, a part of the dimensional data in the compressed dimensional data can be stored in the first memory, and this part of the dimensional data can be data of a data dimension with a higher call frequency, for example, its call frequency can be greater than or equal to a preset call threshold, and the rest of the data in the compressed dimensional data can be stored in the second memory, which can be data of a data dimension with a lower call frequency, for example, its call frequency can be less than the preset call threshold, and when this part of the data is called, it can be obtained from the second memory via the first memory. In this way, the storage space in the first memory can be saved while maintaining a high query hit rate for the first memory.

[0068] Taking the historical job information query service as an example, only the data of the job dimension that is called frequently can be stored in the first memory, and the data of the task dimension and the aggregation dimension can be stored in the second memory. In the case of caching only the job dimension data in the first memory, the cache hit rate can be increased from 50% to 90% due to the reduction of data to be cached and the increase of the number of cache records in the case of caching the dimension data of all dimensions.

[0069] As an example, Figure 3As shown, in the case where the server includes a first memory and a second memory, in the step S12 described above, compressed dimensional data corresponding to the data dimension of the data query request can be searched in the first memory (step S121); if the compressed dimensional data corresponding to the data dimension of the data query request is not found in the first memory, the large object data corresponding to the data query request can be obtained from the distributed file system (step S122).

[0070] Here, in step S122, taking the application scenario of the historical job information query service as an example, if the corresponding dimension data is not found in the first memory, the hs manager (hsManager) can be requested for the large object data corresponding to the data query request. The manager can search for the large object data from the distributed file system via the job list cache (JobListCache) and feed back the found large object data to the server.

[0071] In this example, the capacity of the second memory can be greater than that of the first memory, the first memory can be used as a first-level cache, and the second memory can be used as a second-level cache. In this way, on the one hand, the second memory can persistently store the split dimension data, and in subsequent queries, it can provide the split dimension data to the first memory, so that it can respond to data query requests more quickly to provide dimension data corresponding to the data dimension of the data query request, reduce query time, and improve query efficiency; on the other hand, the second memory can coordinate the speed between the first memory and the internal memory, and can be used as a space for temporary data exchange between the first memory and the internal memory.

[0072] As an example, in the application scenario of historical job information query service, step S13 may include: for the task data dimension, job data dimension, aggregate data dimension and job statistical data dimension, splitting the large object data into task data (TaskMap), job data (JobInfo), aggregate data (Counters) and job statistical data (JobAnalysis), and storing the task data, job data, aggregate data and job statistical data in the second memory.

[0073] Specifically, the task data may include information under the task data dimension, for example, it may include a mapTask information list, a reduceTask information list and / or attempt information.

[0074] The job data may include information under the job data dimension, for example, it may include basic information of the job such as start, stop, status, map / reduce quantity, etc.

[0075] Aggregate data may include information under the aggregate data dimension, for example, it may include totalcounters information, mapCounters information obtained according to mapTask information, reduecCounters information obtained according to reduceTask information, etc.

[0076] The job statistics data may include information under the job statistics data dimension, which may include the statistics information required by the web end, for example, it may include the summary page information of the web end, the basic indicator display of the task operation, etc. The job statistics data may be obtained from the task data, for example, it may be obtained by performing a cyclic calculation based on the mapTask information list or the reduceTask information list.

[0077] It is worth noting that before the present disclosure, in large object data, since one or more types of data are coupled together, when a data query request requests data with specific characteristics, it is necessary to traverse all data in the huge and complex large object data to extract the required data.

[0078] Taking the application scenario of the historical job information query service described above as an example, a complete job history information contains data of the coupled job dimension and task dimension. Each job contains multiple tasks. For example, in some large-scale data calculations, there may be more than 1 million tasks in a job, and the size of the corresponding historical data may exceed 30GB. In the existing query method, each query of a complete job history information requires obtaining all the data of the job dimension and task dimension, and then extracting the corresponding data from these data, which makes data loading time-consuming. Due to the large amount of data, it may cause service garbage collection (GC), which may lead to service unavailability.

[0079] In this regard, as described in the above exemplary embodiments of the present disclosure, the present disclosure adopts a method of splitting large object data by data dimension and storing the split dimensional data in the server, which can perform dimensional processing on large object data obtained from data storage such as HDFS, pre-calculate all preset data dimensions, and when a data query request is received later, the corresponding dimensional data can be directly extracted from the split data, and can be directly provided to the client from the server. In this way, the situation where each query needs to traverse all data for computational waste can be minimized, the work efficiency of the processor and memory can be improved, and the service performance and stability can be improved.

[0080] Return to reference Figure 2 In step S14, dimension data under the data dimension corresponding to the data query request may be sent to the client.

[0081] As described above, in response to the data query request sent by the client, the dimensional data corresponding to the data dimension of the data query request can be searched on the server locally, and when the corresponding dimensional data is not found, the corresponding large object data can be obtained (step S12). Here, if the dimensional data corresponding to the data dimension of the data query request is found on the server locally, the corresponding dimensional data can be sent to the client in response to the data query request.

[0082] In the case where the dimension data is serialized and stored, in step S14, the serialized dimension data under the data dimension corresponding to the data query request may be deserialized and sent to the client. Here, as an example, the dimension data may be deserialized through a GET statement.

[0083] As mentioned above, in the process of deserializing dimension data, the stored byte stream can be directly read and the dimension data can be reconstructed, which has a faster reading speed and reduces query time.

[0084] As an example, in the case where the server includes a first memory and a second memory, such as Figure 3 As shown, the corresponding dimension data can be searched in the first memory. If the compressed dimension data corresponding to the data dimension requested by the data query is found in the first memory, step S141 can be executed. Specifically, the corresponding compressed dimension data can be decompressed from the second memory, the decompressed dimension data can be deserialized, and the deserialized dimension data can be sent to the client.

[0085] In addition, as an example, the data processing method according to the exemplary embodiment of the present disclosure may further include: according to a plurality of preset data dimensions, setting a plurality of data query interfaces respectively corresponding to the plurality of data dimensions.

[0086] In this step, when receiving the data query request sent by the client, in order to quickly identify the data dimensions of the data query request and improve the efficiency of the data query request, a data query interface corresponding to multiple data dimensions can be set, so that in response to data query requests from different data query interfaces, the dimension data corresponding to the interface can be provided.

[0087] Taking the application scenario of historical job information query service as an example, the data query interface may include a task data query interface (GetTaskCompletionEvents), a job data query interface (GetJobReport) and an aggregate data query interface (GetCounters), wherein the task data query interface may receive data query requests for task data dimensions, the job data query interface may receive data query requests for job data dimensions and job statistical data dimensions, and the aggregate data query interface may receive data query requests for aggregate data dimensions.

[0088] An exemplary embodiment of a data processing method according to the present disclosure is described above. In this method, large object data can be split according to data dimensions, and the data of each dimension is pre-calculated and stored on the server. Each subsequent query can quickly feed back the corresponding data by accurately locating the queried data dimension on the server without traversing all the large object data for calculation and extraction. This is very helpful in improving query efficiency in information query services.

[0089] Taking the historical job information query service as an example, most query requests require some basic information about the job, that is, job dimension data, which can account for more than 80%, while a few query requests require information on the task dimension and aggregation dimension. For example, query requests for the aggregation dimension may only account for about 5%. In this case, when the data dimension of the data query request is the job data dimension, the job dimension data can be directly extracted and sent from the split data stored on the server. This process does not involve task dimension and aggregation dimension data, which greatly improves the data query speed.

[0090] For example, Figure 4 A flowchart of the existing data query and feedback method is shown. The history server (HistoryServer) can provide a job history service (jobHistoryService), a customer service (clientService) and an aggLogDel service (aggLogDelService), wherein the customer service can query data through the job history service (step S01), and then the job history service can query the cached history storage (CachedHistoryStorage) for the data (step S02). The cached history storage searches for the corresponding data in the cache (step S04). If it is not found, it sends an extraction list to the hs manager (hsManager) (step S03), and the hs manager further sends the list to HDFS, thereby, the cache can penetrate the hs manager (step S05) to obtain the large object data.

[0091] Compared with the above-mentioned existing query methods, for responses to job data and job statistics data queries, while the overall average time consumption of the existing query method is about 2.62 seconds, the average time consumption of the method according to the exemplary embodiment of the present disclosure is about 374 milliseconds, which is shortened to 14% of the average time consumption of the existing query method; for responses to aggregate data queries, while the average time consumption of the existing query method is about 596 milliseconds, the average time consumption of the method according to the exemplary embodiment of the present disclosure is about 30 milliseconds, which is shortened to 5% of the average time consumption of the existing query method; for responses to task data queries, while the average time consumption of the existing query method is about 316 milliseconds, the average time consumption of the method according to the exemplary embodiment of the present disclosure is about 104 milliseconds, which is shortened to 32% of the average time consumption of the existing query method.

[0092] Figure 5 is a block diagram of a data processing device according to an exemplary embodiment. Figure 5 The device includes a searching unit 100, an acquiring unit 200, a splitting unit 300 and a sending unit 400.

[0093] The search unit 100 is configured to receive a data query request sent by a client.

[0094] The acquisition unit 200 is configured to acquire large object data corresponding to the data query request. Here, the large object data includes data of multiple dimensions coupled together.

[0095] The splitting unit 300 is configured to split the large object data into dimension data under each data dimension according to a plurality of preset data dimensions, and store the dimension data on the server.

[0096] The sending unit 400 is configured to send dimension data under the data dimension corresponding to the data query request to the client.

[0097] As an example, the splitting unit 300 may also be configured to: serialize each dimension data, and store the serialized dimension data on the server.

[0098] Correspondingly, the sending unit 400 may also be configured to: deserialize the serialized dimension data under the data dimension corresponding to the data query request, and send the deserialized dimension data to the client.

[0099] As an example, the acquisition unit 200 can also be configured to: based on the data query request, search for dimension data corresponding to the data dimension of the data query request on the server; if the corresponding dimension data is not found, execute the step of acquiring the large object data corresponding to the data query request.

[0100] As an example, the server may include a first memory and a second memory. The splitting unit 300 may also be configured to: store the serialized dimensional data in the second memory, compress the serialized dimensional data, and store the compressed dimensional data in the first memory.

[0101] As an example, the server may be in communication with a distributed file system. The acquisition unit 200 may also be configured to: if no compressed dimension data corresponding to the data dimension of the data query request is found in the first memory, acquire the large object data corresponding to the data query request from the distributed file system.

[0102] As an example, the data query request may include a historical job information query request, and the preset data dimensions may include a task data dimension, a job data dimension, an aggregate data dimension, and a job statistical data dimension.

[0103] The splitting unit 300 can also be configured to: split the large object data into task data, job data, aggregate data and job statistical data dimensions according to the task data dimension, job data dimension, aggregate data dimension and job statistical data dimension, and store the task data, job data, aggregate data and job statistical data in a second memory.

[0104] As an example, the acquisition unit 200 can also be configured as follows: if compressed dimensional data corresponding to the data dimension of the data query request is found in the first memory, the corresponding compressed dimensional data is decompressed from the second memory, the decompressed dimensional data is deserialized, and the deserialized dimensional data is sent to the client.

[0105] Regarding the device in the above embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0106] Figure 6 FIG. 1 is a block diagram of a server according to an exemplary embodiment. Figure 6 As shown, the server 10 includes a processor 101 and a memory 102 for storing processor executable instructions. Here, when the processor executable instructions are executed by the processor, the processor is prompted to execute the data processing method as described in the above exemplary embodiment.

[0107] As an example, the server 10 may be a history server or any type of server capable of managing computing resources. Here, the server 10 is not necessarily a single device, but may also be any collection of devices or circuits capable of executing the above instructions (or instruction sets) individually or in combination. The server 10 may also be part of an integrated control system or system manager, or may be configured as a server interconnected with a local or remote (e.g., via wireless transmission) interface.

[0108] In the server 10, the processor 101 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. By way of example and not limitation, the processor 101 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0109] The processor 101 may execute instructions or codes stored in the memory 102, wherein the memory 102 may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.

[0110] The memory 102 may be integrated with the processor 101, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 102 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 102 and the processor 101 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 101 can read files stored in the memory 102.

[0111] In addition, the server 10 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the server 10 may be connected to each other via a bus and / or a network.

[0112] Figure 7 FIG. 1 is a block diagram of a data processing system according to an exemplary embodiment. Figure 7 As shown, the data processing system 1 includes a server 10 and a distributed file system 20 as described in the above exemplary embodiments.

[0113] The server 10 may communicate with the distributed file system 20 , for example, via an I / O port, a network connection, etc., so that the server 10 can read files stored in the distributed file system 20 .

[0114] As an example, the distributed file system 20 may be a Hadoop distributed file system (HDFS), but exemplary embodiments of the present disclosure are not limited thereto, and the distributed file system 20 may also be any distributed file storage system that stores large object data.

[0115] In an exemplary embodiment, a computer-readable storage medium may also be provided, and when the instructions in the computer-readable storage medium are executed by the processor of the server, the server can execute the data processing method as described in the above exemplary embodiment. The computer-readable storage medium may be, for example, a memory including instructions. Optionally, the computer-readable storage medium may be: a read-only memory (ROM), a random access memory (RAM), a random access programmable read-only memory (PROM), an electrically erasable programmable read-only memory (EEPROM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a flash memory, a non-volatile memory, a CD-ROM, a CD-R, a CD+R, a CD-RW, a CD+RW, a DVD-ROM, a DVD-R, a DVD+R, a DVD-RW, a DVD+RW, a DVD-RAM, a BD-ROM, a B D-R, BD-RLTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0116] In an exemplary embodiment, a computer program product may also be provided. The computer program product includes computer instructions. When the computer instructions are executed by a processor, the data processing method as described in the exemplary embodiment above is implemented.

[0117] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0118] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A data processing method, characterized in that: Applied to a server, the data processing method includes: Receive data query requests sent by the client; Acquire large object data corresponding to the data query request, wherein the large object data includes data of multiple dimensions coupled together; According to a plurality of preset data dimensions, the large object data is split into dimension data under each data dimension, and the dimension data is stored on a server, wherein the data dimension includes a task data dimension, an operation data dimension, an aggregate data dimension and an operation statistical data dimension, and the dimension data includes task data, operation data, aggregate data and operation statistical data; Sending dimension data under the data dimension corresponding to the data query request to the client.

2. The data processing method according to claim 1, characterized in that: The step of storing the dimension data on the server comprises: Serialize each dimension data and store the serialized dimension data on the server. The step of sending dimension data under the data dimension corresponding to the data query request to the client includes: Deserialize the serialized dimension data under the data dimension corresponding to the data query request, and send the deserialized dimension data to the client.

3. The data processing method according to claim 2, characterized in that: The step of acquiring the large object data corresponding to the data query request includes: Based on the data query request, searching on a server for dimension data corresponding to the data dimension of the data query request; If the corresponding dimension data is not found, the step of obtaining the large object data corresponding to the data query request is executed.

4. The data processing method according to claim 2, characterized in that: The server includes a first memory and a second memory, wherein the step of storing the serialized dimension data on the server includes: The serialized dimensional data is stored in the second memory, and the serialized dimensional data is compressed, and the compressed dimensional data is stored in the first memory.

5. The data processing method according to claim 4, characterized in that: The server is in communication connection with the distributed file system, wherein the step of obtaining the large object data corresponding to the data query request comprises: If the compressed dimension data corresponding to the data dimension of the data query request is not found in the first memory, the large object data corresponding to the data query request is obtained from the distributed file system.

6. The data processing method according to claim 4 or 5, characterized in that: The data query request includes a historical job information query request, The step of splitting the large object data into dimension data under each data dimension according to the preset multiple data dimensions and storing the dimension data on the server includes: With respect to the task data dimension, the job data dimension, the aggregate data dimension and the job statistical data dimension, the large object data is split into the task data, the job data, the aggregate data and the job statistical data, and the task data, the job data, the aggregate data and the job statistical data are stored in the second memory.

7. The data processing method according to claim 4 or 5, characterized in that: The data processing method further includes: If compressed dimension data corresponding to the data dimension of the data query request is found in the first memory, the corresponding compressed dimension data is decompressed from the second memory, the decompressed dimension data is deserialized, and the deserialized dimension data is sent to the client.

8. A data processing device, characterized in that: Applied to a server, the data processing device comprises: A search unit, configured to receive a data query request sent by a client; an acquisition unit, configured to acquire large object data corresponding to the data query request, wherein the large object data includes data of multiple dimensions coupled together; A splitting unit is configured to split the large object data into dimension data under each data dimension according to a plurality of preset data dimensions, and store the dimension data on a server, wherein the data dimensions include a task data dimension, a job data dimension, an aggregate data dimension, and a job statistical data dimension, and the dimension data includes task data, job data, aggregate data, and job statistical data; The sending unit is configured to send dimension data under the data dimension corresponding to the data query request to the client.

9. The data processing device according to claim 8, characterized in that: The splitting unit is further configured to: Serialize each dimension data and store the serialized dimension data on the server. Wherein, the sending unit is further configured as: Deserialize the serialized dimension data under the data dimension corresponding to the data query request, and send the deserialized dimension data to the client.

10. The data processing device according to claim 9, characterized in that: The acquisition unit is further configured to: Based on the data query request, searching on a server for dimension data corresponding to the data dimension of the data query request; If the corresponding dimension data is not found, the step of obtaining the large object data corresponding to the data query request is executed.

11. The data processing device according to claim 9, characterized in that: The server includes a first memory and a second memory, wherein the splitting unit is further configured to: The serialized dimensional data is stored in the second memory, and the serialized dimensional data is compressed, and the compressed dimensional data is stored in the first memory.

12. The data processing device according to claim 11, characterized in that: The server is in communication connection with the distributed file system, wherein the acquisition unit is further configured to: If the compressed dimension data corresponding to the data dimension of the data query request is not found in the first memory, the large object data corresponding to the data query request is obtained from the distributed file system.

13. The data processing device according to claim 11 or 12, characterized in that: The data query request includes a historical job information query request, Wherein, the splitting unit is further configured as: With respect to the task data dimension, the job data dimension, the aggregate data dimension and the job statistical data dimension, the large object data is split into the task data, the job data, the aggregate data and the job statistical data, and the task data, the job data, the aggregate data and the job statistical data are stored in the second memory.

14. The data processing device according to claim 11 or 12, characterized in that: The acquisition unit is further configured to: If compressed dimension data corresponding to the data dimension of the data query request is found in the first memory, the corresponding compressed dimension data is decompressed from the second memory, the decompressed dimension data is deserialized, and the deserialized dimension data is sent to the client.

15. A server, characterized in that: The server comprises: processor; a memory for storing instructions executable by the processor, When the processor-executable instructions are executed by the processor, the processor is prompted to execute the data processing method according to any one of claims 1 to 7.

16. A data processing system, characterized in that: The data processing system comprises a distributed file system and a server according to claim 15 .

17. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of a server, the server is enabled to execute the data processing method according to any one of claims 1 to 7.

18. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the data processing method according to any one of claims 1 to 7 is implemented.

Images