Data classification method and device

Abstract

The invention discloses a data classification method and device, and the method comprises the steps: receiving a query request, and storing the query request to a request queue; the query request comprises an object identifier and a set identifier; judging whether the request queue meets a preset aggregation condition or not; if a preset aggregation condition is met, performing aggregation processing on the plurality of query requests contained in the request queue according to a preset aggregation dimension to obtain an aggregated query request subjected to aggregation processing, and executing the aggregated query request to obtain an aggregated query result; the preset aggregation dimension comprises an object identification dimension or a set identification dimension; and splitting the aggregated query result to obtain a plurality of object query results, and respectively returning the plurality of object query results to the corresponding query requests. By aggregating the query requests according to the preset aggregation dimension, a plurality of query requests can be aggregated into one aggregated query request, so that the concurrent query requests are greatly reduced, and the query processing efficiency is improved.

Description

Technical Field

[0001] The present application relates to the technical field of request aggregation, and in particular to a data classification method and device. Background Art

[0002] When publishing a task and needing to query the target object from a collection, data classification is generally used to filter the target object of the task, such as treating multiple objects as a collection, selecting the target object from the collection, and then publishing the task to the target object.

[0003] When classifying existing tasks, it is directly determined whether an object is in a certain collection. An independent classification query interface is used for judgment. When the query is at the peak of requests, if multiple requests are initiated, the corresponding multiple independent query interfaces will all perform queries, which will make the QPS (Queries-per-second) very high. Due to the limited query business processing capability, it is unable to support a large number of requests, resulting in a large number of query failures, which seriously affects related functions such as task publishing, as well as processing efficiency. Summary of the invention

[0004] In view of the above problems, embodiments of the present application are proposed to provide a data classification method and device that overcome the above problems or at least partially solve the above problems.

[0005] According to a first aspect of an embodiment of the present application, a data classification method is provided, which includes:

[0006] receiving a query request and storing the query request in a request queue; the query request includes an object identifier and a collection identifier;

[0007] Determine whether the request queue meets the preset aggregation conditions;

[0008] If the preset aggregation condition is met, multiple query requests contained in the request queue are aggregated according to the preset aggregation dimension to obtain an aggregated query request, and the aggregated query request is executed to obtain an aggregated query result; the preset aggregation dimension includes: an object identification dimension or a collection identification dimension;

[0009] The aggregate query result is split and processed to obtain multiple object query results, and the multiple object query results are returned to the corresponding query requests respectively.

[0010] Optionally, determining whether the request queue meets a preset aggregation condition further includes:

[0011] Determine whether the difference between the time when the first query request is stored in the request queue and the current time reaches a preset duration, or determine whether the number of query requests with the same object identifier or the same collection identifier in the request queue reaches a preset number;

[0012] If the preset duration or the preset number is reached, it is determined that the request queue meets the preset aggregation condition.

[0013] Optionally, if the preset aggregation dimension is an object identification dimension;

[0014] Aggregating multiple query requests included in the request queue according to a preset aggregation dimension to obtain an aggregated query request processed by the aggregation, executing the aggregated query request, and obtaining an aggregated query result further includes:

[0015] Aggregate the query requests in the request queue according to the object identifier dimension to obtain a multi-parameter aggregate query request; the multi-parameter includes an object identifier and multiple collection identifiers;

[0016] Execute the aggregate query request to obtain the aggregate query result of whether each collection contains the object identifier.

[0017] Optionally, if the dimension is a collection identification dimension;

[0018] Aggregating multiple query requests included in the request queue according to a preset aggregation dimension to obtain an aggregated query request processed by the aggregation, executing the aggregated query request, and obtaining an aggregated query result further includes:

[0019] Aggregate the query requests in the request queue according to the collection identifier dimension to obtain a multi-parameter aggregate query request; the multi-parameter includes a collection identifier and multiple object identifiers;

[0020] Execute the aggregate query request to obtain the aggregate query result of whether each object identifier is in the collection.

[0021] Optionally, splitting the aggregate query result to obtain multiple object query results, and returning the multiple object query results to corresponding query requests respectively further includes:

[0022] Split the aggregate query results to obtain multiple object query results;

[0023] According to the object identifier and the collection identifier included in the object query result, the corresponding query request is determined, and the multiple object query results are returned to the corresponding query requests respectively.

[0024] Optionally, there are multiple request queues;

[0025] Receiving a query request and storing the query request in a request queue further comprises:

[0026] Receive a query request, and store the query request in a corresponding request queue according to the object identifier or collection identifier.

[0027] Optionally, the method further comprises:

[0028] Monitor the request queue based on the distributed thread to determine whether the length of the request queue is greater than the maximum length threshold;

[0029] If it is greater than the maximum length threshold, dynamic capacity expansion of distributed threads is performed;

[0030] Determine whether the length of the request queue is less than the minimum length threshold;

[0031] If it is less than the minimum length threshold, the distributed thread is dynamically scaled down.

[0032] Optionally, the method further comprises:

[0033] Each object is divided into different sets according to preset division conditions, each set includes multiple object identifiers; each set includes a set identifier; the preset division conditions are determined according to the object information.

[0034] According to a second aspect of an embodiment of the present application, there is provided a data classification device, comprising:

[0035] A storage module, adapted to receive a query request and store the query request in a request queue; the query request includes an object identifier and a collection identifier;

[0036] A judgment module, adapted to judge whether the request queue meets a preset aggregation condition;

[0037] The aggregation module is adapted to aggregate the multiple query requests contained in the request queue according to the preset aggregation dimension if the preset aggregation condition is met, obtain the aggregated query request processed by the aggregation, execute the aggregated query request, and obtain the aggregated query result; the preset aggregation dimension includes: object identification dimension or collection identification dimension;

[0038] The splitting module is suitable for splitting the aggregate query result to obtain multiple object query results, and returning the multiple object query results to the corresponding query requests respectively.

[0039] According to a third aspect of an embodiment of the present application, there is provided a computing device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus;

[0040] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute operations corresponding to the above-mentioned data classification method.

[0041] According to a fourth aspect of an embodiment of the present application, a computer storage medium is provided, wherein the storage medium stores at least one executable instruction, and the executable instruction enables a processor to perform operations corresponding to the above-mentioned data classification method.

[0042] According to a fifth aspect of an embodiment of the present application, a computer program product is provided, comprising at least one executable instruction, wherein the executable instruction enables a processor to perform operations corresponding to the above-mentioned data classification method.

[0043] According to the data classification method and device provided in the present application, by aggregating query requests according to preset aggregation dimensions, multiple query requests can be aggregated into one aggregated query request, which greatly reduces concurrent query requests and improves query processing efficiency.

[0044] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0046] Figure 1 A flow chart of a data classification method according to an embodiment of the present application is shown;

[0047] Figure 2 A flow chart showing a data classification method according to another embodiment of the present application is shown;

[0048] Figure 3 A schematic diagram of a data classification architecture is shown;

[0049] Figure 4 A schematic diagram of the structure of a data classification device according to an embodiment of the present application is shown;

[0050] Figure 5 A schematic diagram of the structure of a computing device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0051] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0052] First, the terms involved in one or more embodiments of the present application are explained.

[0053] Object ID: A unique id for each object.

[0054] Collection: A collection contains multiple object identifiers, and each collection also has an independent collection identifier.

[0055] Data classification: Determine whether an object identifier is in a specific set.

[0056] Aggregate request: Aggregate multiple requests into one request.

[0057] Figure 1 A flow chart of a data classification method according to an embodiment of the present application is shown. Figure 1 As shown, the method comprises the following steps:

[0058] Step S101, receiving a query request, and storing the query request in a request queue.

[0059] The query request includes an object identifier and a set identifier, which is used to query whether the object identifier is located in the set corresponding to the set identifier, so as to select an object from the set and issue an object task.

[0060] When a query request is received, the prior art directly performs an independent query based on the query request. When a large number of query requests are received, the query efficiency is slow, the query fails, and the like due to the large number of requests. This embodiment takes the above-mentioned problem into consideration. After receiving the query request, the query request is first stored in the request queue, rather than immediately performing a query based on the query request. The query request is stored in the request queue, and then aggregation processing can be performed based on multiple query requests in the request queue, so that multiple query requests can be aggregated into one request, that is, multiple requests are aggregated into one request, which can greatly reduce the number of requests and improve query efficiency.

[0061] Step S102, determining whether the request queue meets a preset aggregation condition.

[0062] When the request queue contains multiple query requests, it is necessary to further determine the aggregation timing in order to aggregate the multiple query requests in the request queue.

[0063] The aggregation timing is determined by determining whether the request queue meets a preset aggregation condition. The preset aggregation condition may include multiple aspects, such as time or quantity.

[0064] Specifically, for example, it is determined whether the difference between the time when the first query request stored in the request queue is stored and the current time reaches a preset time length, such as 100ms. The preset time length can avoid the problem of query requests not being executed for a long time. The preset time length is set according to the implementation situation. When it is determined that the preset time length is reached, such as when the difference reaches 100ms, step S103 is executed to aggregate the query requests in the request queue whose difference is within the preset time length.

[0065] Alternatively, determine whether the number of query requests with the same object identifier or the same collection identifier in the request queue reaches a preset number, that is, determine whether the number of query requests based on the same object identifier reaches a preset number, or determine whether the number of query requests based on the same collection identifier reaches a preset number, the preset number is 10, when the number of query requests with the same object identifier in the request queue reaches a preset number of 10, execute step S103, and aggregate the 10 query requests with the same object identifier, or when the number of query requests with the same collection identifier in the request queue reaches a preset number of 10, execute step S103, and aggregate the 10 query requests with the same collection identifier.

[0066] The preset duration is used to prevent query requests from being unprocessed for a long time, and the preset number can greatly reduce the number of requests. When judging whether the preset aggregation condition is met, when the preset duration is reached or the preset number is reached, if either of the two is met, it is determined that the request queue meets the preset aggregation condition, and step S103 can be executed to aggregate the query requests in the request queue.

[0067] Step S103, aggregate the multiple query requests included in the request queue according to the preset aggregation dimension to obtain an aggregated query request, execute the aggregated query request, and obtain an aggregated query result.

[0068] During the aggregation processing, multiple query requests in the request queue that meet the preset aggregation conditions are aggregated to obtain an aggregated query request after the aggregation processing. During the aggregation processing, the aggregation processing can be performed according to the preset aggregation dimension, and the preset aggregation dimension includes, for example, the object identification dimension or the collection identification dimension. For example, according to the object identification dimension, multiple query requests including the same object identification are aggregated, such as multiple query requests for querying whether object identification 1 is in collection 1, querying whether object identification 1 is in collection 2, querying whether object identification 1 is in collection 3, and querying whether object identification 1 is in collection 4. Aggregation processing is performed according to the same object identification, and the obtained aggregate query request is to query whether object identification 1 is in collection 1, collection 2, collection 3, and collection 4. The four requests are simplified into one request. During execution, only the aggregate query request is executed to obtain an aggregate query result containing multiple object query results, thereby greatly reducing the request volume, reducing the query failure rate, and ensuring that the query results can be obtained. The above example uses the object ID dimension for aggregation processing. Aggregation processing can also be performed based on the set ID dimension. For example, if multiple query requests are respectively to query whether object ID 1 is in set 1, whether object ID 2 is in set 1, and whether object ID 4 is in set 1, aggregation processing is performed based on the same set ID. The resulting aggregate query request is to query whether object IDs 1, 2, and 4 are in set 1, thus simplifying three requests into one request.

[0069] Whether according to the object identification dimension or the collection identification dimension, multiple query requests can be aggregated to obtain aggregated query requests after aggregation. Aggregate query requests greatly reduce the request volume. Executing an aggregate query request once can obtain multiple object query results, and also reduces the query failure rate.

[0070] Step S104, splitting the aggregate query result to obtain multiple object query results, and returning the multiple object query results to the corresponding query requests respectively.

[0071] The aggregate query result itself contains multiple object query results corresponding to multiple query requests, but the aggregate query result cannot be directly returned to multiple query requests. The aggregate query result needs to be split into multiple object query results, and then each object query result is returned to the query request respectively.

[0072] Specifically, if the aggregate query results are object ID 1 in set 1, object ID 1 not in set 2, object ID 1 not in set 3, and object ID 1 in set 4, the aggregate query result is split into 4 object query results, namely object ID 1 in set 1; object ID 1 not in set 2; object ID 1 not in set 3; object ID 1 in set 4. By comparing the object ID and set ID included in the object query result with the object ID and set ID included in the query request, the query request corresponding to each object query result can be determined, and the object query result is returned to the corresponding query request.

[0073] In an optional embodiment, taking the example of publishing an object task to an object, the object is in different sets, and the query request is to query whether a certain object is in a certain set. The query request includes an object identifier and a set identifier. When the query request is received, the query request is first stored in a request queue, and then the request queue is judged to determine whether the preset aggregation condition is met. For example, if the difference between the time of the first query request stored in the request queue and the current time reaches a preset duration, it is determined that the preset aggregation condition is met, or the number of query requests with the same object identifier in the request queue reaches a preset number, or the number of query requests with the same set identifier in the request queue reaches a preset number, it is determined that the preset aggregation condition is met. The query requests in the request queue are aggregated according to the object identifier or the set identifier, such as aggregating multiple query requests with the same object identifier to obtain an aggregated query request for querying whether an object is in multiple sets, or aggregating multiple query requests with the same set identifier to obtain an aggregated query request for querying whether multiple objects are in one set. The aggregated query request is executed to obtain an aggregated query result. The aggregate query results are split to obtain multiple object query results. Each object query result is a query result of whether a certain object is in a certain set. The query result is returned to the corresponding query request to determine the query result, and the object task is issued to the object whose query result is yes. By aggregating the query requests to obtain the aggregate query request, only executing the aggregate query request can greatly reduce the request volume and reduce the concurrency pressure of the object task when performing the query. The above examples are applicable to the data classification of any object.

[0074] According to the data classification method provided in the present application, by aggregating query requests according to preset aggregation dimensions, multiple query requests can be aggregated into one aggregated query request, which greatly reduces concurrent query requests and improves query processing efficiency.

[0075] Figure 2 A flow chart of a data classification method according to an embodiment of the present application is shown. Figure 2 As shown, the method comprises the following steps:

[0076] Step S201, dividing each object into different sets according to preset division conditions.

[0077] When performing data classification, objects are queried from different sets so that object tasks can be issued to the queried objects. Before performing data classification, objects need to be divided into different sets to facilitate querying objects according to object tasks. The division can be performed according to preset division conditions, which can be determined based on object information. For example, taking object A as an example, the preset division conditions can be determined based on the number of views of the object, the length of time the object is played, and the level information of the object. Different objects are divided into different sets. Each set can contain multiple object identifiers, and each set has its own set identifier. An object can be in different sets, such as object 1 is in set 5 based on the number of views, in set 10 based on the length of time the object is played, and so on. There is no limitation here.

[0078] Step S202: receiving a query request, and storing the query request in a corresponding request queue according to the object identifier or the collection identifier.

[0079] In order to facilitate the acquisition of query requests, this embodiment constructs multiple request queues, each request queue stores query requests according to object identifiers or collection identifiers. If the range of object identifiers or collection identifiers is large, the object identifiers or collection identifiers can be classified, and each request object stores query requests for the same type of object identifiers or collection identifiers. When classifying, taking the object identifier as an example, if the object identifier is a digital identifier, the modulus result of the object identifier can be obtained through modulus processing, and the object identifiers with the same modulus result are classified into one category and stored in the same request queue; if the object identifier is an alphabetic identifier, it can be classified according to the last letter of the alphabetic identifier, 4 letters are one category, and a category of object identifiers is determined according to the last letter, etc. If the object identifier contains both letters and numbers, the classification can be determined by combining the two. The collection identifier can be classified in the same way. The above is an example, which is set according to the implementation situation and is not limited here.

[0080] After classification, query requests of the same type of object identifiers or the same type of collection identifiers are stored in the corresponding request queue. Taking object identifiers as an example, after modulo processing, query requests containing object 1 are stored in request queue 1, query requests containing object 2 are stored in request queue 2, query requests containing object 3 are stored in request queue 3... and so on, the query requests are stored in the request queues.

[0081] Furthermore, when the request queue stores a query request, the object identifier and the collection identifier may be stored for subsequent aggregation processing, and the object identifier and the collection identifier may be used as parameters of the aggregate query request to execute the aggregate query request.

[0082] Step S203, determining whether the request queue meets a preset aggregation condition.

[0083] When there are multiple request queues, each request queue needs to be monitored and judged. When any request queue meets the preset aggregation condition, step S204 is executed for the request queue. Specifically, it is necessary to determine whether to aggregate according to the object identification dimension or the collection identification dimension based on the query request in the request queue.

[0084] Specifically, the request queue stores query requests with the same object identifier or the same collection identifier. For any request queue, the difference between the time of the first query request stored in the request queue and the current time is calculated to determine whether the difference reaches a preset time length, such as 100ms. If the preset time length is reached, each query request in the request queue whose difference is within the preset time length (i.e., stored) can be aggregated. When performing the aggregate processing, it is necessary to further analyze each query request whose difference is within the preset time length, and further divide the query requests with the same object identifier or the same collection identifier contained in the request queue, execute step S204 for aggregate processing for query requests with the same object identifier, and execute step S204 for aggregate processing for query requests with the same collection identifier.

[0085] Alternatively, it is determined whether the number of query requests with the same object identifier or the same collection identifier in the request queue has reached a preset number. Here, it should be noted that the same object identifier or the same collection identifier is determined, not the same type of object identifier or the same type of collection identifier. If the number of query requests with the same object identifier in the request queue has reached a preset number, the query requests with the same object identifier are aggregated in step S204; or, if the number of query requests with the same collection identifier in the request queue has reached a preset number, the query requests with the same collection identifier are aggregated in step S204.

[0086] Further, when judging, if any of the above judgments is met, aggregation processing can be performed. If the preset number is reached first, the query requests in the request queue are aggregated according to the same object identifier or the same collection identifier; if the preset duration is reached first, the query requests stored in the request queue are aggregated, even if the query requests with the same object identifier or the same collection identifier in the request queue are less than the preset number at this time, aggregation processing is performed. If there is only one query request with a certain object identifier or collection identifier, the query request is executed and the query request is not aggregated. If the query requests stored in the request queue include 4 query requests with object identifier 1, 3 query requests with object identifier 11, and 1 query request with object identifier 21, then the query requests with the same object identifier are aggregated, and the 4 query requests including object identifier 1 are aggregated, and the 3 query requests including object identifier 11 are aggregated, and the query request with object identifier 21 does not need to be aggregated. The above is an example, which is set according to the implementation situation and is not limited here.

[0087] Step S204, aggregate the multiple query requests included in the request queue according to the preset aggregation dimension to obtain an aggregated query request, execute the aggregated query request, and obtain an aggregated query result.

[0088] If it is determined that the request queue meets the preset aggregation condition, then the multiple query requests included in the request queue are aggregated. The aggregation process of this embodiment can be performed using two different aggregation dimensions: object identification dimension and collection identification dimension.

[0089] Specifically, according to the query requests contained in the request queue, when they are stored in the object identifier dimension, each query request in the request queue is aggregated according to the object identifier dimension to obtain a multi-parameter aggregate query request. The multiple parameters include an object identifier and multiple set identifiers. For example, each query request in the request queue includes object identifier 1 and set identifier 1, object identifier 1 and set identifier 2, object identifier 1 and set identifier 4, object identifier 1 and set identifier 6, respectively. After the aggregation process, the multiple parameters in the obtained multi-parameter aggregate query request are object identifier 1, set identifier 1, set identifier 2, set identifier 4, and set identifier 6. The aggregate query request is executed to obtain an aggregate query result of whether each set contains the object identifier. The set corresponding to set identifier 1 contains object identifier 1, the set corresponding to set identifier 2 does not contain object identifier 1, the set corresponding to set identifier 4 contains object identifier 1, and the set corresponding to set identifier 6 contains object identifier 1. The above four results are the aggregate query results.

[0090] If each query request in the request queue is stored in the dimension of set identifier, aggregation processing is performed according to the dimension of set identifier to obtain a multi-parameter aggregate query request. The multiple parameters include a set identifier and multiple object identifiers. If each query request in the request queue includes object identifier 1 and set identifier 1, object identifier 3 and set identifier 1, object identifier 5 and set identifier 1, object identifier 6 and set identifier 1, then after aggregation processing, the multiple parameters in the obtained multi-parameter aggregate query request are object identifier 1, object identifier 3, object identifier 5, object identifier 6 and set identifier 1. Execute the aggregate query request to obtain the aggregate query result of whether each object identifier is located in the set. Object identifier 1 is located in the set corresponding to set identifier 1, object identifier 3 is located in the set corresponding to set identifier 1, object identifier 5 is located in the set corresponding to set identifier 1, and object identifier 6 is not located in the set corresponding to set identifier 1. The above four results are the aggregate query results.

[0091] Step S205, splitting the aggregate query result to obtain multiple object query results, determining the corresponding query request according to the object identifier and collection identifier contained in the object query result, and returning the multiple object query results to the corresponding query requests respectively.

[0092] After obtaining the aggregate query results, the aggregate query results need to be split into multiple object query results. Each object query result includes an object identifier and a collection identifier, as well as a result of whether the object identifier is in the collection corresponding to the collection identifier. Based on the object identifier and collection identifier contained in the object query result, the corresponding query request can be determined, and each object query result is returned to the corresponding query request to complete the data classification.

[0093] Furthermore, when the aggregate query request is executed to obtain the aggregate query result, it is also necessary to consider various abnormal execution situations of the aggregate query request, such as request timeout, and the result of the current request timeout needs to be returned to each query request; if there is an execution error such as query failure, it is necessary to determine which query request failed and return the corresponding query result of the failed query.

[0094] Step S206: monitor the request queue based on the distributed thread to perform capacity expansion or capacity reduction.

[0095] Before aggregation processing, query requests are first stored in the request queue. As query requests continue to be added to the request queue, the length of the request queue will also continue to increase. If the length of the request queue is too long, it means that the amount of data to be processed is large and needs to be processed to ensure that query requests can be quickly obtained from the request queue for aggregation processing later, avoiding problems such as taking too long to retrieve query requests and affecting processing efficiency.

[0096] Specifically, this embodiment uses distributed threads to monitor the load of the request queue in real time, and determines whether the length of the request queue is greater than the maximum length threshold. If it is greater than the maximum length threshold, dynamic expansion processing can be performed in real time to increase the number of distributed threads to improve processing capabilities. Further, it can also be determined whether the length of the request queue is less than the minimum length threshold. If it is less than the minimum length threshold, dynamic reduction processing is performed to recycle the idle resources of the distributed threads in time. The maximum length threshold and the minimum length threshold can be set according to the implementation situation and are not limited here.

[0097] By monitoring the request queue and dynamically expanding or shrinking the capacity in real time, the processing pressure can be relieved in time.

[0098] There is no limitation on the execution order between step S206 and steps S203-S205. After the query request is stored in the request queue, the distributed thread can perform real-time monitoring.

[0099] In this embodiment, the following data structure can be used to implement data classification, as shown below:

[0100] type TitanProxy struct{

[0101] workMod --aggregation dimension

[0102] reqChanByUid -- request queue based on object identity aggregation

[0103] reqChanByCondition -- request queue based on collection identifier aggregation

[0104] closeChan -- Close the request queue for the object ID

[0105] workerCount -- Number of distributed threads

[0106] minWorkerCount - minimum number of expansions

[0107] maxWorkerCount - Maximum number of expansions

[0108] adjustInterval - load monitoring interval

[0109] stopConsumerChannel – shuts down the thread

[0110] }

[0111] The above is an example for explanation, and the specific data structure used can be set according to the implementation situation and is not limited here.

[0112] like Figure 3 As shown, when the object task unit (object task system) issues a task, multiple concurrent requests will be generated. If the multiple concurrent requests are executed independently, the pressure on the classification business processing unit will be too heavy. The data classification initialization thread pool of this embodiment, wherein each distributed thread is independent of each other. Concurrent requests can be stored in a message queue of multiple requests. The distributed thread aggregates the stored query requests by judging the aggregation timing, and the aggregated query requests after the aggregation processing are queried by the classification business processing to obtain the aggregated query results, and the corresponding results are returned after the split processing. When the request times out or errors such as query failure occur, the distributed thread performs request timeout processing or error processing. At the same time, the distributed thread will also monitor the real-time load of the message queue, and dynamically expand or shrink the processing capacity in a timely manner.

[0113] According to the data classification method provided by the present application, query requests are stored in the corresponding request queue according to the same object identifier or the same collection identifier. When the storage duration or the number of aggregations required meets the preset aggregation conditions, the query requests are aggregated according to the object identifier dimension or the collection identifier dimension, and multiple query requests are aggregated into one aggregate query request, thereby reducing the number of concurrent query requests. By executing an aggregate query request once, an aggregate query result containing multiple object query results can be obtained. The aggregate query results are split and processed, and the obtained object query results are returned to the corresponding query request, thereby greatly improving the query processing efficiency. At the same time, the distributed thread can monitor the length of the request queue in real time, perform dynamic expansion or reduction in time, ensure processing capacity, or recycle idle resources in time to avoid data waste.

[0114] Figure 4 FIG. 1 is a schematic diagram showing the structure of a data classification device provided by an embodiment of the present application. Figure 4 As shown, the device comprises:

[0115] The storage module 410 is adapted to receive a query request and store the query request in a request queue; the query request includes an object identifier and a collection identifier;

[0116] A determination module 420, adapted to determine whether the request queue meets a preset aggregation condition;

[0117] Aggregation module 430 is adapted to aggregate multiple query requests included in the request queue according to preset aggregation dimensions to obtain aggregated query requests if preset aggregation conditions are met, execute the aggregated query requests, and obtain aggregated query results; the preset aggregation dimensions include: object identification dimensions or collection identification dimensions;

[0118] The splitting module 440 is adapted to split the aggregate query result to obtain multiple object query results, and return the multiple object query results to the corresponding query requests respectively.

[0119] Optionally, the determination module 420 is further adapted to:

[0120] Determine whether the difference between the time when the first query request is stored in the request queue and the current time reaches a preset duration, or determine whether the number of query requests with the same object identifier or the same collection identifier in the request queue reaches a preset number;

[0121] If the preset duration or the preset number is reached, it is determined that the request queue meets the preset aggregation condition.

[0122] Optionally, if the preset aggregation dimension is an object identification dimension;

[0123] The aggregation module 430 is further adapted to:

[0124] Aggregate the query requests in the request queue according to the object identifier dimension to obtain a multi-parameter aggregate query request; the multi-parameter includes an object identifier and multiple collection identifiers;

[0125] Execute the aggregate query request to obtain the aggregate query result of whether each collection contains the object identifier.

[0126] Optionally, if the dimension is a collection identification dimension;

[0127] The aggregation module 430 is further adapted to:

[0128] Aggregate the query requests in the request queue according to the collection identifier dimension to obtain a multi-parameter aggregate query request; the multi-parameter includes a collection identifier and multiple object identifiers;

[0129] Execute the aggregate query request to obtain the aggregate query result of whether each object identifier is in the collection.

[0130] Optionally, the splitting module 440 is further adapted to:

[0131] Split the aggregate query results to obtain multiple object query results;

[0132] According to the object identifier and the collection identifier included in the object query result, the corresponding query request is determined, and the multiple object query results are returned to the corresponding query requests respectively.

[0133] Optionally, there are multiple request queues;

[0134] The storage module 410 is further adapted to:

[0135] Receive a query request, and store the query request in a corresponding request queue according to the object identifier or collection identifier.

[0136] Optionally, the apparatus further comprises: a monitoring module 450 adapted to monitor the request queue based on the distributed thread, and determine whether the length of the request queue is greater than a maximum length threshold;

[0137] If it is greater than the maximum length threshold, dynamic capacity expansion of distributed threads is performed;

[0138] Determine whether the length of the request queue is less than the minimum length threshold;

[0139] If it is less than the minimum length threshold, the distributed thread is dynamically scaled down.

[0140] Optionally, the device further includes: a division module 460, adapted to divide each object into different sets according to preset division conditions, each set including multiple object identifiers; each set including a set identifier; the preset division conditions are determined according to the object information.

[0141] The description of each module above refers to the corresponding description in the method embodiment and will not be repeated here.

[0142] According to the data classification device provided in the present application, by aggregating query requests according to preset aggregation dimensions, multiple query requests can be aggregated into one aggregated query request, which greatly reduces concurrent query requests and improves query processing efficiency.

[0143] The present application also provides a non-volatile computer storage medium, which stores at least one executable instruction, and the executable instruction can execute the operation corresponding to the data classification method in any of the above method embodiments.

[0144] The present application also provides a computer program product, which includes at least one executable instruction or computer program, and the executable instruction or computer program can enable a processor to execute operations corresponding to the data classification method in any of the above method embodiments.

[0145] Figure 5 A schematic diagram of the structure of a computing device according to an embodiment of the present application is shown. The specific embodiment of the present application does not limit the specific implementation of the computing device.

[0146] like Figure 5 As shown, the computing device may include: a processor (processor) 502 , a communications interface (Communications Interface) 504 , a memory (memory) 506 , and a communication bus 508 .

[0147] in:

[0148] The processor 502 , the communication interface 504 , and the memory 506 communicate with each other via a communication bus 508 .

[0149] The communication interface 504 is used to communicate with other devices such as clients or other servers.

[0150] The processor 502 is used to execute the program 510, and specifically can execute the relevant steps in the above data classification method embodiment.

[0151] Specifically, the program 510 may include program codes, which include computer operation instructions.

[0152] The processor 502 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the present application. The one or more processors included in the computing device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0153] The memory 506 is used to store the program 510. The memory 506 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0154] The program 510 can be specifically used to enable the processor 502 to execute the data classification method in any of the above method embodiments. The specific implementation of each step in the program 510 can refer to the corresponding descriptions in the corresponding steps and units in the above data classification embodiments, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the above-mentioned method embodiments, which will not be repeated here.

[0155] The algorithm or display provided herein is not inherently related to any particular computer, virtual system or other device. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious to construct the structure required for this type of system. In addition, the application is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to realize the content of the application described herein, and the description of the above specific language is to disclose the preferred embodiment of the application.

[0156] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0157] Similarly, it should be understood that in order to streamline the present application and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following intention: the claimed application requires more features than the features clearly stated in each claim. More specifically, as reflected in the claims below, the inventive aspects are less than all the features of the single embodiment disclosed above. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present application.

[0158] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0159] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0160] The various component embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all functions of some or all components of the present application. The present application can also be implemented as a device or apparatus program (e.g., computer program and computer program product) for executing a part or all of the methods described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0161] It should be noted that the above embodiments illustrate the present application rather than limit the present application, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be constructed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The present application may be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be understood as limitations on the order of execution.

Claims

1. A data classification method, comprising: receiving a query request, and storing the query request in a request queue; The query request includes an object identifier and a collection identifier; Determining whether the request queue meets a preset aggregation condition; If the preset aggregation condition is met, the multiple query requests included in the request queue are aggregated according to the preset aggregation dimension to obtain an aggregated query request, and the aggregated query request is executed to obtain an aggregated query result; The preset aggregation dimension includes: an object identification dimension or a collection identification dimension; The aggregate query result is split to obtain multiple object query results, and the multiple object query results are returned to the corresponding query requests respectively.

2. The method according to claim 1, wherein: The determining whether the request queue meets the preset aggregation condition further includes: Determine whether the difference between the time when the first query request is stored in the request queue and the current time reaches a preset duration, or determine whether the number of query requests with the same object identifier or the same collection identifier in the request queue reaches a preset number; If the preset duration or the preset number is reached, it is determined that the request queue meets the preset aggregation condition.

3. The method according to claim 1 or 2, wherein: If the preset aggregation dimension is an object identification dimension; The step of aggregating the multiple query requests included in the request queue according to a preset aggregation dimension to obtain an aggregated query request processed by the aggregation, and executing the aggregated query request to obtain an aggregated query result further includes: Aggregate the query requests in the request queue according to the object identifier dimension to obtain a multi-parameter aggregate query request; the multi-parameters include an object identifier and multiple collection identifiers; The aggregate query request is executed to obtain an aggregate query result of whether each collection contains the object identifier.

4. The method according to claim 1 or 2, wherein: If the dimension is a set identification dimension; The step of aggregating the multiple query requests included in the request queue according to a preset aggregation dimension to obtain an aggregated query request processed by the aggregation, and executing the aggregated query request to obtain an aggregated query result further includes: Aggregate the query requests in the request queue according to the set identifier dimension to obtain a multi-parameter aggregate query request; the multi-parameters include a set identifier and multiple object identifiers; The aggregate query request is executed to obtain an aggregate query result of whether each object identifier is located in the set.

5. The method according to any one of claims 1 to 4, wherein: The splitting of the aggregate query result to obtain multiple object query results, and returning the multiple object query results to the corresponding query requests respectively further comprises: Splitting the aggregate query result to obtain multiple object query results; According to the object identifier and the collection identifier included in the object query result, the corresponding query request is determined, and the multiple object query results are returned to the corresponding query request respectively.

6. The method according to any one of claims 1 to 5, wherein: There are multiple request queues; The receiving a query request and storing the query request in a request queue further comprises: A query request is received, and the query request is stored in a corresponding request queue according to an object identifier or a collection identifier.

7. The method according to any one of claims 1 to 6, wherein: The method further comprises: Monitor the request queue based on the distributed thread, and determine whether the length of the request queue is greater than a maximum length threshold; If it is greater than the maximum length threshold, dynamic capacity expansion of distributed threads is performed; Determine whether the length of the request queue is less than a minimum length threshold; If it is less than the minimum length threshold, the distributed thread is dynamically scaled down.

8. The method according to any one of claims 1 to 7, wherein: The method further comprises: Each object is divided into different sets according to a preset division condition, each set includes a plurality of object identifiers; each set includes a set identifier; and the preset division condition is determined according to the object information.

9. A data classification device, comprising: A storage module, adapted to receive a query request and store the query request in a request queue; The query request includes an object identifier and a collection identifier; A determination module, adapted to determine whether the request queue satisfies a preset aggregation condition; an aggregation module adapted to aggregate the multiple query requests contained in the request queue according to a preset aggregation dimension if a preset aggregation condition is met, obtain an aggregated query request processed by the aggregation, execute the aggregated query request, and obtain an aggregated query result; The preset aggregation dimension includes: an object identification dimension or a collection identification dimension; The splitting module is adapted to split the aggregate query result to obtain multiple object query results, and return the multiple object query results to the corresponding query requests respectively.

10. A computing device comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the data classification method according to any one of claims 1-8.

11. A computer storage medium, wherein at least one executable instruction is stored in the storage medium, and the executable instruction enables a processor to execute an operation corresponding to the data classification method according to any one of claims 1 to 8.

12. A computer program product, comprising at least one executable instruction, wherein the executable instruction enables a processor to execute operations corresponding to the data classification method according to any one of claims 1 to 8.

Images