Method, device, server and storage medium for obtaining data to be processed

Abstract

The present disclosure is about a method, device, server and storage medium for obtaining data to be processed, and relates to the field of data processing technology, wherein the method comprises: receiving an acquisition request, the acquisition request comprises the category of a list to be processed; counting the current total amount of to-be-marked data in the list to be processed and the current amount of to-be-marked data belonging to the category in each sub-table; wherein the sub-table is a database table for storing the data to be processed in each list to be processed; according to the current total amount of to-be-marked data in the list to be processed and the current amount of to-be-marked data belonging to the category in each sub-table, determining the number of to-be-processed data belonging to the category in each sub-table to be read; according to the number of to-be-processed data belonging to the category in each sub-table to be read, reading the corresponding number of to-be-processed data from each sub-table. Thus, the present disclosure can read the corresponding number of to-be-processed data from each database sub-table, and solve the problem of uneven acquisition of to-be-processed data from the database sub-table in the related art.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a method, device, server and storage medium for acquiring data to be processed. Background Art

[0002] Usually, in order to ensure the security of video content, each video uploaded by a user needs to be processed, such as audited, before the user publishes the video. However, with the continuous development of short video business, the amount of videos uploaded by users is also increasing.

[0003] In the related art, video data is stored in a single data table. However, as the amount of video increases, the query efficiency will decrease. In order to support the needs of large data volume and high concurrency, video data is stored in a database sub-table. Since the video data to be processed in the database is often assigned to multiple processors for processing, each processor corresponds to a different list to be processed, and each list to be processed has a certain amount of video data to be processed. However, when there are many lists to be processed, there may be a situation where the data to be processed is always obtained from the same database sub-table, resulting in uneven data acquisition from the database sub-table. Summary of the invention

[0004] The present disclosure provides a method, device, server and storage medium for obtaining data to be processed, so as to at least solve the problem of uneven acquisition of data to be processed from database sub-tables in the related art. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, a method for acquiring data to be processed is provided, comprising:

[0006] receiving an acquisition request, wherein the acquisition request includes a category of a to-be-processed list;

[0007] Counting the current total amount of data to be marked in the to-be-processed list and the current amount of data to be marked in each sub-table belonging to the category; wherein the sub-table is a database table storing the data to be processed in each to-be-processed list;

[0008] Determine the read quantity of the to-be-processed data belonging to the category in each sub-table according to the current total to-be-marked quantity of the to-be-processed list and the current to-be-marked quantity of the to-be-processed data belonging to the category in each sub-table;

[0009] According to the read quantity of the to-be-processed data belonging to the category in each sub-table, a corresponding read quantity of the to-be-processed data is read from each sub-table.

[0010] In some embodiments of the present disclosure, determining the read quantity of the to-be-processed data belonging to the category in each sub-table according to the current total to-be-marked quantity of the to-be-processed list and the current to-be-marked quantity of the to-be-processed data belonging to the category in each sub-table includes:

[0011] According to the current total amount of data to be marked in the to-be-processed list and the current amount of data to be marked that belongs to the category in each sub-table, determining the proportion of the data to be processed that belongs to the category in each sub-table to the current total amount of data to be marked;

[0012] The read quantity of the to-be-processed data belonging to the category in each sub-table is determined according to the proportion of the to-be-processed data belonging to the category in each sub-table to the current total to-be-marked quantity.

[0013] In some embodiments of the present disclosure, the method further includes:

[0014] According to the categories of the to-be-processed lists and the identifiers of the sub-tables, periodically reading the to-be-processed data of each category from the sub-tables;

[0015] The to-be-processed data of each category read from the sub-tables is loaded into the cache unit of the to-be-processed list of the corresponding category.

[0016] In a possible implementation, reading a corresponding number of to-be-processed data from each sub-table according to the number of to-be-processed data read from each sub-table belonging to the category includes:

[0017] According to the read quantity of the to-be-processed data belonging to the category in each sub-table, the to-be-processed data of the corresponding read quantity in each sub-table is read from the target cache unit of the to-be-processed list corresponding to the category.

[0018] In a possible implementation, reading the corresponding number of to-be-processed data read from each sub-table from a target cache unit in a to-be-processed list corresponding to the category according to the number of to-be-processed data read from each sub-table includes:

[0019] Determine processed data and locked data in the target cache unit;

[0020] Filtering the processed data and locked data in the target cache unit;

[0021] According to the read quantity of the to-be-processed data belonging to the category in each sub-table, the to-be-processed data of the corresponding read quantity in each sub-table is read from the target cache unit from which the processed data and the locked data are filtered out.

[0022] According to a second aspect of an embodiment of the present disclosure, there is provided a device for acquiring data to be processed, including:

[0023] A receiving module, used for receiving an acquisition request, wherein the acquisition request includes a category of a list to be processed;

[0024] A statistical module, used to count the current total amount of data to be marked in the to-be-processed list and the current amount of data to be marked in each sub-table belonging to the category; wherein the sub-table is a database table storing the data to be processed in each to-be-processed list;

[0025] A determination module, configured to determine the read quantity of the to-be-processed data belonging to the category in each sub-table according to the current total to-be-marked quantity of the to-be-processed list and the current to-be-marked quantity of the to-be-processed data belonging to the category in each sub-table;

[0026] The reading module is used to read a corresponding number of to-be-processed data from each sub-table according to the number of to-be-processed data read in each sub-table belonging to the category.

[0027] In some embodiments of the present disclosure, the determining module is specifically used to:

[0028] According to the current total amount of data to be marked in the to-be-processed list and the current amount of data to be marked that belongs to the category in each sub-table, determining the proportion of the data to be processed that belongs to the category in each sub-table to the current total amount of data to be marked;

[0029] The read quantity of the to-be-processed data belonging to the category in each sub-table is determined according to the proportion of the to-be-processed data belonging to the category in each sub-table to the current total to-be-marked quantity.

[0030] In some embodiments of the present disclosure, the device further includes:

[0031] The cache module is used to read the to-be-processed data of each category from the sub-tables at regular intervals according to the categories of the to-be-processed lists and the identifiers of the sub-tables, and load the to-be-processed data of each category read from the sub-tables into the cache unit of the to-be-processed list of the corresponding category.

[0032] In a possible implementation, the reading module is specifically used to:

[0033] According to the read quantity of the to-be-processed data belonging to the category in each sub-table, the to-be-processed data of the corresponding read quantity in each sub-table is read from the target cache unit of the to-be-processed list corresponding to the category.

[0034] In a possible implementation, the reading module is specifically used to:

[0035] Determine processed data and locked data in the target cache unit;

[0036] Filtering the processed data and locked data in the target cache unit;

[0037] According to the read quantity of the to-be-processed data belonging to the category in each sub-table, the to-be-processed data of the corresponding read quantity in each sub-table is read from the target cache unit from which the processed data and the locked data are filtered out.

[0038] According to a third aspect of an embodiment of the present disclosure, a server is provided, including:

[0039] processor;

[0040] a memory for storing instructions executable by the processor;

[0041] Among them, the processor is configured as the instruction to implement the method for obtaining the data to be processed described in the embodiment of the first aspect.

[0042] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided. When instructions in the storage medium are executed by a processor of a server, the server can execute the method for obtaining data to be processed described in the embodiment of the first aspect.

[0043] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program implements the steps of the method described in the embodiment of the first aspect when executed by a processor.

[0044] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0045] Receive an acquisition request, the acquisition request includes the category of a pending list; count the current total amount of pending data to be marked in the pending list and the current amount of pending data to be marked in each sub-table belonging to the category; wherein the sub-table is a database table storing the pending data in each pending list; determine the read quantity of pending data belonging to the category in each sub-table according to the current total amount of pending data to be marked in the pending list and the current amount of pending data to be marked in each sub-table; read the respective read quantity of pending data from each sub-table according to the read quantity of pending data belonging to the category in each sub-table. Thus, the present disclosure can read respective quantities of pending data from each database sub-table, solve the problem of uneven acquisition of pending data from database sub-tables in the related art, thereby achieving the function of uniform distribution of mixed data streams, ensuring that the same category of pending data in each database sub-table can be read, and avoiding excessive differences in the number of data of the same category read in different database sub-tables.

[0046] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0048] Figure 1 is a flow chart of a method for obtaining data to be processed according to an exemplary embodiment;

[0049] Figure 2 is a flow chart of another method for obtaining data to be processed according to an exemplary embodiment;

[0050] Figure 3 is a flowchart of another method for obtaining data to be processed according to an exemplary embodiment;

[0051] Figure 4 is a flow chart of another method for obtaining data to be processed according to an exemplary embodiment;

[0052] Figure 5 is a block diagram of a device for acquiring data to be processed according to an exemplary embodiment;

[0053] Figure 6 is a block diagram of a device for acquiring data to be processed according to an exemplary embodiment;

[0054] Figure 7 It is a block diagram of a server according to an exemplary embodiment. DETAILED DESCRIPTION

[0055] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0056] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0057] It should be noted that, in order to facilitate the understanding of the present disclosure, the terms of the present disclosure are first explained accordingly.

[0058] In the embodiments of the present disclosure, the term "single data table" refers to a single data table. For example, when video data is stored in a database, the video data is stored in the same data table in the database. This situation can be referred to as storing video data in a single data table manner.

[0059] In the embodiments of the present disclosure, the term "storing video data in a database in a sub-table manner" can be understood as using multiple data tables in a database to store video data. For example, a large amount of video data can be stored in different data tables in a database to achieve the purpose of storing video data in a sub-table manner.

[0060] Usually, in order to ensure the security of video content, each video uploaded by a user needs to be processed before the user publishes the video. However, with the continuous development of short video business, the amount of videos uploaded by users is also increasing.

[0061] In the related art, video data is stored in a single data table. However, as the amount of video increases, the query efficiency will decrease. In order to support the needs of large data volume and high concurrency, video data is stored in a database sub-table. Since the video data to be processed in the database is often assigned to multiple processors for processing, each processor corresponds to a different list to be processed, and each list to be processed has a certain amount of video data to be processed. However, when there are many lists to be processed, there may be a situation where the data to be processed is always obtained from the same database sub-table, resulting in uneven data acquisition from the database sub-table.

[0062] In view of the above problems, the present disclosure proposes a method for obtaining data to be processed, which solves the problem of uneven acquisition of data to be processed from database sub-tables in related technologies by reading respective quantities of data to be processed from each database sub-table.

[0063] Figure 1 is a flow chart of a method for obtaining data to be processed according to an exemplary embodiment. It should be noted that the method for obtaining data to be processed in the embodiment of the present application can be applied to a server, such as Figure 1 As shown, the method for obtaining the data to be processed may include the following steps:

[0064] In step S101, an acquisition request is received, where the acquisition request includes the category of the list to be processed.

[0065] It is understood that the processor can perform processing operations on the respective data to be processed through the processing operation terminal. The respective data to be processed of each processing operation terminal is assigned to the processing operation terminal in the form of a list to be processed. For example, the processor can use the processing operation terminal to trigger a request to obtain the data to be processed, and perform operations such as processing on the data to be processed, wherein the processing operation terminal can be a personal computer, a mobile terminal, etc. In the embodiment of the present disclosure, the server can simultaneously receive requests to obtain the data to be processed sent by one or more processing operation terminals.

[0066] The processing operation terminal can send a request to obtain the data to be processed in many ways, such as using a remote procedure call (RPC) service to send a request to obtain the data to be processed. In addition, in order to distinguish different processing operation terminals, different identifiers can be used to distinguish them, that is, the identifier can uniquely identify a processing operation terminal.

[0067] In an embodiment of the present disclosure, after receiving a request to obtain data to be processed sent by a processing operation terminal, the category of the list to be processed can be extracted from the request to obtain. Among them, the category of the list to be processed can be understood as being used to distinguish the type of data to be processed in each list to be processed, that is, the data in the list to be processed represents data of the same category. For example, the data to be processed are all stored in a database sub-table. Since the categories to be processed may be different, the category type field is usually used to distinguish the data to be processed in the database sub-table. Therefore, the data to be processed belonging to the same category can be read out from the database sub-table and assigned to the processing operation terminal in the form of a list for processing. For example, after the data to be processed belonging to the same category is read out from the database sub-table, the read data to be processed can be placed in the corresponding category of the list to be processed according to different categories. In this way, the list to be processed can be assigned to the processing operation terminal so that the processing operation terminal processes the data in the list.

[0068] For example, taking video data as an example, assuming that a large amount of unprocessed video data is stored in the database sub-table, and these unprocessed video data belong to different categories, for example, some belong to short videos, some belong to live recorded videos, or some belong to film and television videos, or some belong to funny videos, etc. These categories of video data have corresponding fields in the database sub-table for identification. When reading the unprocessed video data from the database sub-table and assigning these unprocessed video data to the processing operation terminal in the form of a list, this application will put the unprocessed video data belonging to the same category into a pending list, that is, the unprocessed video data in the pending list belongs to the same category, so as to facilitate data reading and facilitate the processing of video data by the processing operation terminal.

[0069] In step S102, the current total amount of data to be marked in the list to be processed and the current amount of data to be marked in each sub-table belonging to the category to be processed are counted.

[0070] In the embodiment of the present disclosure, the sub-table can be understood as a database table storing the data to be processed in each list to be processed.

[0071] Among them, in the embodiment of the present disclosure, the current total amount of data to be marked in the pending list can be understood as the total amount of data waiting to be processed under the same category; the current amount to be marked can be understood as the number of marks currently waiting to be processed. Since the data to be processed is distributed and stored in each sub-table of the database, each sub-table stores the data currently waiting to be processed. In this way, when the processing operation terminal sends a request to obtain, the present application can read the data to be processed from each sub-table, and fill the read data to be processed into the pending list and assign it to the processing operation terminal. Therefore, the current amount of data to be marked that belongs to the category in each sub-table can be understood as: the number of data in each sub-table that has the same category as the data to be processed.

[0072] For example, taking the pending list A as an example, assuming that the category of the pending data in the pending list A is category A, there are 10 sub-tables in the database (such as sub-table 1, sub-table 2, ..., sub-table 10), and the total number of current pending marks of category A is 1000, that is, the current total number of pending marks in the pending list A is 1000, and the pending data in the pending list A are stored in these 10 sub-tables respectively. The number of current pending marked data of category A contained in each sub-table may be different. For example, the number of current pending marked data of category A stored in sub-table 1 is 100, and the number of current pending marked data of category A stored in sub-table 2 is 1 00, the number of currently waiting-to-be-processed marked data of category A stored in sub-table 3 is 500, the number of currently waiting-to-be-processed marked data of category A stored in sub-table 4 is 200, the number of currently waiting-to-be-processed marked data of category A stored in sub-table 5 is 20, the number of currently waiting-to-be-processed marked data of category A stored in sub-table 6 is 20, the number of currently waiting-to-be-processed marked data of category A stored in sub-table 7 is 20, the number of currently waiting-to-be-processed marked data of category A stored in sub-table 8 is 20, the number of currently waiting-to-be-processed marked data of category A stored in sub-table 9 is 10, and the number of currently waiting-to-be-processed marked data of category A stored in sub-table 10 is 10. Among them, the total number of pending data of the same category in all sub-tables should be the same as the current total number of pending markings in the pending list of that category. For example, the total number of currently pending marking data of category A in the above 10 sub-tables is the same as the current total number of pending markings in the pending list A, which is 100+100+500+200+20+20+20+20+20+10+10=1000.

[0073] In one implementation, since each row in the database sub-table represents a piece of data, and the field type is used in the sub-table to distinguish the category of the data to be processed, the current amount of data to be marked that belongs to the category in each sub-table can be determined by counting the number of data with the same type value in each sub-table. For example, taking the above database with 10 sub-tables as an example, for sub-table 1, assuming that there are 1000 pieces of data in sub-table 1, of which the number of data with the field type value of A is 100, that is, the current amount of data to be marked that belongs to category A in sub-table 1 can be determined by counting the number of data with the field type value of A in sub-table 1, for example, the current amount to be marked is 100.

[0074] In one implementation, after the current amount of to-be-marked data belonging to the category in each sub-table is obtained by counting, the current amount of to-be-marked data can be summed up, and the sum value obtained is the current total amount of to-be-marked data in the list to be processed. For example, continuing to take the above database having 10 sub-tables, and the category of the to-be-processed data in the list to be processed A is category A as an example, after the number of the current waiting-to-be-processed marked data of category A stored in each of the 10 sub-tables is counted, the number of the 10 current waiting-to-be-processed marked data can be summed up, so that the current total amount of to-be-marked data in the list to be processed A can be obtained as 1000.

[0075] In step S103, the read quantity of the to-be-processed data belonging to the category in each sub-table is determined according to the current total to-be-marked quantity of the to-be-processed list and the current to-be-marked quantity of the to-be-processed data belonging to the category in each sub-table.

[0076] In one implementation, based on the current total amount of data to be marked in the to-be-processed list and the current amount of data to be marked belonging to the category in each sub-table, the proportion of the data to be processed belonging to the category in each sub-table to the current total amount to be marked is determined, and based on the proportion of the data to be processed belonging to the category in each sub-table to the current total amount to be marked, the number of read data to be processed belonging to the category in each sub-table is determined.

[0077] For example, continuing with the above-mentioned database having 10 sub-tables and the category of the data to be processed in the to-be-processed list A being category A, the current total amount of data to be marked in the to-be-processed list A is 1000, the number of currently awaiting-processed marked data of category A stored in sub-table 1 is 100, the number of currently awaiting-processed marked data of category A stored in sub-table 2 is 100, the number of currently awaiting-processed marked data of category A stored in sub-table 3 is 500, the number of currently awaiting-processed marked data of category A stored in sub-table 4 is 200, the number of currently awaiting-processed marked data of category A stored in sub-table 5 is 20, the number of currently awaiting-processed marked data of category A stored in sub-table 6 is 20, the number of currently awaiting-processed marked data of category A stored in sub-table 7 is 20, the number of currently awaiting-processed marked data of category A stored in sub-table 8 is 20, the number of currently awaiting-processed marked data of category A stored in sub-table 9 is 10, and the number of currently awaiting-processed marked data of category A stored in sub-table 10 is 10.

[0078] In order to achieve the function of uniform distribution of mixed data streams, the embodiment of the present application calculates the proportion of the pending data belonging to category A in each sub-table to the current total amount of pending data to be marked through the current total amount of pending data to be marked in the pending list A and the current amount of pending data to be marked belonging to category A in each of the 10 sub-tables, and determines the read quantity of the pending data belonging to category A in each sub-table based on the proportion. For example, if the proportion of the data to be processed belonging to category A in sub-table 1 is calculated to be 0.1 of the current total amount to be marked, then the number of data to be processed belonging to category A in sub-table 1 should be read as 10; if the proportion of the data to be processed belonging to category A in sub-table 2 is calculated to be 0.1 of the current total amount to be marked, then the number of data to be processed belonging to category A in sub-table 2 should be read as 10; if the proportion of the data to be processed belonging to category A in sub-table 3 is calculated to be 0.5 of the current total amount to be marked, then the number of data to be processed belonging to category A in sub-table 3 should be read as 50; if the proportion of the data to be processed belonging to category A in sub-table 4 is calculated to be 0.2 of the current total amount to be marked, then the number of data to be processed belonging to category A in sub-table 4 should be read as 20; if the proportion of the data to be processed belonging to category A in sub-table 5 is calculated to be 0.02 of the current total amount to be marked, then the number of data to be processed belonging to category A in sub-table 5 should be read as 2; It is calculated that the proportion of the pending data belonging to category A in sub-table 6 accounts for 0.02 of the current total amount to be marked, so the number of pending data belonging to category A in sub-table 6 should be read as 2; it is calculated that the proportion of the pending data belonging to category A in sub-table 7 accounts for 0.02 of the current total amount to be marked, so the number of pending data belonging to category A in sub-table 7 should be read as 2; it is calculated that the proportion of the pending data belonging to category A in sub-table 8 accounts for 0.02 of the current total amount to be marked, so the number of pending data belonging to category A in sub-table 8 should be read as 2; it is calculated that the proportion of the pending data belonging to category A in sub-table 9 accounts for 0.01 of the current total amount to be marked, so the number of pending data belonging to category A in sub-table 9 should be read as 1; it is calculated that the proportion of the pending data belonging to category A in sub-table 10 accounts for 0.01 of the current total amount to be marked, so the number of pending data belonging to category A in sub-table 10 should be read as 1. It can be seen from this that the current total amount of data to be marked in the to-be-processed list and the current amount of data to be marked that belongs to the category in each sub-table can be used to determine the proportion of the to-be-processed data belonging to the category in each sub-table to the current total amount of data to be marked. In this way, the proportion can be used to read the corresponding amount of to-be-processed data from each sub-table, ensuring that the same category of to-be-processed data in each database sub-table can be read, avoiding excessive differences in the number of data of the same category read in different database sub-tables, thereby solving the problem of uneven acquisition of to-be-processed data from database sub-tables in related technologies and achieving the function of uniform distribution of mixed data streams.

[0079] In step S104, according to the read quantity of the to-be-processed data belonging to the category in each sub-table, a corresponding read quantity of the to-be-processed data is read from each sub-table.

[0080] Optionally, after obtaining the read number of data to be processed belonging to a category in each sub-table, the corresponding read number of data to be processed can be read from each sub-table, and these data to be processed can be allocated to the processing operation terminal in the form of a list (such as loading these data to be processed into a list to be processed) so that the user of the processing operation terminal (such as a processor) can perform processing operations on the data to be processed in the list to be processed.

[0081] For example, continuing to describe the example in the above step S103, the corresponding read number of to-be-processed data can be read from the 10 sub-tables according to the read number of to-be-processed data belonging to category A in the 10 sub-tables. For example, assuming that the read number of sub-table 1 belonging to category A is 10, the read number of sub-table 2 belonging to category A is 10, the read number of sub-table 3 belonging to category A is 50, the read number of sub-table 4 belonging to category A is 20, the read number of sub-table 5 belonging to category A is 2, the read number of sub-table 6 belonging to category A is 2, the read number of sub-table 7 belonging to category A is 2, the read number of sub-table 8 belonging to category A is 2, the read number of sub-table 9 belonging to category A is 1, and the read number of sub-table 10 belonging to category A is 1. 10 pieces of pending data belonging to category A can be read from sub-table 1, 10 pieces of pending data belonging to category A can be read from sub-table 2, 50 pieces of pending data belonging to category A can be read from sub-table 3, 20 pieces of pending data belonging to category A can be read from sub-table 4, 2 pieces of pending data belonging to category A can be read from sub-table 5, 2 pieces of pending data belonging to category A can be read from sub-table 6, 2 pieces of pending data belonging to category A can be read from sub-table 7, 2 pieces of pending data belonging to category A can be read from sub-table 8, 1 piece of pending data belonging to category A can be read from sub-table 9, and 1 piece of pending data belonging to category A can be read from sub-table 10.

[0082] According to the method for obtaining the data to be processed according to the embodiment of the present disclosure, an acquisition request can be received, and the acquisition request includes the category of the list to be processed; the current total amount of to-be-marked data in the list to be processed and the current amount of to-be-marked data belonging to the category in each sub-table are counted; wherein the sub-table is a database table for storing the data to be processed in each list to be processed; according to the current total amount of to-be-marked data in the list to be processed and the current amount of to-be-marked data belonging to the category in each sub-table, the number of to-be-processed data to be read belonging to the category in each sub-table is determined; according to the number of to-be-processed data to be read belonging to the category in each sub-table, the corresponding number of to-be-processed data to be read is read from each sub-table. Thus, the present disclosure can read the corresponding number of to-be-processed data from each database sub-table, solve the problem of uneven acquisition of to-be-processed data from the database sub-table in the related art, thereby achieving the function of uniform distribution of the mixed data stream, ensuring that the to-be-processed data of the same category in each database sub-table can be read, and avoiding the large difference in the number of data of the same category read in different database sub-tables.

[0083] In order to greatly reduce the query of the database sub-table and improve the response speed, optionally, the to-be-processed data of the same category from the database sub-table can be cached in respective cache units at regular intervals, so that the to-be-processed data can be read from the cache unit. Figure 2 and Figure 3 As shown, the method for obtaining the data to be processed may include the following steps:

[0084] In step S201, data to be processed of each category is read from each sub-table at a fixed time according to the category of each to-be-processed list and the identifier of each sub-table.

[0085] In one implementation, each list to be processed has its own cache unit, where the cache unit can be represented by a two-dimensional array LocalCache<TwoTuple<type, table_suffix>, Queue>. Here, the LocalCache is a field representing the cache unit, TwoTuple is a field representing the two-dimensional array, type represents the category of the data to be processed (or the list to be processed), table_suffix is an identifier used to represent the sub-table, and <type, table_suffix> in the two-dimensional array indicates in which sub-tables the data to be processed of category type is stored, and Queue represents the value of <type, table_suffix>. For example, assume that type is A, and the data to be processed of category A is stored in 3 sub-tables (such as sub-table 1, sub-table 2, and sub-table 3), then in LocalCache<TwoTuple<type, table_suffix>, Queue>, there should be: <TwoTuple<typeA, sub-table 1>, Queue 1> (where Queue1 is the value of the data to be processed of category typeA in sub-table 1, that is, this Queue can be understood as the value in something similar to key-value); <TwoTuple<typeA, sub-table 2>, Queue2> (where Queue2 is the value of the data to be processed of category typeA in sub-table 2); <TwoTuple<typeA, sub-table 3>, Queue3> (where Queue3 is the value of the data to be processed of category typeA in sub-table 3).

[0086] In the embodiments of the present disclosure, the Queue value of the list to be processed can be obtained according to the category of the list to be processed and the identifier of the sub-table. Each Queue corresponds to the data to be processed of type category in one sub-table, and the data to be processed can be periodically retrieved from the sub-table for querying so as to load the queried data to be processed into the cache unit of the corresponding list to be processed.

[0087] In step S202, the data to be processed of each category read from each sub-table is loaded into the cache unit of the list to be processed of the corresponding category.

[0088] For example, taking a sub-table as sub-table 1, sub-table 1 stores multiple categories of pending data. Assuming that sub-table 1 stores category A, category B, category C, and category D, the pending data of category A read from sub-table 1 can be loaded into the cache unit of category A's pending list a; the pending data of category B read from sub-table 1 can be loaded into the cache unit of category B's pending list b; the pending data of category C read from sub-table 1 can be loaded into the cache unit of category C's pending list c; the pending data of category D read from sub-table 1 can be loaded into the cache unit of category D's pending list d.

[0089] In step S203, an acquisition request is received, where the acquisition request includes the category of the list to be processed.

[0090] In the embodiments of the present disclosure, step S203 may be implemented in any manner in the embodiments of the present disclosure, and the embodiments of the present disclosure do not limit this and will not be described in detail.

[0091] In step S204, the current total amount of data to be marked in the list to be processed and the current amount of data to be marked in each sub-table belonging to the category are counted; wherein the sub-table is a database table storing the data to be processed in each list to be processed.

[0092] In the embodiments of the present disclosure, step S204 may be implemented in any manner in the embodiments of the present disclosure, and the embodiments of the present disclosure do not limit this and will not be described in detail.

[0093] In step S205, the read quantity of the to-be-processed data belonging to the category in each sub-table is determined according to the current total to-be-marked quantity of the to-be-processed list and the current to-be-marked quantity of the to-be-processed data belonging to the category in each sub-table.

[0094] In the embodiments of the present disclosure, step S204 may be implemented in any manner in the embodiments of the present disclosure, and the embodiments of the present disclosure do not limit this and will not be described in detail.

[0095] In step S206, according to the read quantity of to-be-processed data belonging to the category in each sub-table, the to-be-processed data of the corresponding read quantity in each sub-table is read from the target cache unit of the to-be-processed list corresponding to the category.

[0096] For example, taking two sub-tables as Sub-table 1 and Sub-table 2 respectively, there are multiple categories of data to be processed stored in Sub-table 1. Assume that the data to be processed of category A, category B, category C, and category D are stored in Sub-table 1, and the data to be processed of category A and category B are stored in Sub-table 2. The data to be processed of category A read from Sub-table 1 and the data to be processed of category A read from Sub-table 2 can be loaded into the cache unit of the processing list a of category A. The cache unit is represented by a two-dimensional array as: LocalCache<TwoTuple<Category A, Sub-table 1>, Queue1>, <TwoTuple<Category A, Sub-table 2>, Queue2>, where it means that the cache unit contains the data to be processed Queue1 of category A from Sub-table 1 and the data to be processed Queue2 of category A from Sub-table 2; the data to be processed of category B read from Sub-table 1 and the data to be processed of category B read from Sub-table 2 are loaded into the cache unit of the processing list b of category B. The cache unit is represented by a two-dimensional array as: LocalCache<TwoTuple<Category B, Sub-table 1>, Queue3>, <TwoTuple<Category B, Sub-table 2>, Queue4>, where it means that the cache unit contains the data to be processed Queue3 of category B from Sub-table 1 and the data to be processed Queue4 of category B from Sub-table 2; the data to be processed of category C read from Sub-table 1 is loaded into the cache unit of the processing list c of category C. The cache unit is represented by a two-dimensional array as: LocalCache<TwoTuple<Category C, Sub-table 1>, Queue5>, where it means that the cache unit contains the data to be processed Queue5 of category C from Sub-table 1; the data to be processed of category D read from Sub-table 1 is loaded into the cache unit of the processing list d of category D. The cache unit is represented by a two-dimensional array as: LocalCache<TwoTuple<Category D, Sub-table 1>, Queue6>, where it means that the cache unit contains the data to be processed Queue6 of category D from Sub-table 1.

[0097] Assume that it is necessary to read the data to be processed belonging to category A from the above Sub-table 1 and Sub-table 2, where the reading quantity of category A in Sub-table 1 is 10, and the reading quantity of category A in Sub-table 2 is 10. 10 pieces of the data to be processed belonging to category A can be read from the cache unit of the processing list a of category A, and from the cache unit of the processing list b of category B

[0098] 10 pieces of pending data belonging to category A are read from sub-table 2, 50 pieces of pending data belonging to category A are read from sub-table 3, 20 pieces of pending data belonging to category A are read from sub-table 4, 2 pieces of pending data belonging to category A are read from sub-table 5, 2 pieces of pending data belonging to category A are read from sub-table 6, 2 pieces of pending data belonging to category A are read from sub-table 7, 2 pieces of pending data belonging to category A are read from sub-table 8, 1 piece of pending data belonging to category A is read from sub-table 9, and 1 piece of pending data belonging to category A is read from sub-table 10.

[0099] It can be understood that since the data to be processed of each category can be read from each sub-table at regular intervals, and the data to be processed of each category read from each sub-table can be loaded into the cache unit of the to-be-processed list of the corresponding category, after receiving the request to obtain the data to be processed, the corresponding number of to-be-processed data read from each sub-table can be read from the target cache unit of the to-be-processed list corresponding to the requested category. In other words, the corresponding number of to-be-processed data can be read from the target cache unit of the to-be-processed list, so that the use of cache technology can greatly reduce the query of the database sub-table and improve the response speed.

[0100] In order to avoid repeated reading of data, in one implementation, Figure 4 As shown, the implementation method of reading the corresponding number of to-be-processed data in each sub-table from the target cache unit of the to-be-processed list corresponding to the category according to the read number of to-be-processed data in each sub-table belonging to the category may include the following steps:

[0101] In step S401, the processed data and the locked data in the target cache unit are determined.

[0102] Among them, in the embodiments of the present disclosure, processed data can be understood as data that has been processed, where a certain identifier can be used to indicate that the data has been processed. For example, the data that has been processed in the cache unit is marked as 1. For example, if a certain data is marked as 1, it can be understood that the data has been processed.

[0103] In the embodiment of the present disclosure, locked data can be understood as currently occupied data. For example, certain data or some data may be occupied, such as being assigned to other processing operation terminals for processing. For such data, such data can be marked as locked data, so that different processing operation terminals can be prevented from processing the same data at the same time.

[0104] In step S402, the processed data and locked data in the target cache unit are filtered.

[0105] In step S403, according to the read quantity of the to-be-processed data belonging to the category in each sub-table, the to-be-processed data of the corresponding read quantity in each sub-table is read from the target cache unit from which the processed data and the locked data are filtered out.

[0106] That is to say, the corresponding number of to-be-processed data of each sub-table can be read from the target cache unit excluding the processed data and the locked data, so that different processing operation terminals can be prevented from processing the same data at the same time, and repeated reading of data can be avoided.

[0107] It should be noted that in some embodiments of the present disclosure, the processing operation terminal can use the remote procedure call RPC service to obtain the data to be processed. For example, when a certain pending list is dispatched, it can be hashed to a certain RPC service instance through the list category type. Under normal circumstances, the pending data acquisition request of the same category type is always on the same instance, and can be switched to another instance after the service is restarted or crashed. In one implementation, there can be one or more cache units LocalCache of the pending list in each RPC instance. Each time the pending data is read, the corresponding data can be found from the respective cache units. The cache units of each pending list can be traversed in turn to ensure that the high-latency data in each sub-table in the cache unit will be read, which can reduce the latency of business processing and ensure the fairness of each sub-table.

[0108] Figure 5 FIG. 1 is a block diagram of a device for acquiring data to be processed according to an exemplary embodiment. Figure 5 As shown, the acquisition device may include: a receiving module 501 , a statistical module 502 , a determining module 503 and a reading module 504 .

[0109] The receiving module 501 is used to receive an acquisition request, and the acquisition request includes the category of the list to be processed;

[0110] The statistical module 502 is used to count the current total amount of data to be marked in the list to be processed and the current amount of data to be marked in each sub-table belonging to the category; wherein the sub-table is a database table storing the data to be processed in each list to be processed;

[0111] The determination module 503 is used to determine the read quantity of the data to be processed belonging to the category in each sub-table according to the current total amount of data to be marked in the list to be processed and the current amount of data to be marked in each sub-table; in one implementation, the determination module 503 is specifically used to: determine the proportion of the data to be processed belonging to the category in each sub-table to the current total amount to be marked according to the current total amount to be marked in the list to be processed and the current amount of data to be marked in each sub-table; determine the read quantity of the data to be processed belonging to the category in each sub-table according to the proportion of the data to be processed belonging to the category in each sub-table to the current total amount to be marked.

[0112] The reading module 504 is used to read a corresponding number of to-be-processed data from each sub-table according to the number of to-be-processed data belonging to a category in each sub-table.

[0113] In one implementation, Figure 6 As shown, the acquisition device may further include: a cache module 605. The cache module 505 is used to read the to-be-processed data of each category from each sub-table at a regular interval according to the category of each to-be-processed list and the identifier of each sub-table, and load the to-be-processed data of each category read from each sub-table into the cache unit of the to-be-processed list of the corresponding category.

[0114] In the embodiment of the present disclosure, the reading module 604 is specifically used to read the corresponding number of to-be-processed data in each sub-table from the target cache unit of the to-be-processed list corresponding to the category according to the read number of to-be-processed data belonging to the category in each sub-table.

[0115] In the embodiment of the present disclosure, the reading module 604 is specifically used to: determine the processed data and locked data in the target cache unit; filter the processed data and locked data in the target cache unit; and read the corresponding number of to-be-processed data in each sub-table from the target cache unit that has filtered out the processed data and locked data according to the read number of to-be-processed data belonging to the category in each sub-table. Figure 6 601-604 and Figure 5 501-504 have the same function and structure.

[0116] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0117] According to the device for acquiring data to be processed in the embodiment of the present disclosure, respective quantities of data to be processed can be read from each database sub-table, thereby solving the problem of uneven acquisition of data to be processed from database sub-tables in the related art, thereby achieving the function of uniform distribution of mixed data streams, ensuring that the same category of data to be processed in each database sub-table can be read, thereby avoiding excessive differences in the number of data of the same category read from different database sub-tables.

[0118] In order to implement the above embodiment, the embodiment of the present disclosure also proposes a server.

[0119] The server includes:

[0120] processor;

[0121] a memory for storing processor-executable instructions;

[0122] The processor is configured to execute instructions to implement the method for acquiring the data to be processed as described above.

[0123] As an example, Figure 7 is a block diagram of a server 200 for obtaining data to be processed according to an exemplary embodiment. Figure 7 As shown, the server 200 may further include:

[0124] The memory 210 and the processor 220, a bus 230 connecting different components (including the memory 210 and the processor 220), the memory 210 stores a computer program, and when the processor 220 executes the program, the method for obtaining the data to be processed described in the embodiment of the present disclosure is implemented.

[0125] Bus 230 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnect (PCI) bus.

[0126] The server 200 typically includes a variety of server-readable media. These media can be any available media that can be accessed by the server 200, including volatile and non-volatile media, removable and non-removable media.

[0127] The memory 210 may also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 240 and / or cache memory 250. The server 200 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 260 may be used to read and write non-removable, non-volatile magnetic media ( Figure 7 not shown, usually called a "hard drive"). Although Figure 7 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 230 via one or more data medium interfaces. The memory 210 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present disclosure.

[0128] A program / utility 280 having a set (at least one) of program modules 270 may be stored, for example, in the memory 210, such program modules 270 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 270 generally perform the functions and / or methods of the embodiments described in the present disclosure.

[0129] The server 200 may also communicate with one or more external devices 290 (e.g., keyboards, pointing devices, displays 291, etc.), one or more devices that enable a user to interact with the server 200, and / or any device that enables the server 200 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 292. Furthermore, the server 200 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 293. Figure 7 As shown, the network adapter 293 communicates with other modules of the server 200 via the bus 230. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the server 200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0130] The processor 220 executes various functional applications and data processing by running the programs stored in the memory 210 .

[0131] It should be noted that the implementation process and technical principles of the server of this embodiment refer to the aforementioned explanation of the method for obtaining the data to be processed in the embodiment of the present disclosure, and will not be repeated here.

[0132] The server of the disclosed embodiment can read respective quantities of to-be-processed data from each database sub-table, thereby solving the problem of uneven acquisition of to-be-processed data from database sub-tables in related technologies, thereby achieving the function of uniform distribution of mixed data streams, ensuring that the same category of to-be-processed data in each database sub-table can be read, and avoiding excessive differences in the number of data of the same category read from different database sub-tables.

[0133] In order to implement the above embodiment, the embodiment of the present disclosure also proposes a storage medium.

[0134] When the instructions in the storage medium are executed by the processor of the server, the server can execute the method for obtaining the data to be processed as described above.

[0135] In order to implement the above embodiments, the present disclosure further provides a computer program product. When the instruction processor in the computer program product executes, the server can execute the method for obtaining the data to be processed as described above.

[0136] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0137] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for obtaining video data to be processed, characterized in that: include: According to the categories of each to-be-processed list and the identifiers of each sub-table, the to-be-processed video data of each category is read from each sub-table at a regular interval, and the read to-be-processed video data of each category is loaded into the cache unit of the to-be-processed list of the corresponding category; Receiving a request for obtaining the to-be-processed video data sent by a processing operation terminal, wherein the request for obtaining includes a category of a to-be-processed list; Counting the current total amount of to-be-marked data in the to-be-processed list and the current amount of to-be-marked data of the to-be-processed video data belonging to the category in each sub-table; wherein the sub-table is a database table storing the to-be-processed video data in each to-be-processed list; Determine the proportion of the to-be-processed video data belonging to the category in each sub-table to the current total to-be-marked quantity according to the current total to-be-marked quantity and determine the read quantity of the to-be-processed video data belonging to the category in each sub-table according to the proportion; According to the read quantity, the to-be-processed video data corresponding to the read quantity of each sub-table is read from the target cache unit of the to-be-processed list corresponding to the category.

2. The method according to claim 1, characterized in that The step of reading the corresponding number of to-be-processed video data from each sub-table from a target cache unit in a to-be-processed list corresponding to the category according to the number of to-be-processed video data read from each sub-table, includes: Determine processed data and locked data in the target cache unit; Filtering the processed data and locked data in the target cache unit; According to the read quantity of the to-be-processed video data belonging to the category in each sub-table, the to-be-processed video data of the corresponding read quantity in each sub-table is read from the target cache unit from which the processed data and the locked data are filtered out.

3. A device for acquiring video data to be processed, characterized in that: include: A cache module, used for reading the to-be-processed video data of each category from each sub-table regularly according to the category of each to-be-processed list and the identifier of each sub-table, and loading the to-be-processed video data of each category read from each sub-table into the cache unit of the to-be-processed list of the corresponding category; A receiving module, used for receiving an acquisition request, wherein the acquisition request includes a category of a list to be processed; A statistics module, used to count the current total amount of to-be-marked data in the to-be-processed list and the current amount of to-be-marked video data belonging to the category in each sub-table; wherein the sub-table is a database table storing the video data to be processed in each to-be-processed list; A determination module, configured to determine the proportion of the to-be-processed video data belonging to the category in each sub-table to the current total to-be-marked quantity according to the current total to-be-marked quantity and the current to-be-marked quantity, and determine the read quantity of the to-be-processed video data belonging to the category in each sub-table according to the proportion; The reading module is used to read the to-be-processed video data of the corresponding read quantity of each sub-table from the target cache unit of the to-be-processed list corresponding to the category according to the read quantity.

4. The device according to claim 3, characterized in that The reading module is specifically used for: Determine processed data and locked data in the target cache unit; Filtering the processed data and locked data in the target cache unit; According to the read quantity of the to-be-processed video data belonging to the category in each sub-table, the to-be-processed video data of the corresponding read quantity in each sub-table is read from the target cache unit from which the processed data and the locked data are filtered out.

5. A server, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method for acquiring the video data to be processed as described in claim 1 or 2.

6. A storage medium, when instructions in the storage medium are executed by a processor of a server, enables the server to execute the method for obtaining the video data to be processed as claimed in claim 1 or 2.

7. A computer program product, comprising a computer program, characterized in that The computer program implements the steps of the method according to claim 1 or 2 when executed by a processor.

Images