Data delay processing method and device, storage medium and computer equipment

Abstract

The invention discloses a data delay processing method and device, a storage medium and computer equipment, relates to the technical field of data processing and the technical field of finance, and mainly solves the problems of high processing cost and long time consumption of a high concurrency phenomenon. Distributing the data requests to different types of chain queues according to a preset distribution rule on the basis of task information or timeliness marks carried by the data requests; determining priorities of the data requests based on historical records of the same type of data requests, and performing an in-column operation on the data requests based on the priorities to obtain to-be-processed queues corresponding to various types; according to a first-in first-out principle, sequentially obtaining identification codes corresponding to the data request from the to-be-processed queue, and obtaining corresponding parameter information based on the identification codes so as to obtain target data from the big data server based on the parameter information.

Description

Technical Field

[0001] The present invention relates to the fields of data processing technology and financial technology, and in particular to a data delay processing method and device, a storage medium, and a computer device. Background Art

[0002] With the increasing popularity of computers, information management through computer data processing has become the main application in the field of computer applications. The basic purpose of data processing is to extract and derive valuable and meaningful data for certain specific people from a large amount of data that may be disorganized and difficult to understand. The main characteristics and challenges of data processing in the era of big data are high concurrency and high real-time requirements. At the same time, there may be thousands of users accessing and operating, such as ticket purchasing platforms, online shopping platforms, and investment platforms in the financial field. Their concurrent visits can reach millions at peak times.

[0003] High concurrency poses a huge challenge to computer system resources, costs, and architecture. Existing technologies use methods such as increasing resource investment and optimizing software algorithms to improve the real-time data processing capabilities of computer systems under high concurrency conditions. However, the cost of increasing resource investment is high, and the optimization cycle of optimizing software algorithms is too long. Therefore, a low-cost and short-time data processing optimization method is urgently needed to adapt to the high concurrency phenomenon in the big data era. Summary of the invention

[0004] In view of this, the present invention provides a data delay processing method and device, a storage medium, and a computer device, the main purpose of which is to solve the existing problems of high cost and long time consumption in processing high concurrency phenomena in the big data era.

[0005] According to one aspect of the present invention, a data delay processing method is provided, comprising:

[0006] Receiving a data request to a big data server, wherein the data request carries task information or a timeliness mark;

[0007] Distributing the data request to different types of chain queues based on the task information or the timeliness mark according to a preset distribution rule;

[0008] Determine the priority of the data request based on the historical records of data requests of the same type, and perform a queue operation on the data request based on the priority to obtain a to-be-processed queue corresponding to each type;

[0009] According to the first-in-first-out principle, the identification codes corresponding to the data requests are obtained from the queue to be processed in sequence, and the corresponding parameter information is obtained based on the identification codes, so that the target data can be obtained from the big data server based on the parameter information.

[0010] Furthermore, distributing the data request to different types of chain queues based on the task information or the timeliness mark according to a preset distribution rule includes:

[0011] Dividing the chain queue according to the task type to obtain chain queues of different types; and determining the target chain queue of the data request based on the task information so as to distribute the data request to the target chain queue; or,

[0012] The chain queue is divided according to timeliness requirements to obtain different types of chain queues; and a target chain queue of the data request is determined based on the timeliness label so that the data request is distributed to the target chain queue.

[0013] Further, determining the priority of the data request based on the historical records of the same type of data requests includes:

[0014] Obtaining historical records of the same type of data requests, and obtaining historical execution time from the historical records;

[0015] The priority of the data request is determined based on the normal distribution law of the historical execution time.

[0016] Further, the determining the priority of the data request based on the normal distribution law of the historical execution time includes:

[0017] Calculate the expected time consumption value corresponding to the data request based on the historical execution time;

[0018] Eliminate abnormal values ​​from the historical execution time based on the expected time consumption to obtain filtered historical execution time;

[0019] A weighted average of the filtered historical execution times is calculated, and a priority of the data request is determined based on the weighted average.

[0020] Furthermore, after receiving the data request to the big data server, the method further includes:

[0021] Generate a corresponding identification code based on the data request;

[0022] Parameter information related to the data request is obtained, and the parameter information and the identification code are saved in a request information table in correspondence.

[0023] Further, the enqueuing operation is performed on the data request based on the priority to obtain a queue to be processed corresponding to each type, including:

[0024] Respectively obtain multiple data requests distributed to the same type of linked queue, and create a node corresponding to each of the data requests, wherein the node includes a data field and a pointer field;

[0025] The identification code corresponding to the data request is passed into the data domain, and each node is inserted into the target linked queue in the order of priority, so that the pointer in the pointer domain of the current node points to the next node, and the queue to be processed corresponding to each type is obtained.

[0026] Further, the obtaining the identification code corresponding to the data request from the queue to be processed in sequence according to the first-in-first-out principle includes:

[0027] Define a pointer P to point to the first node in the queue to be processed, and define an intermediate variable x;

[0028] If the first node is a non-empty node, obtaining a first identification code in the first node, and assigning the first identification code to the intermediate variable x;

[0029] Point the pointer P to the second node in the queue to be processed, and release the first node;

[0030] If the second node is a non-empty node, the second identification code in the second node is obtained, and the second identification code is assigned to the intermediate variable x until the pointer P points to an empty node.

[0031] According to another aspect of the present invention, a data delay processing device is provided, comprising:

[0032] A receiving module, used to receive a data request to a big data server, wherein the data request carries task information or a timeliness mark;

[0033] A distribution module, used for distributing the data request to different types of chain queues based on the task information or the timeliness mark according to a preset distribution rule;

[0034] An enqueue module, used to determine the priority of the data request based on the historical records of data requests of the same type, and enqueue the data request based on the priority to obtain a queue to be processed corresponding to each type;

[0035] The dequeue module is used to obtain the identification codes corresponding to the data requests from the queue to be processed in sequence according to the first-in-first-out principle, and obtain the corresponding parameter information based on the identification codes, so as to obtain the target data from the big data server based on the parameter information.

[0036] Furthermore, the distribution module is also used for:

[0037] Dividing the chain queue according to the task type to obtain chain queues of different types; and determining the target chain queue of the data request based on the task information so as to distribute the data request to the target chain queue; or,

[0038] The chain queue is divided according to timeliness requirements to obtain different types of chain queues; and a target chain queue of the data request is determined based on the timeliness label so that the data request is distributed to the target chain queue.

[0039] Further, the enqueue module includes a priority determination unit;

[0040] The priority determination unit is used to: obtain the historical records of the same type of data requests, and obtain the historical execution time from the historical records;

[0041] The priority of the data request is determined based on the normal distribution law of the historical execution time.

[0042] The priority determination unit is further used to: calculate the expected time consumption value corresponding to the data request based on the historical execution time;

[0043] Eliminate abnormal values ​​from the historical execution time based on the expected time consumption to obtain filtered historical execution time;

[0044] A weighted average of the filtered historical execution times is calculated, and a priority of the data request is determined based on the weighted average.

[0045] Furthermore, the device also includes a request entry module, which is used to:

[0046] Generate a corresponding identification code based on the data request;

[0047] Parameter information related to the data request is obtained, and the parameter information and the identification code are saved in a request information table in correspondence.

[0048] Furthermore, the enqueue module further comprises an enqueue request unit;

[0049] The request enqueue unit is used to: respectively obtain multiple data requests distributed to the same type of linked queue, and create a node corresponding to each of the data requests, wherein the node includes a data field and a pointer field;

[0050] The identification code corresponding to the data request is passed into the data domain, and each node is inserted into the target linked queue in the order of priority, so that the pointer in the pointer domain of the current node points to the next node, and the queue to be processed corresponding to each type is obtained.

[0051] Furthermore, the dequeue module is also used for:

[0052] Define a pointer P to point to the first node in the queue to be processed, and define an intermediate variable x;

[0053] If the first node is a non-empty node, obtaining a first identification code in the first node, and assigning the first identification code to the intermediate variable x;

[0054] Point the pointer P to the second node in the queue to be processed, and release the first node;

[0055] If the second node is a non-empty node, the second identification code in the second node is obtained, and the second identification code is assigned to the intermediate variable x until the pointer P points to an empty node.

[0056] According to another aspect of the present invention, a storage medium is provided, wherein the storage medium stores at least one executable instruction, wherein the executable instruction enables a processor to execute operations corresponding to the above-mentioned data delay processing method.

[0057] According to another aspect of the present invention, there is provided a computer device, comprising a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus;

[0058] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the above-mentioned data delay processing method.

[0059] By means of the above technical solution, the technical solution provided by the embodiment of the present invention has at least the following advantages:

[0060] The present invention provides a data delay processing method and device, a storage medium, and a computer device. Compared with the prior art, the present invention distributes data requests to different types of chain queues through preset distribution rules, which can meet the diversity requirements of data delay processing and improve the scope of application of data delay processing. The present invention determines the priority of each data request through the historical records of the same type of data requests, which can effectively reduce the impact of long waiting of other requests caused by time-consuming operations with low priority, and can improve the response speed in high concurrency scenarios. And with the accumulation of scheduling sample data, the deviation of prediction time consumption will continue to shrink, and the prediction will be more accurate. The method of the present invention delays data requests with low real-time requirements, which can release CPU resources and use them for real-time calculations in high-concurrency scenarios. Under the premise that server resources remain unchanged, the system concurrency capability can be greatly improved. In addition, the chain queue adopted has obvious advantages in time complexity, and the larger the amount of data, the more obvious the accumulated efficiency advantage.

[0061] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0063] Figure 1 A schematic diagram of a process flow of a data delay processing method provided by an embodiment of the present invention is shown;

[0064] Figure 2 A schematic diagram showing a flow chart of another data delay processing method provided by an embodiment of the present invention;

[0065] Figure 3 A schematic diagram showing a flow chart of another data delay processing method provided by an embodiment of the present invention;

[0066] Figure 4 A schematic diagram showing a flow chart of another data delay processing method provided by an embodiment of the present invention;

[0067] Figure 5 A schematic diagram showing the structure of a data delay processing device provided by an embodiment of the present invention is shown;

[0068] Figure 6A schematic diagram of the structure of a computer device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0069] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0070] The embodiment of the present invention provides a data delay processing method, such as Figure 1 As shown, the method includes:

[0071] 101. Receive a data request to a big data server, where the data request carries task information or a timeliness mark;

[0072] In an embodiment of the present invention, the current execution end receives data requests from multiple business systems to a big data server, such as receiving data requests from financial settlement services in the financial field, data requests from report generation services, etc., which are not specifically limited in the embodiment of the present invention. In an embodiment of the present invention, the data request obtained by the current execution end carries task information or a timeliness mark. Among them, the task information may include a task name, a task association relationship, etc., which are not specifically limited in the embodiment of the present invention. Among them, the timeliness mark is a mark defined according to the timeliness requirements of the processed data, such as, after sorting the failure requirements from high to low, set timeliness marks such as t, t+1, t+2, t+3, t+4, etc., wherein t can represent a data request that needs to be processed in real time, t+1 represents a data request that is slightly lower than the real-time processing requirement, t+2 represents a data request that is slightly lower than the timeliness requirement of t+1, and so on, which are not specifically limited in the embodiment of the present invention.

[0073] 102. Distribute the data request to different types of chain queues based on the task information or the timeliness mark according to a preset distribution rule;

[0074] In the embodiment of the present invention, the current execution end distributes the data request to different types of chain queues based on the task information or timeliness mark carried by the data request in step 101 according to the preset distribution rules. Since the business processes of various business platforms have upstream and downstream associations when processing data, such as the settlement business in the financial field, hundreds of associated systems will send DATA to the settlement system after the start of the financial settlement period, with large amounts and diverse calibers, and serious upstream and downstream dependence of DATA (the DATA of the b system must be calculated based on the DATA results of the a system), the urgency of the task can be determined according to the task information or timeliness mark, and data requests of different urgency can be distributed to chain queues of various types with the same urgency, which is not specifically limited in the embodiment of the present invention.

[0075] 103. Determine the priority of the data request based on the historical records of data requests of the same type, and perform a queue operation on the data request based on the priority to obtain a to-be-processed queue corresponding to each type;

[0076] In an embodiment of the present invention, since the historical records of the same type of data request have the same or similar execution time as the current data request, the current execution end obtains the historical records of the same type of data request to predict the execution time of the current data request. The current execution end determines the priority of the current data request based on the execution time predicted by the historical records of the same type of data request. Among them, the priority can represent the order of data request processing, such as using numbers 0000, 0001, 0002, 0003, etc. to represent the priority from high to low, which is not specifically limited in the embodiment of the present invention.

[0077] In the embodiment of the present invention, the current execution end performs a queue operation on each data request based on the determined priority order, that is, stores the relevant information of the data request in the corresponding chain queue to obtain a queue to be processed corresponding to each type.

[0078] 104. Obtain identification codes corresponding to the data request from the queue to be processed in sequence according to the first-in-first-out principle, and obtain corresponding parameter information based on the identification codes, so as to obtain target data from the big data server based on the parameter information.

[0079] In an embodiment of the present invention, the current execution end obtains the identification code corresponding to the data request from the queue to be processed in sequence according to the first-in-first-out principle. The identification code is identity information that characterizes the uniqueness of the data request, that is, the identification code of each data request is different. The current execution end obtains the corresponding parameter information based on the identification code, wherein the parameter information includes the interface name, method name, input parameter, output parameter information, etc., which is not specifically limited in the embodiment of the present invention. The current execution end then obtains the target data from the corresponding big data server based on the obtained parameter information.

[0080] Further, as a refinement and extension of the specific implementation of the above embodiment, in order to perform data delay processing in a more hierarchical manner, another data delay processing method is provided, such as Figure 2 As shown, the steps distribute the data request to different types of chain queues based on the task information or the timeliness mark according to a preset distribution rule, including:

[0081] 201a. Dividing the chain queue according to the task type to obtain chain queues of different types;

[0082] 202a. Determine a target chain queue of the data request based on the task information, so as to distribute the data request to the target chain queue;

[0083] In the embodiment of the present invention, the current execution end divides and processes the chain queue according to the task type according to the preset distribution rules. For example, in the financial field, the chain queue can be divided according to the transaction task type, the settlement task type, the statistical task type, etc., which is not specifically limited in the embodiment of the present invention. After the chain queues of different types are obtained, the current execution end determines the target chain queue corresponding to the data request based on the task information carried by the data request, such as the task name, the task association relationship, etc., and distributes the data request to the corresponding target chain queue.

[0084] 201b. Dividing the chain queue according to timeliness requirements to obtain chain queues of different types;

[0085] 202b. Determine a target chain queue of the data request based on the timeliness tag, so as to distribute the data request to the target chain queue.

[0086] In an embodiment of the present invention, in addition to the preset distribution rules described in steps 201a to 202a above, the current execution end can also divide the chain queue according to the timeliness requirements to obtain different types of chain queues; for example, the chain queue is divided into chain queues corresponding to each timeliness tag, timeliness tag t corresponds to type one chain queue; timeliness tag t+1 corresponds to type two chain queue; timeliness tag t+2 corresponds to type three chain queue, etc., which are not specifically limited in the embodiment of the present invention. After the chain queues of different types are divided, the current execution end determines the target chain queue of the data request based on the timeliness tag, such as, if the timeliness tag carried by the current data request is t+1, the chain queue of type two is determined as the target chain queue; if the timeliness tag carried by the current data request is t+2, the chain queue of type three is determined as the target chain queue, etc., which are not specifically limited in the embodiment of the present invention. After determining the target chain queue, the current execution end distributes the data request to the corresponding target chain queue.

[0087] Further, as a refinement and extension of the specific implementation of the above embodiment, in order to accurately determine the order in which each data request is queued, another data delay processing method is provided, such as Figure 3 As shown, the step of determining the priority of the data request based on the historical records of the same type of data requests includes:

[0088] 301. Obtain historical records of the same type of data requests, and obtain historical execution time from the historical records;

[0089] 302. Determine the priority of the data request based on the normal distribution rule of the historical execution time.

[0090] In the embodiment of the present invention, since the historical records of the same type of data request have the same or similar execution time as the current data request, the current execution end obtains the historical records of the same type of data request to predict the execution time of the current data request. The current execution end obtains the historical records of the same type of data request, obtains the historical execution time from the historical records, and then determines the priority of the data request based on the normal distribution law of each historical execution time obtained.

[0091] It should be noted that the step of determining the priority of the data request based on the normal distribution law of the historical execution time includes:

[0092] (1) Calculate the expected time value corresponding to the data request based on the historical execution time; the specific formula is as follows:

[0093]

[0094] Among them, E[X] is the expected value of time consumption, X is the discrete historical execution time, and the output value is x 1 、x 2 、x 3 ......, and the probability corresponding to the above output value is p 1 , P 2 , P 3 ...(the sum of probabilities is 1).

[0095] (2) based on the expected time value, outliers are removed from the historical execution time to obtain filtered historical execution time; wherein, when removing outliers, a deviation threshold may be set, and when the deviation is higher than the deviation threshold, the corresponding data is removed, etc., which is not specifically limited in the embodiment of the present invention. The data of the filtered historical execution time obtained after the removal is smoother.

[0096] (3) Calculate the weighted average of the filtered historical execution time, and determine the priority of the data request based on the weighted average. The specific formula for calculating the weighted average is as follows:

[0097]

[0098] Among them, x 1 、x 2 、x 3 ......is the filtered historical execution time, f 1 、f 2 、f 3 ......is the weight corresponding to each historical execution time, and n is the total number of historical records after filtering.

[0099] In the embodiment of the present invention, the weighted average of the historical execution time of the same type of data requests corresponding to each data request can be obtained through the calculation of the above steps (1), (2), and (3). In the embodiment of the present invention, the obtained weighted average of the historical execution time is used as the execution time prediction value of the current corresponding data request of the same type, and the prediction value can reflect the execution speed of the data request of this type. Therefore, the current execution end can assign a high priority to the data request with a small execution time prediction value, and assign a low priority to the data request with a large execution time prediction value, which is not specifically limited in the embodiment of the present invention.

[0100] Further, as a refinement and extension of the specific implementation of the above embodiment, in order to facilitate the data request enqueue and dequeue operations, another data delay processing method is provided. After receiving the data request to the big data server, the method further includes:

[0101] Generate a corresponding identification code based on the data request;

[0102] Parameter information related to the data request is obtained, and the parameter information and the identification code are saved in a request information table in correspondence.

[0103] In an embodiment of the present invention, after the current execution end receives a data request for a big data server, a corresponding identification code is generated based on the data request, wherein the identification code is identity information that characterizes the uniqueness of the data request, that is, the identification code of each data request is different. The data request also carries parameter information for accessing the data, and the parameter information includes interface name, method name, input parameter, output parameter information, etc., which is not specifically limited in the embodiment of the present invention. After receiving the data request for the big data server, the current execution end also needs to obtain parameter information related to the data request, and save the obtained parameter information and the generated corresponding identification code in the request information table in correspondence. Among them, the request information table records the relevant parameter information of the data access request sent by each business platform to the big data server.

[0104] Further, as a refinement and extension of the specific implementation of the above embodiment, in order to perform data queueing operations in a more orderly manner, another data delay processing method is provided, such as Figure 4 As shown, the step of enqueuing the data request based on the priority to obtain a queue to be processed corresponding to each type includes:

[0105] 401. Respectively obtain multiple data requests distributed to the same type of linked queue, and create a node corresponding to each of the data requests, wherein the node includes a data field and a pointer field;

[0106] In an embodiment of the present invention, the current execution end obtains multiple data requests distributed to the same type of linked queue, and creates nodes corresponding to the data requests in the current type of linked queue. For example, if there are m data requests distributed to type one linked queue, then m nodes are created in type one linked queue; if there are h data requests distributed to type two linked queue, then h nodes are created in type two linked queue, etc., which are not specifically limited in the embodiment of the present invention. The above-mentioned created node includes a data field and a pointer field for storing relevant information of the data request.

[0107] 402. The identification code corresponding to the data request is passed into the data domain, and each node is inserted into the target linked queue in the order of priority, so that the pointer in the pointer domain of the current node points to the next node, and the queue to be processed corresponding to each type is obtained.

[0108] In the embodiment of the present invention, the current execution end passes the identification code corresponding to the data request in the data domain, and inserts each node into the target chain queue in the order of priority set from step 301 to step 302, and makes the pointer of the pointer domain of the current node point to the next node. After the relevant information of the data request is inserted into each target chain queue, a queue to be processed corresponding to each type is obtained. Among them, the queue to be processed is a chain queue for data delay processing.

[0109] Further, as a refinement and extension of the specific implementation of the above embodiment, in order to dequeue data in a more orderly manner, another data delay processing method is provided, wherein the steps of sequentially obtaining the identification code corresponding to the data request from the queue to be processed according to the first-in-first-out principle include:

[0110] Define a pointer P to point to the first node in the queue to be processed, and define an intermediate variable x;

[0111] If the first node is a non-empty node, obtaining a first identification code in the first node, and assigning the first identification code to the intermediate variable x;

[0112] Point the pointer P to the second node in the queue to be processed, and release the first node;

[0113] If the second node is a non-empty node, the second identification code in the second node is obtained, and the second identification code is assigned to the intermediate variable x until the pointer P points to an empty node.

[0114] In an embodiment of the present invention, the current execution end defines a pointer P to point to the first node in the queue to be processed, wherein the first node represents the node corresponding to the data request with the highest priority in the queue to be processed. In order to release the node in time after obtaining the information in the queue to be processed, the current execution end also defines an intermediate variable x. The current execution end determines whether there is information in the first node. If the first node is a non-empty node, the first identification code stored in the first node is obtained, and the amplitude of the obtained first identification code is given to the defined intermediate variable x, that is, the relevant information of the data request with the highest priority is extracted from the queue to be processed. Then, the corresponding parameter information of the access data can be obtained from the request information table according to the first identification code in the intermediate variable x. The parameter information includes the interface name, method name, input parameter, output parameter information, etc., and the target data in the big data server is accessed based on the parameter information.

[0115] Afterwards, the current execution end points the pointer P to the second node in the queue to be processed, and releases the first node from which the information has been extracted. At this time, the pointer P still points to the node with the highest priority in the entire queue to be processed. Next, referring to the operation on the first node, the current execution end determines whether there is information in the second node. If the second node is a non-empty node, the second identification code stored in the second node is obtained, and the obtained second identification code amplitude is given to the defined intermediate variable x. Then, the corresponding parameter information of the access data can be obtained from the request information table according to the second identification code in the intermediate variable x. The parameter information includes the interface name, method name, input parameter, output parameter information, etc. The target data in the big data server is accessed based on the parameter information until the pointer P points to an empty node, and the data request is dequeued.

[0116] The embodiment of the present invention provides a data delay processing method. Compared with the prior art, the present invention distributes data requests to different types of chain queues through preset distribution rules, which can meet the diversity requirements of data delay processing and improve the scope of application of data delay processing. The present invention determines the priority of each data request through the historical records of data requests of the same type, which can effectively reduce the impact of long waiting of other requests caused by time-consuming operations with low priority, and can improve the response speed in high concurrency scenarios. And with the accumulation of scheduling sample data, the deviation of prediction time consumption will continue to shrink, and the prediction will be more accurate. The method of the present invention delays data requests with low real-time requirements, which can release CPU resources and use them for real-time calculations in high-concurrency scenarios. Under the premise that server resources remain unchanged, the system concurrency capability can be greatly improved. In addition, the chain queue adopted has obvious advantages in time complexity, and the larger the amount of data, the more obvious the accumulated efficiency advantage.

[0117] As the above Figure 1 In order to realize the method shown in the figure, the embodiment of the present invention provides a data delay processing device, such as Figure 5 As shown, the device comprises:

[0118] A receiving module 51 is used to receive a data request to a big data server, wherein the data request carries task information or a timeliness mark;

[0119] A distribution module 52, configured to distribute the data request to different types of chain queues based on the task information or the timeliness mark according to a preset distribution rule;

[0120] An enqueue module 53, configured to determine the priority of the data request based on the historical records of data requests of the same type, and perform an enqueue operation on the data request based on the priority to obtain a to-be-processed queue corresponding to each type;

[0121] The dequeue module 54 is used to obtain the identification code corresponding to the data request from the queue to be processed in sequence according to the first-in-first-out principle, and obtain the corresponding parameter information based on the identification code, so as to obtain the target data from the big data server based on the parameter information.

[0122] Furthermore, the distribution module 52 is also used for:

[0123] Dividing the chain queue according to the task type to obtain chain queues of different types; and determining the target chain queue of the data request based on the task information so as to distribute the data request to the target chain queue; or,

[0124] The chain queue is divided according to timeliness requirements to obtain different types of chain queues; and a target chain queue of the data request is determined based on the timeliness label so that the data request is distributed to the target chain queue.

[0125] Further, the queueing module 53 includes a priority determination unit;

[0126] The priority determination unit is used to: obtain the historical records of the same type of data requests, and obtain the historical execution time from the historical records;

[0127] The priority of the data request is determined based on the normal distribution law of the historical execution time.

[0128] The priority determination unit is further used to: calculate the expected time consumption value corresponding to the data request based on the historical execution time;

[0129] Eliminate abnormal values ​​from the historical execution time based on the expected time consumption to obtain filtered historical execution time;

[0130] A weighted average of the filtered historical execution times is calculated, and a priority of the data request is determined based on the weighted average.

[0131] Furthermore, the device also includes a request entry module, which is used to:

[0132] Generate a corresponding identification code based on the data request;

[0133] Parameter information related to the data request is obtained, and the parameter information and the identification code are saved in a request information table in correspondence.

[0134] Furthermore, the enqueue module further comprises an enqueue request unit;

[0135] The request enqueue unit is used to: respectively obtain multiple data requests distributed to the same type of linked queue, and create a node corresponding to each of the data requests, wherein the node includes a data field and a pointer field;

[0136] The identification code corresponding to the data request is passed into the data domain, and each node is inserted into the target linked queue in the order of priority, so that the pointer in the pointer domain of the current node points to the next node, and the queue to be processed corresponding to each type is obtained.

[0137] Furthermore, the dequeue module 54 is also used for:

[0138] Define a pointer P to point to the first node in the queue to be processed, and define an intermediate variable x;

[0139] If the first node is a non-empty node, obtaining a first identification code in the first node, and assigning the first identification code to the intermediate variable x;

[0140] Point the pointer P to the second node in the queue to be processed, and release the first node;

[0141] If the second node is a non-empty node, the second identification code in the second node is obtained, and the second identification code is assigned to the intermediate variable x until the pointer P points to an empty node.

[0142] The embodiment of the present invention provides a data delay processing device. Compared with the prior art, the present invention distributes data requests to different types of chain queues through preset distribution rules, which can meet the diversity requirements of data delay processing and improve the scope of application of data delay processing. The present invention determines the priority of each data request through the historical records of the same type of data requests, which can effectively reduce the impact of long waiting time of other requests caused by time-consuming operations with low priority, and improve the response speed in high concurrency scenarios. And with the accumulation of scheduling sample data, the deviation of prediction time consumption will continue to decrease, and the prediction will be more accurate. The method of the present invention delays data requests with low real-time requirements, which can release CPU resources for real-time calculations in high-concurrency scenarios, and can greatly improve the system concurrency capability under the premise that server resources remain unchanged. In addition, the chain queue used has obvious advantages in time complexity, and the larger the amount of data, the more obvious the accumulated efficiency advantage.

[0143] According to an embodiment of the present invention, a storage medium is provided, wherein the storage medium stores at least one executable instruction, and the computer executable instruction can execute the data delay processing method in any of the above method embodiments.

[0144] Figure 6A schematic diagram of the structure of a computer device provided according to an embodiment of the present invention is shown. The specific embodiment of the present invention does not limit the specific implementation of the computer device.

[0145] like Figure 6 As shown, the computer device may include: a processor (processor) 602 , a communication interface (Communications Interface) 604 , a memory (memory) 606 , and a communication bus 608 .

[0146] The processor 602 , the communication interface 604 , and the memory 606 communicate with each other via a communication bus 608 .

[0147] The communication interface 604 is used to communicate with other devices such as clients or other servers.

[0148] The processor 602 is used to execute the program 610, and specifically can execute the relevant steps of the above-mentioned data delay processing method.

[0149] Specifically, the program 610 may include program codes, which include computer operation instructions.

[0150] The processor 602 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the computer device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0151] The memory 606 is used to store the program 610. The memory 606 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0152] The program 610 may be specifically configured to enable the processor 602 to perform the following operations:

[0153] Receiving a data request to a big data server, wherein the data request carries task information or a timeliness mark;

[0154] Distributing the data request to different types of chain queues based on the task information or the timeliness mark according to a preset distribution rule;

[0155] Determine the priority of the data request based on the historical records of data requests of the same type, and perform a queue operation on the data request based on the priority to obtain a to-be-processed queue corresponding to each type;

[0156] According to the first-in-first-out principle, the identification codes corresponding to the data requests are obtained from the queue to be processed in sequence, and the corresponding parameter information is obtained based on the identification codes, so that the target data can be obtained from the big data server based on the parameter information.

[0157] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0158] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data delay processing method, characterized in that: include: Receiving a data request to a big data server, wherein the data request carries task information or a timeliness mark; Distributing the data request to different types of chain queues based on the task information or the timeliness mark according to a preset distribution rule; Determine the priority of the data request based on the historical records of data requests of the same type, and perform a queue operation on the data request based on the priority to obtain a to-be-processed queue corresponding to each type; According to the first-in-first-out principle, the identification codes corresponding to the data requests are obtained from the queue to be processed in sequence, and the corresponding parameter information is obtained based on the identification codes, so that the target data can be obtained from the big data server based on the parameter information.

2. The method according to claim 1, characterized in that The distributing the data request to different types of chain queues based on the task information or the timeliness mark according to a preset distribution rule includes: Dividing the chain queue according to the task type to obtain different types of chain queues; and determining the target chain queue of the data request based on the task information, so that the data request is distributed to the target chain queue; or, The chain queue is divided according to timeliness requirements to obtain different types of chain queues; and a target chain queue of the data request is determined based on the timeliness label so that the data request is distributed to the target chain queue.

3. The method according to claim 1, characterized in that The determining the priority of the data request based on the historical records of the same type of data requests includes: Obtaining historical records of the same type of data requests, and obtaining historical execution time from the historical records; The priority of the data request is determined based on the normal distribution law of the historical execution time.

4. The method according to claim 3, characterized in that The determining the priority of the data request based on the normal distribution law of the historical execution time includes: Calculate the expected time consumption value corresponding to the data request based on the historical execution time; Eliminate abnormal values ​​from the historical execution time based on the expected time consumption to obtain filtered historical execution time; A weighted average of the filtered historical execution times is calculated, and a priority of the data request is determined based on the weighted average.

5. The method according to claim 1, characterized in that After receiving the data request to the big data server, the method further includes: Generate a corresponding identification code based on the data request; Parameter information related to the data request is obtained, and the parameter information and the identification code are saved in a request information table in correspondence.

6. The method according to claim 1, characterized in that The enqueuing operation is performed on the data request based on the priority to obtain a queue to be processed corresponding to each type, including: Respectively obtain multiple data requests distributed to the same type of linked queue, and create a node corresponding to each of the data requests, wherein the node includes a data field and a pointer field; The identification code corresponding to the data request is passed into the data domain, and each node is inserted into the target linked queue in the order of priority, so that the pointer in the pointer domain of the current node points to the next node, and the queue to be processed corresponding to each type is obtained.

7. The method according to claim 1, characterized in that The obtaining the identification code corresponding to the data request from the queue to be processed in sequence according to the first-in-first-out principle includes: Define a pointer P to point to the first node in the queue to be processed, and define an intermediate variable x; If the first node is a non-empty node, obtaining a first identification code in the first node, and assigning the first identification code to the intermediate variable x; Point the pointer P to the second node in the queue to be processed, and release the first node; If the second node is a non-empty node, the second identification code in the second node is obtained, and the second identification code is assigned to the intermediate variable x until the pointer P points to an empty node.

8. A data delay processing device, characterized in that: include: A receiving module, used to receive a data request to a big data server, wherein the data request carries task information or a timeliness mark; A distribution module, used for distributing the data request to different types of chain queues based on the task information or the timeliness mark according to a preset distribution rule; An enqueue module, used to determine the priority of the data request based on the historical records of data requests of the same type, and enqueue the data request based on the priority to obtain a queue to be processed corresponding to each type; The dequeue module is used to obtain the identification codes corresponding to the data requests from the queue to be processed in sequence according to the first-in-first-out principle, and obtain the corresponding parameter information based on the identification codes, so as to obtain the target data from the big data server based on the parameter information.

9. A storage medium, wherein at least one executable instruction is stored in the storage medium, and the executable instruction executes an operation corresponding to the data delay processing method according to any one of claims 1 to 7.

10. A computer device, comprising a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the data delay processing method according to any one of claims 1-7.

Images