Disordered data processing method, device, equipment and readable storage medium

Abstract

The present invention discloses a method, device, equipment and readable storage medium for processing out-of-order data, the method comprising: reading the data to be processed from the message queue based on a preset stream processing framework, and judging whether the time window matching the generation time of the data to be processed is in a clearing state; if in a clearing state, judging that the data to be processed is out-of-order data, re-forming the time window based on the preset stream processing framework, and reading the historical clearing data corresponding to the time window, and restoring the clearing data to the time window; allocating the data to be processed to the time window based on the preset stream processing framework, and aggregating with the historical clearing data restored to the time window, to complete the processing of the out-of-order data. The present invention ensures comprehensive and accurate statistics of data in the real-time calculation process by restoring the historical clearing data to the time window, and allocating the data to be processed determined as out-of-order data to the time window, and aggregating with the restored historical clearing data.

Description

Technical Field

[0001] The present invention relates to the technical field of financial technology (Fintech), and in particular to a method, device, equipment and readable storage medium for processing out-of-order data. Background Art

[0002] With the continuous development of financial technology (Fintech), especially Internet technology finance, more and more technologies (such as artificial intelligence, big data, cloud storage, etc.) are applied in the financial field, but the financial field also puts forward higher requirements for various technologies, such as requiring accurate statistics of large amounts of data in real-time computing.

[0003] In current real-time computing scenarios, data disorder is a very common problem. Accurate processing of disordered data has a significant impact on the accuracy of data statistics. The Flink framework is widely used in real-time computing scenarios due to its efficient processing characteristics. It allocates data to windows and implements data statistics through window state management. However, the Flink framework discards disordered data, which results in inaccurate statistical data.

[0004] Therefore, how to rely on the Flink framework to accurately count the disordered data in the real-time computing process is a technical problem that needs to be solved urgently. Summary of the invention

[0005] The main purpose of the present invention is to provide a method, device, equipment and readable storage medium for processing disordered data, aiming to solve the technical problem of how to rely on the Flink framework to accurately count disordered data in real-time computing processes in the prior art.

[0006] To achieve the above object, the present invention provides a method for processing out-of-order data, the method comprising the following steps:

[0007] Reading the data to be processed from the message queue based on a preset stream processing framework, and determining whether a time window matching the generation time of the data to be processed is in a cleared state;

[0008] If it is in the clearing state, the data to be processed is determined to be disordered data, the time window is re-formed based on the preset stream processing framework, and the historical clearing data corresponding to the time window is read, and the historical clearing data is restored to the time window;

[0009] The data to be processed is allocated to the time window based on the preset stream processing framework, and is aggregated with the historical cleared data restored to the time window to complete the processing of the disordered data.

[0010] Optionally, the step of allocating the to-be-processed data to the time window based on the preset stream processing framework and aggregating the historical cleared data restored to the time window includes:

[0011] Allocating the to-be-processed data to the time window based on the preset stream processing framework, and adding a delay duration to the time window;

[0012] Determine whether the target data to be processed is read within the delay time, wherein the generation time of the target data to be processed matches the time window;

[0013] If the target data to be processed is read, the target data to be processed is allocated to the time window, and after the delay period ends, the data in the time window are aggregated.

[0014] Optionally, after the step of determining whether the time window matching the generation time of the data to be processed is in a cleared state, the method further includes:

[0015] If the time window matching the generation time of the data to be processed is in an uncleared state, the data to be processed is allocated to the time window, and it is determined whether the generation time of the data to be processed reaches the watermark of the time window;

[0016] If the water level of the time window is reached, the data within the time window are aggregated and processed to generate a processing result which is stored in a preset database.

[0017] Optionally, after the step of generating and storing the processing result in a preset database, the method further includes:

[0018] Transfer all data within the time window to the preset database cache, and after the cache is successful, clear all data within the time window;

[0019] The watermark of the time window is adjusted to read new data to be processed in the message queue for aggregation processing.

[0020] Optionally, after the step of transmitting all data within the time window to the preset database cache, the method further includes:

[0021] When it is detected that the cleaning time of the preset database has arrived, the data to be deleted in each data cached in the preset database is searched, and a deletion operation is performed on the data to be deleted.

[0022] Optionally, the step of determining whether the time window matching the generation time of the data to be processed is in a cleared state includes:

[0023] Reading a window time interval of a current time window in an activated state, and determining whether a generation time of the data to be processed is within the window time interval;

[0024] If it is within the window time interval, it is determined that the time window matching the generation time of the data to be processed is in an uncleared state;

[0025] If it is not within the window time interval, it is determined that the time window matching the generation time of the data to be processed is in a cleared state.

[0026] Optionally, the step of reading the data to be processed from the message queue based on the preset stream processing framework includes:

[0027] Read a single piece of data from a message queue based on a preset stream processing framework, and call a preset function to extract data corresponding to a preset field from the single piece of data;

[0028] The data corresponding to the preset field is determined as the data to be processed.

[0029] Furthermore, in order to achieve the above object, the present invention also provides a disordered data processing device, the disordered data processing device comprising:

[0030] A judgment module, used to read the data to be processed from the message queue based on a preset stream processing framework, and judge whether a time window matching the generation time of the data to be processed is in a cleared state;

[0031] A recovery module, configured to determine that the data to be processed is out-of-order data if it is in a clearing state, reform the time window based on the preset stream processing framework, read historical clearing data corresponding to the time window, and restore the historical clearing data to the time window;

[0032] An aggregation module is used to allocate the to-be-processed data to the time window based on the preset stream processing framework, and aggregate the to-be-processed data with the historical cleared data restored to the time window to complete the processing of the disordered data.

[0033] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a disordered data processing device, which includes a memory, a processor, and a disordered data processing program stored in the memory and executable on the processor, and the disordered data processing program, when executed by the processor, implements the steps of the disordered data processing method as described above.

[0034] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a readable storage medium, on which a disordered data processing program is stored, and when the disordered data processing program is executed by a processor, the steps of the disordered data processing method as described above are implemented.

[0035] The out-of-order data processing method, device, equipment and readable storage medium of the present invention first read the data to be processed from the message queue based on the preset stream processing framework, and determine whether the time window matching the generation time of the data to be processed is in a clearing state; if it is determined to be in a clearing state, it means that the data to be processed is out-of-order data, and the time window is re-formed based on the preset stream processing framework, and the historical clearing data corresponding to the time window is read, and the historical clearing data is restored to the time window; then the data to be processed is assigned to the time window based on the preset stream processing framework, and the data is aggregated together with the historical clearing data restored to the time window to complete the processing of the out-of-order data. After the present invention determines that the data to be processed is out-of-order data by the clearing state of the time window, the time window is re-formed, and the historical clearing data cleared from the time window after processing is read and restored to the time window again, and then the data to be processed is assigned to the time window, and the data is aggregated with the restored historical clearing data, so as to avoid the inaccuracy of data statistics caused by the discarding of out-of-order data, and ensure the comprehensive and accurate statistics of data in the real-time calculation process. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a schematic diagram of the structure of the device hardware operating environment involved in the embodiment of the out-of-order data processing device of the present invention;

[0037] Figure 2 It is a flowchart of a first embodiment of a method for processing out-of-order data according to the present invention;

[0038] Figure 3 A schematic diagram of functional modules of a preferred embodiment of the out-of-order data processing device of the present invention;

[0039] Figure 4 It is a flowchart of out-of-order data processing based on the Flink framework in a specific embodiment of the out-of-order data processing method of the present invention.

[0040] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0041] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0042] The present invention provides a disordered data processing device, referring to Figure 1 , Figure 1It is a schematic diagram of the structure of the device hardware operating environment involved in the embodiment of the out-of-order data processing device of the present invention.

[0043] like Figure 1 As shown, the disordered data processing device may include: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or it may be a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may optionally be a storage device independent of the aforementioned processor 1001.

[0044] Those skilled in the art will understand that Figure 1 The hardware structure of the out-of-order data processing device shown in the figure does not constitute a limitation on the out-of-order data processing device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0045] like Figure 1 As shown, the memory 1005 as a readable storage medium may include an operating system, a network communication module, a user interface module and a disordered data processing program. The operating system is a program for managing and controlling disordered data processing equipment and software resources, and supports the operation of the network communication module, the user interface module, the disordered data processing program and other programs or software; the network communication module is used to manage and control the network interface 1004; the user interface module is used to manage and control the user interface 1003.

[0046] exist Figure 1 In the hardware structure of the out-of-order data processing device shown, the network interface 1004 is mainly used to connect to the background server and communicate data with the background server; the user interface 1003 is mainly used to connect to the client (user end) and communicate data with the client; the processor 1001 can call the out-of-order data processing program stored in the memory 1005 and perform the following operations:

[0047] Reading the data to be processed from the message queue based on a preset stream processing framework, and determining whether a time window matching the generation time of the data to be processed is in a cleared state;

[0048] If it is in the clearing state, the data to be processed is determined to be disordered data, the time window is re-formed based on the preset stream processing framework, and the historical clearing data corresponding to the time window is read, and the historical clearing data is restored to the time window;

[0049] The data to be processed is allocated to the time window based on the preset stream processing framework, and is aggregated with the historical cleared data restored to the time window to complete the processing of the disordered data.

[0050] Furthermore, the step of allocating the to-be-processed data to the time window based on the preset stream processing framework and aggregating the historical cleared data restored to the time window includes:

[0051] Allocating the to-be-processed data to the time window based on the preset stream processing framework, and adding a delay duration to the time window;

[0052] Determine whether the target data to be processed is read within the delay time, wherein the generation time of the target data to be processed matches the time window;

[0053] If the target data to be processed is read, the target data to be processed is allocated to the time window, and after the delay period ends, the data in the time window are aggregated.

[0054] Further, after the step of determining whether the time window matching the generation time of the data to be processed is in a cleared state, the processor 1001 may call the out-of-order data processing program stored in the memory 1005, and perform the following operations:

[0055] If the time window matching the generation time of the data to be processed is in an uncleared state, the data to be processed is allocated to the time window, and it is determined whether the generation time of the data to be processed reaches the watermark of the time window;

[0056] If the water level of the time window is reached, the data within the time window are aggregated and processed to generate a processing result which is stored in a preset database.

[0057] Further, after the step of generating and storing the processing result in a preset database, the processor 1001 may call the out-of-order data processing program stored in the memory 1005 and perform the following operations:

[0058] Transfer all data within the time window to the preset database cache, and after the cache is successful, clear all data within the time window;

[0059] The watermark of the time window is adjusted to read new data to be processed in the message queue for aggregation processing.

[0060] Further, after the step of transmitting all data in the time window to the preset database cache, the processor 1001 may call the out-of-order data processing program stored in the memory 1005, and perform the following operations:

[0061] When it is detected that the cleaning time of the preset database has arrived, the data to be deleted in each data cached in the preset database is searched, and a deletion operation is performed on the data to be deleted.

[0062] Furthermore, the step of determining whether the time window matching the generation time of the data to be processed is in a cleared state includes:

[0063] Reading a window time interval of a current time window in an activated state, and determining whether a generation time of the data to be processed is within the window time interval;

[0064] If it is within the window time interval, it is determined that the time window matching the generation time of the data to be processed is in an uncleared state;

[0065] If it is not within the window time interval, it is determined that the time window matching the generation time of the data to be processed is in a cleared state.

[0066] Furthermore, the step of reading the data to be processed from the message queue based on the preset stream processing framework includes:

[0067] Read a single piece of data from a message queue based on a preset stream processing framework, and call a preset function to extract data corresponding to a preset field from the single piece of data;

[0068] The data corresponding to the preset field is determined as the data to be processed.

[0069] The specific implementation of the out-of-order data processing device of the present invention is basically the same as the various embodiments of the out-of-order data processing method described below, and will not be repeated here.

[0070] The invention also provides a method for processing disordered data.

[0071] Reference Figure 2 , Figure 2 Schematic diagram of the flow of the first embodiment of the out-of-order data processing method of the present invention.

[0072] The embodiment of the present invention provides an embodiment of a method for processing out-of-order data. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here. Specifically, the method for processing out-of-order data in this embodiment includes:

[0073] Step S10, reading the data to be processed from the message queue based on the preset stream processing framework, and determining whether the time window matching the generation time of the data to be processed is in a cleared state;

[0074] In this embodiment, the disordered data processing method is applied to a system including a preset stream processing framework, and the preset stream processing framework is preferably a flink framework. Flink is a stream processing framework that combines streams and batches, and is widely used in data statistics in real-time computing. In addition to the flink framework, this system is also provided with a message queue. The data generated by the system is first stored in the message queue, and the flink framework reads and processes it from the message queue. Among them, the message queue can be any one of ActiveMQ, RabbitMQ, RocketMQ, and Kafka. Considering the real-time collection and calculation of big data, a message queue in the form of Kafka is preferred. In addition, the data collected and calculated in real time are mainly business data or log data.

[0075] Furthermore, each piece of data generated by the system is stored in a message queue in a specific data format, and each piece of data contains multiple data items, representing the items targeted by each piece of data, the time of the items, etc. In this embodiment, the multiple data items of each piece of data exist in the form of key_value pairs, representing the parameter values ​​of the items on various factors, such as the items of clicking on a certain advertising business, the number of clicks, the time of clicks, the price of the advertising product and their respective parameter values ​​can form multiple key_value pairs, representing the numerical values ​​of factors such as the number of clicks, the time of clicks and the price of the advertising product. If the number of clicks is 1, the time of clicks is 2020-05-24 12:57.00, and the price of the advertising product is 9.0, the data formed is: {"price": 9.0, "click": 1, "timestamp": 2020-05-24 12:57.00}. The system stores each piece of data formed in this way in the message queue according to their respective formation time, and the flink framework reads and processes them one by one.

[0076] It is understandable that although each piece of data stored in the message queue contains multiple data items in the same data format, during the processing of the flink framework, the required data items are specific, that is, processing is performed on specific data items. For example, in the above-mentioned advertising business, the processed data item may be the average price calculated after statistics of each price, or it may be the total number of clicks obtained by counting the number of clicks. In response to different processing requirements, the key in the required data item is used as the preset field. For example, if the processing requirement for the above data is the number of clicks, "click" is used as the preset field. After reading each piece of data from the message queue, the flink framework continues to extract the data corresponding to the preset field as the data to be processed for processing. Specifically, the steps of reading the data to be processed from the message queue based on the flink framework include:

[0077] Step S11, reading a single piece of data from a message queue based on a preset stream processing framework, and calling a preset function to extract data corresponding to a preset field from the single piece of data;

[0078] Step S12: determining the data corresponding to the preset field as the data to be processed.

[0079] Furthermore, a preset function, such as a map function, is pre-set to extract data corresponding to a preset field. After each piece of data is read from the message queue one by one through the flink framework as a single piece of data, the preset function is called to extract the single piece of data read each time, and the data corresponding to the preset field is extracted therefrom. In addition, the extracted data is determined as data to be processed so as to be processed through flink.

[0080] Furthermore, the way to process data in the flink framework is to set a time window with a certain time interval, and the time window changes with the current time. If the time interval is 5 minutes, starting at 12 o'clock, the time window is formed 5 minutes before 12 o'clock, that is, between 11:55 and 12:00. If the current time exceeds 12:00, the time window changes to between 12:00 and 12:05. When the generation time of the data in the message queue is within the current time window formed by the time interval, the data is assigned to the time window. After all the data with the generation time within the current time window is assigned to the time window, all the data in the time window will be processed, and the data in it will be aggregated together for statistics or average calculation. After that, the time interval of the time window is changed, and other data in the message queue is read to continue processing.

[0081] However, for disordered data, it is not arranged according to the generation time in the message queue, so that when the flink framework reads each piece of data from the message queue in order, the generation time of the obtained data is no longer within the time interval of the currently processed time window, and the previously processed data is also inaccurate due to the lack of this piece of data. Therefore, after extracting the data to be processed according to the preset function, flink in this embodiment judges whether the time window matching the generation time of the data to be processed is in a clear state, wherein the time window where the generation time of the data to be processed is located is the matching time window. And after each time window completes the processing of the data to be processed therein, the data to be processed will be cleared, and the time interval will be updated to process the data to be processed in the updated time interval. The state of the time window after clearing the data to be processed is taken as the clear state, and whether the matching time window is in a clear state can be judged by determining whether the generation time of the data to be processed is within the time interval of the currently working time window.

[0082] Step S20: if the data is in the clearing state, the data to be processed is determined to be out-of-order data, the time window is re-formed based on the preset stream processing framework, and the historical clearing data corresponding to the time window is read, and the historical clearing data is restored to the time window;

[0083] Furthermore, if the time window that matches the generation time of the data to be processed is determined to be in a cleared state, it means that the data to be processed is out of order data, and the flink framework has completed processing of each item of data to be processed in the matching time window. And the processed items of data to be processed are transferred to a specific storage unit for storage. The specific storage unit can be a database connected to the system, such as a redis database, or the system's memory. For the sake of storage stability, an external redis database is preferably used.

[0084] Furthermore, after determining that the data to be processed is out-of-order data, in order to ensure the accuracy of the processing results, the historical clearing data corresponding to the matching time window is read through the flink framework. The historical clearing data is the processed data to be processed. The historical clearing data corresponding to the time window can be obtained by reading the data to be processed in the time window from the external redis database. The read historical clearing data re-enters the system memory and re-forms the time window, and then the historical clearing data in the system memory is restored to the time window, so that flink can process it again to ensure the accuracy of the processing results.

[0085] Step S30: allocating the data to be processed to the time window based on the preset stream processing framework, and aggregating the data with the historical cleared data restored to the time window to complete the processing of the disordered data.

[0086] Furthermore, the Flink framework allocates the data to be processed read from the message queue to the matching time window, aggregates it with the historical cleared data restored to the time window, completes the processing of the disordered data, and ensures the accuracy of the data processing in the time window. Specifically, the steps of allocating the data to be processed to the time window based on the Flink framework and aggregating it with the historical cleared data restored to the time window include:

[0087] Step S31, allocating the to-be-processed data to the time window based on the preset stream processing framework, and adding a delay duration to the time window;

[0088] Step S32, determining whether the target data to be processed is read within the delay time, wherein the generation time of the target data to be processed matches the time window;

[0089] Step S33: if the target data to be processed is read, the target data to be processed is allocated to the time window, and after the delay period ends, each data in the time window is aggregated.

[0090] Understandably, for the data within the time window, there may be more than one out-of-order data. In order to avoid frequently reforming the time window and restoring the historical cleared data, a delay mechanism is set in the process of processing the out-of-order data. Specifically, the delay duration is pre-set, such as a delay of 10 minutes or 20 minutes. After restoring the historical cleared data to the time window and allocating the mutually beneficial data to be tested to the time window, the delay duration is added to the time window so that the time window waits for the delay duration before processing the data therein. During the waiting delay duration, it is determined whether data whose generation time matches the time window is read again. The generation time of the data is within the reformed time window and belongs to other out-of-order data, which is used as the target data to be processed.

[0091] Furthermore, if the generation time of the read data is within the time interval of the time window, it is determined that the target data to be processed is read, and the target data to be processed is assigned to the time window. After the delay period ends, the target data to be processed and the previously processed data, as well as the historical cleared data and other data assigned to the time window are aggregated together as all the data in the time window. If the generation time of the read data is not within the time interval of the time window during the delay period, that is, the target data to be processed is not read, then after the delay period ends, the previously processed data and the historical cleared data assigned to the time window are aggregated together as all the data in the time window. By setting the delay period, it is ensured that all out-of-order data is added to the time window as much as possible to ensure the accuracy of the processing results.

[0092] It should be noted that the order of aggregation and waiting for the delay time can be interchanged, that is, after the data to be processed is assigned to the time window, the data to be processed and the historical cleared data in the time window are aggregated, and then the delay time is waited again to determine whether the data whose generation time matches the time window is read again within the delay time, that is, the target data to be processed. If it is read, all the data contained in the time window are re-aggregated until the delay time ends.

[0093] In a specific embodiment, the process of processing out-of-order data based on the flink framework is as follows: Figure 4 As shown, in Figure 4 In the , the data read by the flink framework from the kafka message queue is used as the data received by the flink framework, and the data corresponding to the preset field is extracted from the data through the preset function to realize data preprocessing. After that, a window is allocated to the data. During the allocation process, it is determined whether there is window data in the system memory corresponding to the flink framework, that is, whether the time window matching the data generation time is in a non-cleared state. If it is determined to be in a non-cleared state, that is, there is window data in the memory, the received data is allocated to the time window for normal aggregation. On the contrary, if it is determined to be in a cleared state and there is no window data in the memory, the window data is read from redis, that is, the historical cleared data is read for recovery, and then the recovered data is aggregated normally. After the window exceeds the watermark and the aggregation is completed, the window data, that is, the aggregated data, is written to redis for caching, and the data in the window that has completed the aggregation state is cleared to facilitate the subsequent aggregation of the watermark.

[0094] The out-of-order data processing method of the present invention first reads the data to be processed from the message queue based on the preset stream processing framework, and determines whether the time window matching the generation time of the data to be processed is in a clearing state; if it is determined to be in a clearing state, it means that the data to be processed is out-of-order data, and the time window is re-formed based on the preset stream processing framework, and the historical clearing data corresponding to the time window is read, and the historical clearing data is restored to the time window; then the data to be processed is assigned to the time window based on the preset stream processing framework, and the data is aggregated together with the historical clearing data restored to the time window to complete the processing of the out-of-order data. After the present invention determines that the data to be processed is out-of-order data by the clearing state of the time window, the time window is re-formed, and the historical clearing data cleared from the time window after processing is read and restored to the time window again, and then the data to be processed is assigned to the time window, and the data is aggregated with the restored historical clearing data, so as to avoid the inaccuracy of data statistics caused by the discarding of out-of-order data, and ensure the comprehensive and accurate statistics of data in the real-time calculation process.

[0095] Furthermore, based on the first embodiment of the out-of-order data processing method of the present invention, a second embodiment of the out-of-order data processing method of the present invention is proposed.

[0096] The difference between the second embodiment of the out-of-order data processing method and the first embodiment of the out-of-order data processing method is that after the step of determining whether the time window matching the generation time of the data to be processed is in a cleared state, the method further includes:

[0097] Step S40: if the time window matching the generation time of the data to be processed is in an uncleared state, the data to be processed is allocated to the time window, and it is determined whether the generation time of the data to be processed reaches the watermark of the time window;

[0098] Step S50: If the water level of the time window is reached, aggregate the data within the time window, generate a processing result and store it in a preset database.

[0099] In this embodiment, when the time window matching the generation time of the data to be processed is in a cleared state, the data to be processed is aggregated. Specifically, if it is determined that the generation time of the data to be processed is within the time interval of the current working time window, it is determined that the matching time window is in an uncleared state, and the current working time window is the matching time window. At this time, the data to be processed is directly assigned to the current working time window, and it is determined whether the generation time of the data to be processed reaches the watermark of the time window. The watermark of the time window represents the upper boundary value of the time interval corresponding to the time window. For example, if the time window is from 11:55 to 12:00, the watermark is 12:00. The generation time of the data to be processed is compared with the upper boundary value of the time interval, and it is determined whether the watermark of the time window is reached by judging whether the generation time reaches the upper boundary value. If the generation time reaches the watermark, it means that the data generated before the watermark in the message queue are all assigned to the time window, so that all the data in the time window are aggregated, and the average value or the total number of statistics is generated according to the processing requirements, and the generated average value or the total number of statistics is stored as the processing result in the preset database. If the generation time does not reach the watermark, continue to read the data to be processed from the message queue for judgment until the generation time of a certain data to be processed reaches the watermark, aggregate all the data in the time window, process the generated processing results and store them in the preset database. The preset database is a database pre-set to connect to the system, such as a redis database, so that the processing structure can be read from the preset database for further processing or analysis and display.

[0100] Furthermore, after the step of generating and storing the processing result in a preset database, the method further includes:

[0101] Step S60, transferring all data in the time window to the preset database cache, and after the cache is successfully completed, clearing all data in the time window;

[0102] In order to avoid the existence of disordered data in the processed time window, which may lead to inaccurate processing results, after all the data in the time window is aggregated and processed, the processing results are generated and stored in the preset database, and all the data in the time window is also transferred to the preset database cache, so that after reading the disordered data matching the time window, the cached data is obtained and restored to the time window for reprocessing to ensure the accuracy of the processing results. In addition, during the caching process, if a cache success flag is received from the preset database, indicating that the cache is successful, all the data in the time window is cleared to update the time interval of the time window, and other data in the message queue is read for processing.

[0103] Furthermore, after the step of transmitting all data in the time window to the preset database cache, the method further includes:

[0104] Step S61, when it is detected that the cleaning time of the preset database has arrived, searching for data to be deleted in various data cached in the preset database, and performing a deletion operation on the data to be deleted.

[0105] Understandably, the storage capacity of the preset database is limited. In order to effectively cache the data in each time window, a timed cleanup mechanism is set for each cached data. For example, the data stored in each time window is cleaned up every 2 days, and the set time is used as the cleanup time of the preset database. When the cleanup time is detected, the data in the preset database whose storage time reaches the scheduled cleanup time is searched, and such data is deleted as data to be deleted, so as to release the memory of the preset database and facilitate the storage of subsequent data.

[0106] Step S70, adjusting the watermark of the time window to read new data to be processed in the message queue for aggregation processing.

[0107] Furthermore, after the cache is successful and all data in the time window is cleared, the watermark of the time window is adjusted according to the time interval; the upper boundary value of the time interval corresponding to the time window is increased by the value of the time interval, and in order to ensure that the time interval remains unchanged, the lower boundary value of the time interval is also increased by the value of the time interval, thereby forming a new time window. For example, if the time window is from 16:00 to 16:10 and the time interval is 10 minutes, the adjusted time window is from 16:10 to 16:20. The new time window formed by adjusting the time window watermark is used to read the various pieces of to-be-processed data corresponding to the new time window from the message queue for aggregation processing.

[0108] After the processing of the data to be processed whose generation time matches the time window is completed, this embodiment transfers all the data in the time window to the preset database cache. If there is disordered data in the time window, all the data in the time window can be restored from the preset database for reprocessing to ensure the accuracy of the processing results. A timed cleanup mechanism is set for the preset database to ensure the effective caching of data.

[0109] Furthermore, based on the first or second embodiment of the out-of-order data processing method of the present invention, a third embodiment of the out-of-order data processing method of the present invention is proposed.

[0110] The difference between the third embodiment of the out-of-order data processing method and the first or second embodiment of the out-of-order data processing method is that the step of determining whether the time window matching the generation time of the data to be processed is in a cleared state includes:

[0111] Step S13, reading the window time interval of the current time window in an activated state, and determining whether the generation time of the data to be processed is within the window time interval;

[0112] Step S14, if it is within the window time interval, determining that the time window matching the generation time of the data to be processed is in an uncleared state;

[0113] Step S15: If it is not within the window time interval, it is determined that the time window matching the generation time of the data to be processed is in a cleared state.

[0114] This embodiment determines whether the matching time window is in a cleared state by whether the generation time of the data to be processed is within the time interval of the time window of the current work. Specifically, the window time interval of the current time window currently in the active state in the flink framework is read, and the generation time of the data to be processed is compared with the window time interval to determine whether the generation time of the data to be processed is within the window time interval. If it is within the window time interval, it means that the time window of the current work is the time window that matches the generation time of the data to be processed, so that it is determined that the time window that matches the generation time of the data to be processed is in an uncleared state. The current time window is the time window for aggregate processing of the data to be processed, and the data to be processed can be directly allocated to the current time window and aggregated together with other data allocated to the current time window.

[0115] Furthermore, if it is determined through comparison that the generation time of the data to be processed is not within the window time interval, it means that the data in the time window matching the generation time of the data to be processed has been cleared and is in a cleared state. It is necessary to reform the time window and read the cleared data from the preset database and restore it to the time window for re-aggregation processing.

[0116] It should be noted that in the process of determining whether the time window matching the generation time of the data to be processed is in a cleared state, there may be multiple time windows for the current operation, one of which is a time window for normal processing of data in the message queue based on time interval updates, and the others are time windows that are newly formed because the time window matching the generation time of the data to be processed is in a cleared state. In the case of multiple time windows, if the generation time of the data to be processed is within the time interval of any time window, it is determined that the time window matching the generation time of the data to be processed is in an uncleared state, and the data to be processed can be aggregated through the time window where the generation time is located to avoid repeated recovery processing.

[0117] This embodiment determines whether the matching time window is in the clear state by whether the generation time of the data to be processed is within the time interval of the current working time window. The judgment method is simple and fast, and can quickly determine whether the matching time window is in the clear state.

[0118] The invention also provides a disordered data processing device.

[0119] Reference Figure 3 , Figure 3 It is a functional module diagram of the first embodiment of the out-of-order data processing device of the present invention.

[0120] The out-of-order data processing device comprises:

[0121] A judgment module 10 is used to read the data to be processed from the message queue based on a preset stream processing framework, and to judge whether a time window matching the generation time of the data to be processed is in a cleared state;

[0122] A recovery module 20, configured to determine that the data to be processed is out-of-order data if it is in a clearing state, reform the time window based on the preset stream processing framework, read historical clearing data corresponding to the time window, and restore the historical clearing data to the time window;

[0123] The aggregation module 30 is used to allocate the to-be-processed data to the time window based on the preset stream processing framework, and aggregate the to-be-processed data with the historical cleared data restored to the time window to complete the processing of the disordered data.

[0124] Furthermore, the aggregation module 30 further includes:

[0125] An allocating unit, configured to allocate the to-be-processed data to the time window based on the preset stream processing framework, and add a delay duration to the time window;

[0126] A judging unit, configured to judge whether the target data to be processed is read within the delay time, wherein the generation time of the target data to be processed matches the time window;

[0127] The aggregation unit is used to allocate the target data to be processed to the time window if the target data to be processed is read, and aggregate the data in the time window after the delay period ends.

[0128] Furthermore, the out-of-order data processing device further comprises:

[0129] an allocation module, configured to allocate the data to be processed to the time window if the time window matching the generation time of the data to be processed is in an uncleared state, and determine whether the generation time of the data to be processed reaches the watermark of the time window;

[0130] The generation module is used to aggregate the data within the time window if the water level of the time window is reached, and generate the processing results and store them in a preset database.

[0131] Furthermore, the out-of-order data processing device further comprises:

[0132] A clearing module, used to transfer all data in the time window to the preset database cache, and clear all data in the time window after the cache is successful;

[0133] The adjustment module is used to adjust the watermark of the time window to read new data to be processed in the message queue for aggregation processing.

[0134] Furthermore, the clearing module is also used for:

[0135] When it is detected that the cleaning time of the preset database has arrived, the data to be deleted in each data cached in the preset database is searched, and a deletion operation is performed on the data to be deleted.

[0136] Furthermore, the judging module 10 further includes:

[0137] A reading unit, used for reading a window time interval of a current time window in an activated state, and determining whether a generation time of the data to be processed is within the window time interval;

[0138] A first determination unit, configured to determine that a time window matching the generation time of the data to be processed is in an uncleared state if it is within the window time interval;

[0139] The second determination unit is configured to determine that the time window matching the generation time of the data to be processed is in a clearing state if the time window is not within the window time interval.

[0140] Furthermore, the judging module 10 further includes:

[0141] A calling unit, configured to read a single piece of data from a message queue based on a preset stream processing framework, and to call a preset function to extract data corresponding to a preset field from the single piece of data;

[0142] A determination unit is used to determine the data corresponding to the preset field as the data to be processed.

[0143] The specific implementation of the out-of-order data processing device of the present invention is basically the same as the above-mentioned embodiments of the out-of-order data processing method, and will not be repeated here.

[0144] In addition, an embodiment of the present invention further provides a readable storage medium.

[0145] The readable storage medium stores an out-of-order data processing program, and when the out-of-order data processing program is executed by the processor, the steps of the out-of-order data processing method described above are implemented.

[0146] The readable storage medium of the present invention may be a computer-readable storage medium, and its specific implementation is substantially the same as the above-mentioned embodiments of the out-of-order data processing method, and will not be described in detail here.

[0147] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims. All equivalent structures or equivalent process changes made using the contents of the specification and drawings of the present invention, or directly or indirectly used in other related technical fields, are protected by the present invention.

Claims

1. A method for processing out-of-order data, characterized in that: The out-of-order data processing method comprises the following steps: Reading the data to be processed from the message queue based on a preset stream processing framework, and determining whether a time window matching the generation time of the data to be processed is in a cleared state; The step of determining whether the time window matching the generation time of the data to be processed is in a cleared state comprises: Reading a window time interval of a current time window in an activated state, and determining whether a generation time of the data to be processed is within the window time interval; If it is within the window time interval, it is determined that the time window matching the generation time of the data to be processed is in an uncleared state; If it is not within the window time interval, it is determined that the time window matching the generation time of the data to be processed is in a clearing state; If it is in the clearing state, the data to be processed is determined to be disordered data, the time window is re-formed based on the preset stream processing framework, and the historical clearing data corresponding to the time window is read, and the historical clearing data is restored to the time window; Allocating the data to be processed to the time window based on the preset stream processing framework, and aggregating the data with the historical cleared data restored to the time window, so as to complete the processing of the disordered data; The step of allocating the to-be-processed data to the time window based on the preset stream processing framework and aggregating the historical cleared data restored to the time window comprises: Allocating the to-be-processed data to the time window based on the preset stream processing framework, and adding a delay duration to the time window; Determine whether the target data to be processed is read within the delay time, wherein the generation time of the target data to be processed matches the time window; If the target data to be processed is read, the target data to be processed is allocated to the time window, and after the delay period ends, the data in the time window are aggregated.

2. The out-of-order data processing method according to claim 1, characterized in that: After the step of determining whether the time window matching the generation time of the data to be processed is in a cleared state, the method further includes: If the time window matching the generation time of the data to be processed is in an uncleared state, the data to be processed is allocated to the time window, and it is determined whether the generation time of the data to be processed reaches the watermark of the time window; If the water level of the time window is reached, the data within the time window are aggregated and processed to generate a processing result which is stored in a preset database.

3. The out-of-order data processing method according to claim 2, characterized in that: After the step of generating the processing result and storing it in a preset database, the method further includes: Transfer all data within the time window to the preset database cache, and after the cache is successful, clear all data within the time window; The watermark of the time window is adjusted to read new data to be processed in the message queue for aggregation processing.

4. The out-of-order data processing method according to claim 3, characterized in that: After the step of transmitting all data within the time window to the preset database cache, the method further includes: When it is detected that the cleaning time of the preset database has arrived, the data to be deleted in each data cached in the preset database is searched, and a deletion operation is performed on the data to be deleted.

5. The out-of-order data processing method according to any one of claims 1 to 4, characterized in that: The step of reading the data to be processed from the message queue based on the preset stream processing framework includes: Read a single piece of data from a message queue based on a preset stream processing framework, and call a preset function to extract data corresponding to a preset field from the single piece of data; The data corresponding to the preset field is determined as the data to be processed.

6. A disordered data processing device, characterized in that: The out-of-order data processing device comprises: A judgment module, used to read the data to be processed from the message queue based on a preset stream processing framework, and judge whether a time window matching the generation time of the data to be processed is in a cleared state; The judging module further includes: A reading unit, used for reading a window time interval of a current time window in an activated state, and determining whether a generation time of the data to be processed is within the window time interval; A first determination unit, configured to determine that a time window matching the generation time of the data to be processed is in an uncleared state if it is within the window time interval; A second determination unit, configured to determine that the time window matching the generation time of the data to be processed is in a clearing state if the time window is not within the window time interval; A recovery module, configured to determine that the data to be processed is out-of-order data if it is in a clearing state, reform the time window based on the preset stream processing framework, read historical clearing data corresponding to the time window, and restore the historical clearing data to the time window; An aggregation module, used to allocate the to-be-processed data to the time window based on the preset stream processing framework, and aggregate the to-be-processed data with the historical cleared data restored to the time window, so as to complete the processing of the disordered data; The aggregation module also includes: An allocating unit, configured to allocate the to-be-processed data to the time window based on the preset stream processing framework, and add a delay duration to the time window; A judging unit, configured to judge whether the target data to be processed is read within the delay time, wherein the generation time of the target data to be processed matches the time window; The aggregation unit is used to allocate the target data to be processed to the time window if the target data to be processed is read, and aggregate the data in the time window after the delay period ends.

7. A disordered data processing device, characterized in that: The out-of-order data processing device includes a memory, a processor, and an out-of-order data processing program stored in the memory and executable on the processor. When the out-of-order data processing program is executed by the processor, the steps of the out-of-order data processing method according to any one of claims 1 to 5 are implemented.

8. A readable storage medium, characterized in that: The readable storage medium stores an out-of-order data processing program, and when the out-of-order data processing program is executed by a processor, the steps of the out-of-order data processing method according to any one of claims 1 to 5 are implemented.

Images