Flink-based Real-time Data Processing Method, System, Device and Storage Medium

Abstract

The present invention relates to a real-time data processing method, system, device and storage medium based on Flink. The key points of the technical solution are as follows: Determine whether there is axis variety data in all current batch axis variety data that is the same as the running axis variety data. If so, determine whether to update the current batch running data corresponding to the axis variety data stored in the Flink State according to the real-time running data; otherwise, determine whether there is axis variety data in all historical axis variety data stored in HBase that is the same as the running axis variety data. If so, write the axis variety data and the historical running data corresponding to the axis variety data stored in HBase into the Flink State. Otherwise, write the axis variety data to be added and the real-time running data into HBase for storage according to all the cycle reported axis variety data stored in HBase; The present application has the effects of facilitating data recovery, reducing the interaction between HBase and Flink State, and improving the operation efficiency.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and more specifically, it relates to a real-time data processing method, system, device and storage medium based on Flink. Background Art

[0002] In the textile industry, multiple textile machines are usually used for textile work. In order to monitor the textile machines, collectors are usually installed on the textile machines to collect the number of running circles of the running shafts on the textile machines.

[0003] During the process of the collector collecting the running data of the textile machine, the collector will upload the running data in real time. When calculating the production output, it usually takes a relatively long time, such as one day or one week, to calculate the daily output or weekly output. In the actual production process, it is inevitable that there will be problems such as collector failures, replacement of running shafts, and no data upload during holidays, resulting in data loss and difficulty in recovery. Summary of the Invention

[0004] Aiming at the deficiencies of the existing technology, the purpose of the present invention is to provide a real-time data processing method, system, device and storage medium based on Flink, which has the functional advantages of facilitating data recovery, reducing the interaction between HBase and Flink State, and improving the operation efficiency.

[0005] The above technical purpose of the present invention is achieved through the following technical solutions:

[0006] A real-time data processing method based on Flink includes:

[0007] S1. Obtain the real-time running data of the real-time running shaft, the running shaft variety data of the real-time running shaft, all the current batch shaft variety data stored in FlinkState, and all the current batch running data corresponding one by one to all the current batch shaft variety data;

[0008] S2. Judge whether there is shaft variety data in all the current batch shaft variety data that is the same as the running shaft variety data. If so, execute step S3; if not, execute step S4;

[0009] S3. Denote this shaft variety data as the first target shaft variety data, and judge whether to update the current batch running data corresponding to the first target shaft variety data according to the real-time running data;

[0010] S4. Obtain all the historical shaft variety data stored in HBase, and judge whether there is shaft variety data in all the historical shaft variety data that is the same as the running shaft variety data. If so, execute step S5; if not, execute step S6;

[0011] S5. Record the axis variety data as the second target axis variety data, and write the second target axis variety data and its corresponding historical operation data stored in HBase into the Flink State to obtain the current batch axis variety data identical to the second target axis variety data and its corresponding current batch operation data. Determine whether to update the current batch operation data corresponding to the second target axis variety data according to the real-time operation data;

[0012] S6. Record the running axis variety data as the axis variety data to be added, obtain all the cycle reported axis variety data stored in the HBase, and write the axis variety data to be added and the real-time operation data into the HBase for storage according to all the cycle reported axis variety data.

[0013] Optionally, the real-time running axis data includes: acquisition time and number of running circles; the current batch operation data includes: first start time, first end time, first start circle number and first end circle number; the historical operation data includes: second start time, second end time, second start circle number and second end circle number.

[0014] Optionally, determining whether to update the current batch operation data corresponding to the first target axis variety data according to the real-time operation data includes:

[0015] Determine whether the acquisition time is less than the first start time corresponding to the first target axis variety data. If so, use the acquisition time as the first start time corresponding to the first target axis variety data, and use the number of running circles as the first start circle number corresponding to the first target axis variety data to complete the update; if not, determine whether the acquisition time is greater than the first end time corresponding to the first target axis variety data. If so, use the acquisition time as the first end time corresponding to the first target axis variety data, and use the number of running circles as the first end circle number corresponding to the first target axis variety data to complete the update. If not, do not update the current batch operation data corresponding to the first target axis variety data stored in the Flink State.

[0016] Optionally, determining whether to update the current batch operation data corresponding to the second target axis variety data according to the real-time operation data includes:

[0017] Determine whether the acquisition time is less than the first start time corresponding to the second target axis variety data. If so, use the acquisition time as the first start time corresponding to the second target axis variety data, and use the number of running cycles as the first start cycle number corresponding to the second target axis variety data to complete the update. If not, determine whether the acquisition time is greater than the first end time corresponding to the second target axis variety data. If so, use the acquisition time as the first end time corresponding to the second target axis variety data, and use the number of running cycles as the first end cycle number corresponding to the second target axis variety data to complete the update. If not, do not update the current batch of running data corresponding to the second target axis variety data stored in Flink State.

[0018] Optionally, writing the to-be-added axis variety data and real-time running data into HBase for storage according to all the periodically reported axis variety data includes:

[0019] Determine whether there is a periodically reported axis variety data in all the periodically reported axis variety data that is the same as the to-be-added axis variety data. If so, use the acquisition time as the third end time corresponding to this periodically reported axis variety data, and use the number of running cycles as the third end cycle number corresponding to this periodically reported axis variety data. If not, use the to-be-added axis variety data as the periodically reported axis variety data, use the acquisition time as the third start time corresponding to this periodically reported axis variety data, and use the number of running cycles as the third start cycle number corresponding to this periodically reported axis variety data.

[0020] Optionally, it further includes:

[0021] Determine whether the current batch of running data corresponding to each current batch of axis variety data stored in Flink State has been updated within a preset period. If not, write the current batch of axis variety data and the corresponding current batch of running data into HBase, and delete the current batch of axis variety data and the corresponding current batch of running data in Flink State.

[0022] A real-time data processing system based on Flink includes:

[0023] A data acquisition module, configured to acquire the real-time running data of the real-time running axis, the running axis variety data of the real-time running axis, all the current batch of axis variety data stored in Flink State, and all the current batch of running data corresponding one-to-one to all the current batch of axis variety data;

[0024] A data judgment module, configured to determine whether there is an axis variety data in all the current batch of axis variety data that is the same as the running axis variety data. If so, start the judgment update module; if not, start the acquisition judgment module;

[0025] A judgment update module, configured to record the axis variety data as the first target axis variety data, and judge whether to update the current batch of operation data corresponding to the first target axis variety data according to the real-time operation data;

[0026] An acquisition judgment module, configured to acquire all historical axis variety data stored in HBase, and judge whether there is axis variety data identical to the running axis variety data among all the historical axis variety data. If so, start the migration update module; if not, start the acquisition migration module;

[0027] A migration update module, configured to record the axis variety data as the second target axis variety data, write the second target axis variety data and the corresponding historical operation data stored in HBase into Flink State, obtain the current batch of axis variety data identical to the second target axis variety data and the corresponding current batch of operation data, and judge whether to update the current batch of operation data corresponding to the second target axis variety data according to the real-time operation data;

[0028] An acquisition migration module, configured to record the running axis variety data as the axis variety data to be added, acquire all the cycle reported axis variety data stored in HBase, and write the axis variety data to be added and the real-time operation data into HBase according to all the cycle reported axis variety data.

[0029] Optionally, the real-time running axis data includes: acquisition time and number of running circles; the current batch of operation data includes: a first start time, a first end time, a first start circle number, and a first end circle number; the judgment update module includes:

[0030] A first judgment unit, configured to judge whether the acquisition time is less than the first start time corresponding to the first target axis variety data. If so, start the first update unit; if not, start the second judgment unit;

[0031] A first update unit, configured to use the acquisition time as the first start time corresponding to the first target axis variety data, and use the number of running circles as the first start circle number corresponding to the first target axis variety data to complete the update;

[0032] A second judgment unit, configured to judge whether the acquisition time is greater than the first end time corresponding to the first target axis variety data. If so, start the second update unit; if not, do not update the current batch of operation data corresponding to the first target axis variety data stored in Flink State;

[0033] The second update unit is used to take the acquisition time as the first end time corresponding to the first target axis variety data and the number of running cycles as the first end cycle number corresponding to the first target axis variety data to complete the update.

[0034] A computer device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps of the above method are implemented.

[0035] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0036] In summary, the present invention has the following beneficial effects: Determine whether the running axis variety data is historical axis variety data or axis variety data to be added. If the running axis variety data is historical axis variety data, it means that the running axis corresponding to the running axis variety data is reused, and then the running axis variety data and the corresponding historical running data are migrated back to the Flink State to become the second target axis variety data and the current batch running data. The current batch running data corresponding to the second target axis variety data in the Flink State is processed according to the real-time running data, which is convenient for the recovery of the data of the historical axis when it is reloaded. If the running axis variety data is axis variety data to be added, determine whether the axis variety data to be added is axis variety data reported periodically, so as to write the real-time running data and the axis variety data to be added into HBase according to the axis variety data to be added, reducing the interaction between HBase and the Flink State to improve the running efficiency. Description of the Drawings

[0037] Figure 1 is a schematic flowchart of the method provided by the present invention;

[0038] Figure 2 is a structural block diagram of a real-time data processing system based on Flink provided by the present invention;

[0039] Figure 3 is an internal structural diagram of the computer device in an embodiment of the present invention. Detailed Embodiments

[0040] To make the objectives, features, and advantages of the present invention more obvious and understandable, the detailed embodiments of the present invention will be described below with reference to the accompanying drawings. Several embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein.

[0041] In the present invention, unless otherwise clearly defined and limited, terms such as "installation", "connection", "linkage", "fixation" shall be understood in a broad sense. For example, it may be a fixed connection, a detachable connection, or an integral connection; it may be a mechanical connection or an electrical connection; it may be a direct connection or an indirect connection through an intermediate medium, and it may be the communication inside two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances. The terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features.

[0042] The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0043] The present invention provides a real-time data processing method based on Flink, as Figure 1 shown, including:

[0044] S1. Obtain the real-time operation data of the real-time operation axis, the operation axis variety data of the real-time operation axis, all the current batch axis variety data stored in FlinkState, and all the current batch operation data corresponding one by one to all the current batch axis variety data; wherein, Flink is an open-source stream processing framework that executes any stream data program in a data parallel and pipeline manner. The pipeline runtime system of Flink can execute batch processing and stream processing programs. In addition, Flink itself also supports the execution of iterative algorithms during runtime. State is a memory-based state mechanism of Flink.

[0045] S2. Determine whether there is axis variety data in all the current batch axis variety data that is the same as the operation axis variety data. If so, execute step S3; if not, execute step S4;

[0046] S3. Denote the axis variety data as the first target axis variety data, and determine whether to update the current batch operation data corresponding to the first target axis variety data according to the real-time operation data;

[0047] S4. Obtain all the historical axis variety data stored in HBase, and determine whether there is axis variety data in all the historical axis variety data that is the same as the operation axis variety data. If so, execute step S5; if not, execute step S6; wherein, HBase is a distributed, column-oriented open-source database and is suitable for storing unstructured data.

[0048] S5. Denote the axis variety data as the second target axis variety data, write the second target axis variety data and its corresponding historical operation data stored in HBase into the Flink State to obtain the current batch axis variety data identical to the second target axis variety data and its corresponding current batch operation data, and determine whether to update the current batch operation data corresponding to the second target axis variety data according to the real-time operation data;

[0049] S6. Denote the running axis variety data as the axis variety data to be added, obtain all the cycle reported axis variety data stored in the HBase, and write the axis variety data to be added and the real-time operation data into the HBase for storage according to all the cycle reported axis variety data.

[0050] In practical applications, in order to monitor a textile factory, collectors are usually installed on textile machines to collect the number of rotations of the running shafts on the textile machines, so as to calculate the production output of the textile machines based on the number of rotations. Both the running shafts and the collectors on the textile machines have numbers. Before starting production, the numbers of the running shafts corresponding to the current textile machines in the textile factory and the corresponding collector numbers are stored in the Flink State to form the current batch of shaft variety data. The collectors upload the collected data, the corresponding collector numbers, and the corresponding running shaft numbers, that is, report the real-time running data and the running shaft variety data to a message middleware such as Kafka. By consuming Kafka, the real-time running data and the running shaft variety data are obtained, and it is judged whether there is shaft variety data in all the current batch of shaft variety data that is the same as the running shaft variety data, that is, it is judged whether the collector number uploaded by the collector is the same as the pre-stored collector number and whether the running shaft number is the same as the pre-stored running shaft number. If both are the same, it means that the collector number and the running shaft number uploaded by the collector are normal and no abnormal situation has occurred, such as replacing the running shaft or replacing the collector, etc., and then directly process the current batch of running data corresponding to the target shaft variety data stored in the Flink State. If both are not the same, it is necessary to judge whether the running shaft variety data is historical shaft variety data or shaft variety data to be added. Historical shaft variety data refers to the shaft variety data that was previously stored in the Flink State but migrated from the Flink State to HBase because it was not used for a long time. Shaft variety data to be added refers to the shaft variety data that was not previously stored in the Flink State, and this shaft variety is a new shaft variety. If the running shaft variety data is historical shaft variety data, it means that the running shaft corresponding to this running shaft variety data is reused, and then this running shaft variety data and the historical running data corresponding to this running shaft variety data are migrated back to the Flink State to become the second target shaft variety data and the current batch of running data, and the current batch of running data corresponding to the second target shaft variety data in the Flink State is processed according to the real-time running data, which is convenient for recovering the data of the historical shaft when it is re-spooled. If the running shaft variety data is shaft variety data to be added, it is judged whether the shaft variety data to be added is periodically reported shaft variety data, so as to write the real-time running data and the shaft variety data to be added into HBase according to the shaft variety data to be added, reducing the interaction between HBase and the Flink State to improve the operation efficiency.

[0051] Further, the real-time running shaft data includes: collection time and number of rotations; the current batch of running data includes: a first start time, a first end time, a first start number of rotations, and a first end number of rotations; the historical running data includes: a second start time, a second end time, a second start number of rotations, and a second end number of rotations.

[0052] In practical applications, after determining that the running axis variety data is the first target axis variety data, the first start time or the first end time can be updated according to the acquisition time, and the first start revolution number or the first end revolution number can be updated according to the running revolution number. When there is no running axis variety data in Flink State but there is running axis variety data stored in HBase, the historical running data corresponding to the running axis variety data is also stored in HBase correspondingly. The historical running data and the running axis variety data are migrated back to Flink State to obtain the current batch of running data and the second target axis variety data.

[0053] Further, determining whether to update the current batch of running data corresponding to the first target axis variety data according to the real-time running data includes:

[0054] Judge whether the acquisition time is less than the first start time corresponding to the first target axis variety data. If so, use the acquisition time as the first start time corresponding to the first target axis variety data, and use the running revolution number as the first start revolution number corresponding to the first target axis variety data to complete the update. If not, judge whether the acquisition time is greater than the first end time corresponding to the first target axis variety data. If so, use the acquisition time as the first end time corresponding to the first target axis variety data, and use the running revolution number as the first end revolution number corresponding to the first target axis variety data to complete the update. If not, do not update the current batch of running data corresponding to the first target axis variety data stored in Flink State.

[0055] In practical applications, compare the acquisition time with the first start time corresponding to the first target axis variety data. When the acquisition time is less than the first start time, the first start time needs to be updated, that is, use the acquisition time as the first start time and the running revolution number as the first start revolution number to ensure the accuracy of the start time and start revolution number of the current batch of running data corresponding to the first target axis variety data. When the acquisition time is not less than the first start time, compare the acquisition time with the first end time corresponding to the first target axis variety data. When the acquisition time is greater than the first end time, the first end time needs to be updated, that is, use the acquisition time as the first end time and the running revolution number as the first end revolution number to ensure the accuracy of the end time and end revolution number of the current batch of running data corresponding to the first target axis variety data, so as to facilitate calculating the accurate production output according to the start revolution number and end revolution number.

[0056] Further, in the case where the running axis variety data is determined as the second target axis variety data, the method for determining whether to update the current batch running data corresponding to the second target axis variety data according to the real-time running data is the same as the method for determining whether to update the current batch running data corresponding to the first target axis variety data according to the real-time running data. The method for determining whether to update the current batch running data corresponding to the second target axis variety data according to the real-time running data includes:

[0057] Determine whether the acquisition time is less than the first start time corresponding to the second target axis variety data. If so, use the acquisition time as the first start time corresponding to the second target axis variety data, and use the running circle number as the first start circle number corresponding to the second target axis variety data to complete the update. If not, determine whether the acquisition time is greater than the first end time corresponding to the second target axis variety data. If so, use the acquisition time as the first end time corresponding to the second target axis variety data, and use the running circle number as the first end circle number corresponding to the second target axis variety data to complete the update. If not, do not update the current batch running data corresponding to the second target axis variety data stored in the Flink State.

[0058] Further, after the update is completed, it further includes: regularly writing the updated current batch running data into HBase for storage.

[0059] In practical applications, after updating the current batch running data in the Flink State according to the actual running data, the updated current batch running data is regularly written into HBase, such as writing the updated current batch running data into HBase every 24 hours, 12 hours, or 8 hours. In the case where historical running data corresponding to the first variety data and / or the second variety data has been stored in HBase, the historical running data in HBase is updated based on the updated current batch running data to facilitate subsequent data recovery to directly access HBase.

[0060] Further, the method for writing the to-be-added axis variety data and the real-time running data into HBase for storage according to all the periodically reported axis variety data includes:

[0061] Determine whether there is periodic reported axis variety data in all the periodic reported axis variety data that is the same as the to-be-added axis variety data. If so, use the acquisition time as the third end time corresponding to this periodic reported axis variety data, and use the running laps as the third end laps corresponding to this periodic reported axis variety data. If not, use the to-be-added axis variety data as the periodic reported axis variety data, use the acquisition time as the third start time corresponding to this periodic reported axis variety data, and use the running laps as the third start laps corresponding to this periodic reported axis variety data.

[0062] In practical applications, the periodic operation data corresponding to the periodic reported variety data is stored in HBase. The periodic operation data includes: the third start time, the third start laps, the third end time, and the third end laps. Compare the to-be-added axis variety data with the periodic reported axis variety data. When there is periodic reported axis variety data in the periodic reported axis variety data that is the same as the to-be-added axis variety data, use the acquisition time as the third end time of the periodic operation data corresponding to this periodic reported axis variety data, and use the running laps as the third end laps of the periodic operation data corresponding to this periodic reported axis variety data to update the periodic operation data. When there is periodic reported axis variety data in the periodic reported axis variety data that is the same as the to-be-added axis variety data, use the to-be-added axis variety data as the new periodic reported axis variety data, and use the real-time operation data as the periodic operation data. Specifically, use the acquisition time as the third start time and the running laps as the third start laps. After using the to-be-added axis variety data as the new periodic reported axis variety data, obtain the real-time operation data corresponding to the periodic reported variety data according to the preset period, use the obtained acquisition time as the third end time, and use the obtained running laps as the third end laps, thus realizing the preservation of the acquisition time and running laps of the running axis that has not reported operation axis data, and also facilitating data recovery in case of collector failure or axis variety replacement.

[0063] Furthermore, it also includes:

[0064] Determine whether there is an update within the preset period for the current batch operation data corresponding to each current batch axis variety data stored in the Flink State. If not, write the current batch axis variety data and its corresponding current batch operation data into HBase, and delete the current batch axis variety data and its corresponding current batch operation data in the Flink State.

[0065] In practical applications, it is necessary to determine whether the current batch of operation data corresponding to the axis variety data of the current batch has been updated within a preset period. In this application, the preset period is 3 days. In other embodiments, the preset period can be adjusted according to the actual vacation situation. If there is no update within the preset period, it is determined that the operating axis corresponding to the unupdated current batch of data is in a non-working state. The current batch of axis variety data and the current batch of operation data corresponding to the operating axis in the non-working state are written into HBase, and the current batch of axis variety data and the current batch of operation data are deleted from the Flink State, reducing the memory occupancy of the Flink State. Moreover, writing the current batch of axis variety data and the current batch of operation data corresponding to the operating axis in the non-working state into HBase also facilitates the recovery of historical operation data from HBase after the operating axis is re-loaded.

[0066] The real-time data processing method based on Flink of the present invention determines whether the operating axis variety data is historical axis variety data or axis variety data to be added. If the operating axis variety data is historical axis variety data, it means that the operating axis corresponding to the operating axis variety data is reused. Then, the operating axis variety data and the corresponding historical operation data are migrated back to the Flink State to become the second target axis variety data and the current batch of operation data. The current batch of operation data corresponding to the second target axis variety data in the Flink State is processed according to the real-time operation data, which is convenient for the recovery of the data of the historical axis re-loaded. If the operating axis variety data is axis variety data to be added, it is determined whether the axis variety data to be added is axis variety data reported periodically, so as to write the real-time operation data and the axis variety data to be added into HBase according to the axis variety data to be added, reducing the interaction between HBase and the Flink State to improve the operation efficiency.

[0067] As Figure 2 shown, the present invention also provides a real-time data processing system based on Flink, including:

[0068] A data acquisition module 10, configured to acquire the real-time operation data of the real-time operating axis, the operating axis variety data of the real-time operating axis, all the current batch axis variety data stored in the Flink State, and all the current batch operation data corresponding one by one to all the current batch axis variety data;

[0069] A data judgment module 20, configured to judge whether there is axis variety data in all the current batch axis variety data that is the same as the operating axis variety data. If so, the judgment update module is started; if not, the acquisition judgment module is started;

[0070] The judgment and update module 30 is configured to record the axis variety data as the first target axis variety data, and determine whether to update the current batch of operation data corresponding to the first target axis variety data according to the real-time operation data;

[0071] The acquisition and judgment module 40 is configured to acquire all historical axis variety data stored in HBase, and determine whether there is axis variety data identical to the running axis variety data among all the historical axis variety data. If so, the migration and update module is started; if not, the acquisition and migration module is started;

[0072] The migration and update module 50 is configured to record the axis variety data as the second target axis variety data, write the second target axis variety data and the historical operation data stored in HBase corresponding thereto into Flink State, to obtain the current batch of axis variety data identical to the second target axis variety data and the current batch of operation data corresponding thereto, and determine whether to update the current batch of operation data corresponding to the second target axis variety data according to the real-time operation data;

[0073] The acquisition and migration module 60 is configured to record the running axis variety data as the axis variety data to be added, acquire all the periodically reported axis variety data stored in HBase, and write the axis variety data to be added and the real-time operation data into HBase for storage according to all the periodically reported axis variety data.

[0074] In one embodiment, the real-time running axis data includes: the acquisition time and the running laps; the current batch of operation data includes: the first start time, the first end time, the first start laps and the first end laps; the judgment and update module includes:

[0075] The first judgment unit is configured to judge whether the acquisition time is less than the first start time corresponding to the first target axis variety data. If so, the first update unit is started; if not, the second judgment unit is started;

[0076] The first update unit is configured to use the acquisition time as the first start time corresponding to the first target axis variety data, and use the running laps as the first start laps corresponding to the first target axis variety data, to complete the update;

[0077] The second judgment unit is configured to judge whether the acquisition time is greater than the first end time corresponding to the first target axis variety data. If so, the second update unit is started; if not, the current batch of operation data corresponding to the first target axis variety data stored in Flink State is not updated;

[0078] A second update unit, configured to use the acquisition time as the first end time corresponding to the first target axis variety data, and use the number of running cycles as the first end cycle number corresponding to the first target axis variety data, to complete the update.

[0079] For the specific limitations of the Flink-based real-time data processing system, reference can be made to the limitations of the Flink-based real-time data processing method in the above text, which will not be elaborated here. Each module of the above Flink-based real-time data processing system can be implemented in whole or in part by software, hardware, and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0080] In one embodiment, a computer device is provided. The computer device can be a server, and its internal structure diagram can be as Figure 3 shown. The computer device includes a processor, a memory, a network interface, and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements a Flink-based real-time data processing method.

[0081] Those skilled in the art can understand that Figure 3 the structure shown in

[0082] is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0083] S1. Obtain the real-time operation data of the real-time operation axis, the operation axis variety data of the real-time operation axis, all the current batch axis variety data stored in the FlinkState, and all the current batch operation data corresponding one-to-one to all the current batch axis variety data;

[0084] S2. Determine whether there is shaft variety data in all the current batch of shaft variety data that is the same as the running shaft variety data. If so, execute step S3; if not, execute step S4;

[0085] S3. Denote this shaft variety data as the first target shaft variety data, and determine whether to update the current batch of running data corresponding to the first target shaft variety data according to the real-time running data;

[0086] S4. Obtain all the historical shaft variety data stored in HBase, and determine whether there is shaft variety data in all the historical shaft variety data that is the same as the running shaft variety data. If so, execute step S5; if not, execute step S6;

[0087] S5. Denote this shaft variety data as the second target shaft variety data, write the second target shaft variety data and the corresponding historical running data stored in HBase into the Flink State to obtain the current batch of shaft variety data that is the same as the second target shaft variety data and the corresponding current batch of running data, and determine whether to update the current batch of running data corresponding to the second target shaft variety data according to the real-time running data;

[0088] S6. Denote the running shaft variety data as the shaft variety data to be added, obtain all the periodically reported shaft variety data stored in the HBase, and write the shaft variety data to be added and the real-time running data into the HBase for storage according to all the periodically reported shaft variety data.

[0089] In one embodiment, the real-time running shaft data includes: acquisition time and number of running circles; the current batch of running data includes: first start time, first end time, first start circle number and first end circle number; the historical running data includes: second start time, second end time, second start circle number and second end circle number.

[0090] In one embodiment, determining whether to update the current batch of running data corresponding to the first target shaft variety data according to the real-time running data includes:

[0091] Determine whether the acquisition time is less than the first start time corresponding to the first target axis variety data. If so, use the acquisition time as the first start time corresponding to the first target axis variety data, and use the number of running cycles as the first start cycle number corresponding to the first target axis variety data, and complete the update; if not, determine whether the acquisition time is greater than the first end time corresponding to the first target axis variety data. If so, use the acquisition time as the first end time corresponding to the first target axis variety data, and use the number of running cycles as the first end cycle number corresponding to the first target axis variety data, and complete the update. If not, do not update the current batch of running data corresponding to the first target axis variety data stored in Flink State.

[0092] In one embodiment, determining whether to update the current batch of running data corresponding to the second target axis variety data according to the real-time running data includes:

[0093] Determine whether the acquisition time is less than the first start time corresponding to the second target axis variety data. If so, use the acquisition time as the first start time corresponding to the second target axis variety data, and use the number of running cycles as the first start cycle number corresponding to the second target axis variety data, and complete the update; if not, determine whether the acquisition time is greater than the first end time corresponding to the second target axis variety data. If so, use the acquisition time as the first end time corresponding to the second target axis variety data, and use the number of running cycles as the first end cycle number corresponding to the second target axis variety data, and complete the update. If not, do not update the current batch of running data corresponding to the second target axis variety data stored in Flink State.

[0094] In one embodiment, writing the to-be-added axis variety data and the real-time running data into HBase for storage according to all the periodically reported axis variety data includes:

[0095] Determine whether there is a periodically reported axis variety data identical to the to-be-added axis variety data among all the periodically reported axis variety data. If so, use the acquisition time as the third end time corresponding to this periodically reported axis variety data, and use the number of running cycles as the third end cycle number corresponding to this periodically reported axis variety data. If not, use the to-be-added axis variety data as the periodically reported axis variety data, use the acquisition time as the third start time corresponding to this periodically reported axis variety data, and use the number of running cycles as the third start cycle number corresponding to this periodically reported axis variety data.

[0096] In one embodiment, it further includes:

[0097] Determine whether the current batch run data corresponding to each current batch axis variety data stored in the Flink State has been updated within a preset period. If not, write the current batch axis variety data and its corresponding current batch run data into HBase, and delete the current batch axis variety data and its corresponding current batch run data in the Flink State.

[0098] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0099] S1. Obtain the real-time run data of the real-time run axis, the run axis variety data of the real-time run axis, all the current batch axis variety data stored in the Flink State, and all the current batch run data corresponding one-to-one to all the current batch axis variety data;

[0100] S2. Determine whether there is axis variety data in all the current batch axis variety data that is the same as the run axis variety data. If so, execute step S3; if not, execute step S4;

[0101] S3. Denote the axis variety data as the first target axis variety data, and determine whether to update the current batch run data corresponding to the first target axis variety data according to the real-time run data;

[0102] S4. Obtain all the historical axis variety data stored in HBase, and determine whether there is axis variety data in all the historical axis variety data that is the same as the run axis variety data. If so, execute step S5; if not, execute step S6;

[0103] S5. Denote the axis variety data as the second target axis variety data, write the second target axis variety data and its corresponding historical run data stored in HBase into the Flink State, obtain the current batch axis variety data that is the same as the second target axis variety data and its corresponding current batch run data, and determine whether to update the current batch run data corresponding to the second target axis variety data according to the real-time run data;

[0104] S6. Denote the run axis variety data as the axis variety data to be added, obtain all the periodically reported axis variety data stored in HBase, and write the axis variety data to be added and the real-time run data into HBase for storage according to all the periodically reported axis variety data.

[0105] In one embodiment, the real-time running axis data includes: acquisition time and number of running cycles; the current batch running data includes: a first start time, a first end time, a first start cycle number, and a first end cycle number; the historical running data includes: a second start time, a second end time, a second start cycle number, and a second end cycle number.

[0106] In one embodiment, the judging whether to update the current batch running data corresponding to the first target axis variety data according to the real-time running data includes:

[0107] Judging whether the acquisition time is less than the first start time corresponding to the first target axis variety data. If so, taking the acquisition time as the first start time corresponding to the first target axis variety data and taking the number of running cycles as the first start cycle number corresponding to the first target axis variety data to complete the update; if not, judging whether the acquisition time is greater than the first end time corresponding to the first target axis variety data. If so, taking the acquisition time as the first end time corresponding to the first target axis variety data and taking the number of running cycles as the first end cycle number corresponding to the first target axis variety data to complete the update. If not, do not update the current batch running data corresponding to the first target axis variety data stored in the Flink State.

[0108] In one embodiment, the judging whether to update the current batch running data corresponding to the second target axis variety data according to the real-time running data includes:

[0109] Judging whether the acquisition time is less than the first start time corresponding to the second target axis variety data. If so, taking the acquisition time as the first start time corresponding to the second target axis variety data and taking the number of running cycles as the first start cycle number corresponding to the second target axis variety data to complete the update; if not, judging whether the acquisition time is greater than the first end time corresponding to the second target axis variety data. If so, taking the acquisition time as the first end time corresponding to the second target axis variety data and taking the number of running cycles as the first end cycle number corresponding to the second target axis variety data to complete the update. If not, do not update the current batch running data corresponding to the second target axis variety data stored in the Flink State.

[0110] In one embodiment, writing the to-be-added axis variety data and the real-time running data into HBase for storage according to all the periodically reported axis variety data includes:

[0111] Determine whether there is periodic reported shaft variety data in all the periodic reported shaft variety data that is the same as the to-be-added shaft variety data. If so, use the acquisition time as the third end time corresponding to the periodic reported shaft variety data, and use the running revolutions as the third end revolution number corresponding to the periodic reported shaft variety data. If not, use the to-be-added shaft variety data as the periodic reported shaft variety data, use the acquisition time as the third start time corresponding to the periodic reported shaft variety data, and use the running revolutions as the third start revolution number corresponding to the periodic reported shaft variety data.

[0112] In one embodiment, it further includes:

[0113] Determine whether the current batch running data corresponding to each current batch shaft variety data stored in the Flink State is updated within a preset period. If not, write the current batch shaft variety data and its corresponding current batch running data into HBase, and delete the current batch shaft variety data and its corresponding current batch running data in the Flink State.

[0114] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0115] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as these combinations of technical features do not conflict, they should be considered as the scope recorded in this specification.

Claims

1. A real-time data processing method based on Flink, characterized in that, it includes: S1. Obtain the real-time operation data of the real-time operation axis, the operation axis variety data of the real-time operation axis, all the current batch axis variety data stored in FlinkState, and all the current batch operation data corresponding one-to-one to all the current batch axis variety data; S2. Judge whether there is axis variety data in all the current batch axis variety data that is the same as the operation axis variety data. If so, execute step S3; if not, execute step S4; S3. Denote this axis variety data as the first target axis variety data, and judge whether to update the current batch operation data corresponding to the first target axis variety data according to the real-time operation data; S4. Obtain all the historical axis variety data stored in HBase, and judge whether there is axis variety data in all the historical axis variety data that is the same as the operation axis variety data. If so, execute step S5; if not, execute step S6; S5. Denote this axis variety data as the second target axis variety data, write the second target axis variety data and the historical operation data stored in HBase corresponding thereto into Flink State to obtain the current batch axis variety data that is the same as the second target axis variety data and the current batch operation data corresponding thereto, and judge whether to update the current batch operation data corresponding to the second target axis variety data according to the real-time operation data; S6. Denote the operation axis variety data as the to-be-added axis variety data, obtain all the periodically reported axis variety data stored in HBase, and write the to-be-added axis variety data and the real-time operation data into HBase for storage according to all the periodically reported axis variety data; The real-time operation axis data includes: acquisition time and number of running circles; the operation axis variety data includes: operation axis number and collector number; the current batch operation data includes: first start time, first end time, first start circle number and first end circle number; the historical operation data includes: second start time, second end time, second start circle number and second end circle number; The judging whether to update the current batch operation data corresponding to the first target axis variety data according to the real-time operation data includes: Judge whether the acquisition time is less than the first start time corresponding to the first target axis variety data. If so, use the acquisition time as the first start time corresponding to the first target axis variety data, and use the number of running circles as the first start circle number corresponding to the first target axis variety data to complete the update; if not, judge whether the acquisition time is greater than the first end time corresponding to the first target axis variety data. If so, use the acquisition time as the first end time corresponding to the first target axis variety data, and use the number of running circles as the first end circle number corresponding to the first target axis variety data to complete the update. If not, do not update the current batch operation data corresponding to the first target axis variety data stored in Flink State; Judging whether to update the current batch of operation data corresponding to the second target axis variety data according to the real-time operation data includes: Judging whether the acquisition time is less than the first start time corresponding to the second target axis variety data. If so, taking the acquisition time as the first start time corresponding to the second target axis variety data and taking the number of running cycles as the first start number of cycles corresponding to the second target axis variety data to complete the update; if not, judging whether the acquisition time is greater than the first end time corresponding to the second target axis variety data. If so, taking the acquisition time as the first end time corresponding to the second target axis variety data and taking the number of running cycles as the first end number of cycles corresponding to the second target axis variety data to complete the update. If not, not updating the current batch of operation data corresponding to the second target axis variety data stored in the Flink State; Writing the to-be-added axis variety data and the real-time operation data into HBase for storage according to all the periodically reported axis variety data includes: Judging whether there is periodically reported axis variety data identical to the to-be-added axis variety data among all the periodically reported axis variety data. If so, taking the acquisition time as the third end time corresponding to the periodically reported axis variety data and taking the number of running cycles as the third end number of cycles corresponding to the periodically reported axis variety data. If not, taking the to-be-added axis variety data as the periodically reported axis variety data, taking the acquisition time as the third start time corresponding to the periodically reported axis variety data, and taking the number of running cycles as the third start number of cycles corresponding to the periodically reported axis variety data.

2. The real-time data processing method based on Flink according to claim 1, characterized in that, further comprising: Judging whether the current batch of operation data corresponding to each current batch of axis variety data stored in the Flink State is updated within a preset time limit. If not, writing the current batch of axis variety data and the corresponding current batch of operation data into HBase and deleting the current batch of axis variety data and the corresponding current batch of operation data in the Flink State.

3. A real-time data processing system based on Flink, characterized in that, comprising: A data acquisition module for executing S1 in claim 1; A data judgment module for executing S2 in claim 1; A judgment and update module for executing S3 in claim 1; An acquisition and judgment module for executing S4 in claim 1; A migration and update module for executing S5 in claim 1; An acquisition and migration module for executing S6 in claim 1; Wherein, the judgment and update logic among the data acquisition module, the data judgment module, the judgment and update module, the acquisition and judgment module, the migration and update module, and the acquisition and migration module is implemented according to the steps of S3-S6 in claim 1.

4. The real-time data processing system based on Flink according to claim 3, characterized in that, The real-time operating axis data includes: acquisition time and number of operating cycles; the current batch of operating data includes: first start time, first end time, first start cycle number, and first end cycle number; the judgment and update module includes: A first judgment unit, configured to judge whether the acquisition time is less than the first start time corresponding to the first target axis variety data. If so, start the first update unit; if not, start the second judgment unit; A first update unit, configured to use the acquisition time as the first start time corresponding to the first target axis variety data, and use the number of operating cycles as the first start cycle number corresponding to the first target axis variety data, and complete the update; A second judgment unit, configured to judge whether the acquisition time is greater than the first end time corresponding to the first target axis variety data. If so, start the second update unit; if not, do not update the current batch of operating data corresponding to the first target axis variety data stored in the Flink State; A second update unit, configured to use the acquisition time as the first end time corresponding to the first target axis variety data, and use the number of operating cycles as the first end cycle number corresponding to the first target axis variety data, and complete the update.

5. A computer device, comprising a memory and a processor, where the memory stores a computer program, characterized in that when the processor executes the computer program, the steps of the method according to claim 1 or 2 are implemented.

6. A computer-readable storage medium, on which a computer program is stored, characterized in that when the computer program is executed by the processor, the steps of the method according to claim 1 or 2 are implemented.

Images