Data real-time sampling method, system and equipment based on Flink and medium

Abstract

The invention discloses a data real-time sampling method, system and device based on Flink and a medium, and the method comprises the steps: collecting Internet of Things data, and writing the collected Internet of Things data into Kakfa; a streaming processing engine Flink consumes data in Kakfa in real time, preprocesses and samples the data and then writes the data into a database, and the method, the system, the equipment and the medium can sample the data in real time.

Description

Technical Field

[0001] The present invention belongs to the technical field of data sampling, and relates to a Flink-based real-time data sampling method, system, device and medium. Background Art

[0002] With the rapid development of information technology and communication networks, the Internet of Things has become a key link between the physical world and the digital world. Through smart devices, sensors and communication technologies, the Internet of Things can achieve real-time monitoring, data collection and intelligent analysis of the environment, equipment and systems, and promote various industries into a new era of digitalization and intelligence.

[0003] IoT data refers to various types of data collected through IoT devices, sensors, smart terminals, etc. These data cover information such as physical phenomena, environmental changes, and device status in the real world. With the continuous development of IoT technology and the continuous expansion of its applications, the generation, transmission, and application scope of IoT data are becoming increasingly extensive, becoming an important information resource in the digital age.

[0004] IoT data includes environmental data (including environmental monitoring data such as temperature, humidity, air pressure, air quality, and light intensity. Common in smart home, environmental monitoring, weather forecast and other fields), location data (location information of equipment or personnel collected through technologies such as GPS, Bluetooth, Wi-Fi, etc. This type of data is widely used in smart transportation, logistics tracking, precise positioning and other fields), equipment status data (operating status data of IoT equipment, such as battery power, equipment fault detection, operating time, performance indicators, etc. Common in smart manufacturing, equipment monitoring, industrial automation and other fields), industrial data (including equipment operating data, production line status, energy consumption, etc. These data are crucial for optimizing the production process and improving production efficiency) and other data types. It has the characteristics of diversity, real-time, sequential, and massive.

[0005] After IoT data is collected, different application scenarios require different data granularities. For example, for alarm scenarios, the data collection frequency needs to be very high to meet the real-time requirements of alarms, requiring a collection frequency of seconds or even milliseconds. However, for scenarios such as trend analysis, second-level or even millisecond-level data is often not required, and minute-level or hour-level data can also meet the needs in specific scenarios. This requires that data storage needs to save data of different granularities and use data of different granularities for different business scenarios, which results in the inability to sample IoT data in real time. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a Flink-based real-time data sampling method, system, device and medium, which can sample data in real time.

[0007] To achieve the above object, the present invention discloses a real-time data sampling method based on Flink, comprising:

[0008] Collect IoT data and write the collected IoT data into Kakfa;

[0009] The stream processing engine Flink consumes the data in Kakfa in real time, preprocesses and samples the data, and then writes it into the database.

[0010] The further improvement of the Flink-based real-time data sampling method of the present invention is:

[0011] Furthermore, the process of preprocessing and sampling the data and then writing it into the database is as follows:

[0012] Filter out invalid data with incorrect format and adjust the granularity of the timestamp bound to the data;

[0013] The streaming processing engine Flink queries the data cache. If the timestamp of the data already exists in the cache, the data is directly discarded. Otherwise, the data is written to the database and the timestamp in the cache is updated to the timestamp of the data.

[0014] Furthermore, in actual business scenarios, data with a granularity of 1 minute is used and the timestamp is formatted as a whole minute timestamp.

[0015] Furthermore, it also includes:

[0016] When the service of the stream processing engine Flink fails in data consumption, preprocessing, and sampling due to external factors, the data is written to the external file system. After the service of the stream processing engine Flink returns to normal, the data is read from the external file system and written to the database.

[0017] The present invention discloses a real-time data sampling system based on Flink, comprising:

[0018] The collection module is used to collect IoT data and write the collected IoT data into Kakfa;

[0019] The processing module is used by the stream processing engine Flink to consume data in Kakfa in real time, preprocess and sample the data, and then write it into the database.

[0020] The further improvement of the Flink-based real-time data sampling system of the present invention is:

[0021] Furthermore, the processing module includes:

[0022] The filtering module is used to filter out invalid data with incorrect formats and adjust the granularity of the timestamp bound to the data;

[0023] The write module is used by the stream processing engine Flink to query the data cache. If the timestamp of the data already exists in the cache, the data is directly discarded. Otherwise, the data is written to the database and the timestamp in the cache is updated to the timestamp of the data.

[0024] Furthermore, in actual business scenarios, data with a granularity of 1 minute is used and the timestamp is formatted as a whole minute timestamp.

[0025] Furthermore, it also includes:

[0026] The configuration module is used to write data to an external file system when the service of the stream processing engine Flink fails in data consumption, preprocessing, and sampling due to external factors. After the service of the stream processing engine Flink returns to normal, the data is read from the external file system and written to the database.

[0027] The present invention discloses a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the Flink-based real-time data sampling method are implemented.

[0028] The present invention discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the real-time data sampling method based on Flink are implemented.

[0029] The present invention has the following beneficial effects:

[0030] In the specific operation of the Flink-based real-time data sampling method, system, device and medium described in the present invention, after the IoT data is collected, the data is temporarily written into Kakfa because the data collection frequency is often high and the data volume is large. Kakfa, as a message middleware, has a powerful data writing capability and is very suitable for scenarios with large data volumes. It can "cut the peak" when processing data to ensure the overall stable operation of the service. Then, the streaming processing engine Flink is used to sample the data in real time. The operation is simple, convenient and extremely practical. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0032] Figure 1It is the principle diagram of the present invention;

[0033] Figure 2 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] In the description of the present invention, it should be understood that the terms “include” and “comprises” indicate the presence of described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0036] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0037] It should be further understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects are in an "or" relationship.

[0038] It should be understood that, although the terms first, second, third, etc. may be used to describe preset ranges, etc. in the embodiments of the present invention, these preset ranges should not be limited to these terms. These terms are only used to distinguish preset ranges from each other. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0039] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0040] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention described and shown in the drawings here can usually be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0041] Various structural schematic diagrams of the embodiments disclosed in the present invention are shown in the accompanying drawings. These figures are not drawn to scale, and some details are magnified and some details may be omitted for the purpose of clear expression. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary, and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may additionally design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0042] Embodiment 1

[0043] refer to Figure 2 The real-time data sampling method based on Flink of the present invention comprises the following steps:

[0044] Collect IoT data and write the collected IoT data into Kakfa;

[0045] Among them, it should be noted that since data is often collected frequently and in large quantities, the data is temporarily written into Kakfa. As a message middleware, Kakfa has powerful data writing capabilities and is very suitable for scenarios with large data volumes. It can achieve peak shaving when processing data and ensure the overall stable operation of the service.

[0046] After the IoT data is written into Kafka, the stream processing engine Flink consumes the data in Kafka in real time, pre-processes and samples the data, and then writes it into the database.

[0047] refer to Figure 1 , the stream processing engine Flink consumes the data in Kakfa in real time, and the process of preprocessing and sampling the data is as follows:

[0048] 1) After the stream processing engine Flink consumes the device data in Kafka, it pre-processes the device data;

[0049] The preprocessing process is: first filter out invalid device data with incorrect format, and adjust the granularity of the timestamp bound to the device data. For example, if device data with a granularity of 1 minute is used in an actual business scenario, the timestamp is formatted as a timestamp of the whole minute.

[0050] 2) The streaming processing engine Flink queries the device's data cache. If the timestamp of the device data already exists in the cache, the device data is directly discarded. Otherwise, the device data is written to the database and the timestamp in the cache is updated to the timestamp of the current device data. After the above processing, only one piece of data of a device exists in the database per minute, which greatly reduces the data storage volume while meeting business needs, effectively saves resources and improves query efficiency.

[0051] 3) When the service fails due to external factors, the stateful computing capability of the stream processing engine Flink is used to periodically persist the cached data to the external file system; when the Flink service is restored, the cached data is loaded from the external file system into the service memory to ensure that subsequent data can still be processed normally and written to the database. The process is as follows: Figure 2 shown.

[0052] Embodiment 2

[0053] The Flink-based real-time data sampling system of the present invention includes:

[0054] The collection module is used to collect IoT data and write the collected IoT data into Kakfa;

[0055] The processing module is used by the stream processing engine Flink to consume data in Kakfa in real time, preprocess and sample the data, and then write it into the database.

[0056] As an embodiment of the present invention, the processing module includes:

[0057] The filtering module is used to filter out invalid data with incorrect formats and adjust the granularity of the timestamp bound to the data;

[0058] The write module is used by the stream processing engine Flink to query the data cache. If the timestamp of the data already exists in the cache, the data is directly discarded. Otherwise, the data is written to the database and the timestamp in the cache is updated to the timestamp of the data.

[0059] As an implementation of the present invention, in an actual business scenario, data with a granularity of 1 minute is used to format the timestamp into a timestamp of a whole minute.

[0060] As an embodiment of the present invention, it also includes:

[0061] The configuration module is used to write data to an external file system when the service of the stream processing engine Flink fails in data consumption, preprocessing, and sampling due to external factors. After the service of the stream processing engine Flink returns to normal, the data is read from the external file system and written to the database.

[0062] The division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present application may be integrated into a processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0063] Embodiment 3

[0064] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method for generating and uploading the trusted status alarm information are implemented, for example, including: collecting IoT data and writing the collected IoT data into Kakfa; the stream processing engine Flink consumes the data in Kakfa in real time, preprocesses and samples the data, and then writes it into a database. The process of preprocessing and sampling the data and then writing it into the database is as follows: filtering out invalid data with format errors and adjusting the granularity of the timestamp bound to the data; the stream processing engine Flink queries the data cache, and when the timestamp of the data already exists in the cache, the data is directly discarded, otherwise, the data is written into the database, and the timestamp in the cache is updated to the timestamp of the data. In actual business scenarios, data with a granularity of 1 minute is used to format the timestamp as a whole minute timestamp. It also includes: when the service of the streaming processing engine Flink fails in data consumption, preprocessing and sampling due to external factors, the data is written to the external file system, and after the service of the streaming processing engine Flink returns to normal, the data is read from the external file system and written to the database. Among them, the memory may include a memory, such as a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk memory, etc.; the processor, network interface, and memory are interconnected through an internal bus, and the internal bus can be an industrial standard architecture bus, a peripheral component interconnection standard bus, an extended industrial standard structure bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory is used to store programs. Specifically, the program may include a program code, and the program code includes computer operation instructions. The memory may include a memory and a non-volatile memory, and provide instructions and data to the processor.

[0065] Embodiment 4

[0066] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for generating and sending the trusted state alarm information are implemented, for example, including: collecting Internet of Things data, and writing the collected Internet of Things data into Kakfa; the stream processing engine Flink consumes the data in Kakfa in real time, and preprocesses and samples the data, and then writes it into a database. The process of preprocessing and sampling the data and then writing it into the database is: filtering out invalid data with format errors, and adjusting the granularity of the timestamp bound to the data; the stream processing engine Flink queries the data cache, and when the timestamp of the data already exists in the cache, the data is directly discarded, otherwise, the data is written to the database, and the timestamp in the cache is updated to the timestamp of the data. In actual business scenarios, data with a granularity of 1 minute is used, and the timestamp is formatted as a timestamp of a whole minute. It also includes: when the service of the stream processing engine Flink fails in data consumption, preprocessing and sampling due to external factors, the data is written into an external file system, and after the service of the stream processing engine Flink returns to normal, the data is read from the external file system and written into the database. Specifically, the computer-readable storage medium includes, but is not limited to, for example, volatile memory and / or non-volatile memory. The volatile memory may include random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include read-only memory (ROM), hard disk, flash memory, optical disk, magnetic disk, etc.

[0067] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0068] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0069] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0070] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0071] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and disclosure of the invention. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed by the present invention. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0072] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

[0073] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any simple modification, change and equivalent structural change made to the above embodiment based on the technical essence of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A real-time data sampling method based on Flink, characterized in that: include: Collect IoT data and write the collected IoT data into Kakfa; The stream processing engine Flink consumes the data in Kakfa in real time, preprocesses and samples the data, and then writes it into the database.

2. The real-time data sampling method based on Flink according to claim 1 is characterized in that: The process of preprocessing and sampling the data and then writing it into the database is as follows: Filter out invalid data with incorrect format and adjust the granularity of the timestamp bound to the data; The streaming processing engine Flink queries the data cache. If the timestamp of the data already exists in the cache, the data is directly discarded. Otherwise, the data is written to the database and the timestamp in the cache is updated to the timestamp of the data.

3. The real-time data sampling method based on Flink according to claim 1 is characterized in that: In actual business scenarios, use data with a granularity of 1 minute and format the timestamp as a whole minute timestamp.

4. The real-time data sampling method based on Flink according to claim 1 is characterized in that: Also includes: When the service of the stream processing engine Flink fails in data consumption, preprocessing, and sampling due to external factors, the data is written to the external file system. After the service of the stream processing engine Flink returns to normal, the data is read from the external file system and written to the database.

5. A real-time data sampling system based on Flink, characterized in that: include: The collection module is used to collect IoT data and write the collected IoT data into Kakfa; The processing module is used by the stream processing engine Flink to consume data in Kakfa in real time, preprocess and sample the data, and then write it into the database.

6. The Flink-based real-time data sampling system according to claim 5, characterized in that: The processing modules include: The filtering module is used to filter out invalid data with incorrect formats and adjust the granularity of the timestamp bound to the data; The write module is used by the stream processing engine Flink to query the data cache. If the timestamp of the data already exists in the cache, the data is directly discarded. Otherwise, the data is written to the database and the timestamp in the cache is updated to the timestamp of the data.

7. The Flink-based real-time data sampling system according to claim 5, characterized in that: In actual business scenarios, use data with a granularity of 1 minute and format the timestamp as a whole minute timestamp.

8. The Flink-based real-time data sampling system according to claim 5, characterized in that: Also includes: The configuration module is used to write data to an external file system when the service of the stream processing engine Flink fails in data consumption, preprocessing, and sampling due to external factors. After the service of the stream processing engine Flink returns to normal, the data is read from the external file system and written to the database.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the real-time data sampling method based on Flink are implemented as described in any one of claims 1 to 4.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the real-time data sampling method based on Flink are implemented as described in any one of claims 1 to 4.

Images