Data link tracking method and device based on event bus mode

Abstract

The embodiment of the invention discloses a data link tracking method and device based on an event bus mode, and the method comprises the steps: collecting the data of a plurality of burying points related to at least one monitoring event issued by a Web end, the data comprising a unique identifier, an affiliated device id, a located endpoint id, a located link id, and data content; preprocessing the collected data through an flink job, aggregating all the data, and converting the aggregated data into a data link diagram; and by consuming a kafka theme, pushing data link diagram information related to the data link diagram and included in the theme to a Web end so as to track the data link. Through the above mode, the embodiment of the invention can realize real-time early warning of events, improves the timeliness and accuracy of the whole data link tracking system, and also has the characteristics of simple operation, visualization, real-time monitoring, high reliability of data source information and the like.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of the Internet of Things, and in particular, to a data link tracing method and device based on an event bus mode. Background Art

[0002] In the information age, with the booming development of the Internet of Things, the development of hardware devices is changing with each passing day, the capabilities of networks and storage spaces are getting stronger and stronger, and data exchanges between devices and between devices and people are becoming increasingly frequent, generating an ever-increasing amount of information. During the process of data exchange and processing, new data will be derived, either diverging or aggregating. As the data continues to change, when the data reaches a certain threshold and triggers a certain event, it becomes increasingly difficult for people to trace the cause of the event.

[0003] Specifically, in the field of the Internet of Vehicles, vehicles send data to the platform gateway through in-vehicle devices. After the gateway receives the data, it first performs processing such as packet combining and authentication, and then transmits it to the Internet of Vehicles platform. The Internet of Vehicles platform will process and analyze the data. In this process, an ETL (Extract-Transform-Load) tool will be used first to extract and store the data based on data characteristics into temporary files or databases, and then operations such as deduplication, filtering, and aggregation will be performed on the data, and then the data will be analyzed and mined. Finally, based on the calculations of various algorithms, a prediction effect will be achieved. As the data diverges and aggregates during the process of processing and transmission, the number of data branch links continues to surge. At the same time, as time goes by, more and more traceable information carried by the data itself is lost. This makes it difficult for operation and maintenance management personnel to trace and reproduce the data when the data reaches a certain set threshold and triggers an abnormal event, and thus it is difficult to troubleshoot. Summary of the Invention

[0004] In view of the above problems, the embodiments of the present invention provide a data link tracing method and device based on an event bus mode, which overcome the above problems or at least partially solve the above problems.

[0005] According to one aspect of the embodiments of the present invention, a data link tracing method based on an event bus mode is provided. The method includes: collecting data of a plurality of buried points related to at least one monitoring event published by the Web end, where the data includes a unique identifier, a device id to which it belongs, an endpoint id where it is located, a link id where it is located, and data content; preprocessing the collected data through a flink job, and aggregating all the data to be transformed into a data link graph; pushing data link graph information included in the topic to the Web end through consuming a kafka topic for data link tracing.

[0006] Optionally, before collecting the data of multiple buried points related to the monitoring event, it includes: receiving the monitoring events subscribed by the Web side, and setting each buried point and the data to be collected by each buried point according to the monitoring event.

[0007] Optionally, the Web side actively selects a monitoring vehicle and publishes a monitoring event to the event bus; or sets a monitoring alarm rule, and when the monitoring alarm rule is reached, actively publishes a monitoring event to the event bus.

[0008] Optionally, setting each buried point and the data to be collected by each buried point according to the monitoring event to be traced includes: setting buried points at the entrance and exit of the device related to at least one monitoring event respectively, and the entrance is the parent node of the exit; generally, the buried points are added to the device related to at least one monitoring event in the form of annotations by AOP technology.

[0009] Optionally, preprocessing the collected data through a flink job and aggregating all the data into a data link diagram, including: consuming the relevant kafka topic, and using the flink job to clean the device upload data information collected, filtering out dirty data and duplicate data; by setting a time window, aggregating the device data within one window according to the parent node information included in the device upload data to be transformed into a data link diagram; transmitting the transformed data link diagram information to the message middleware.

[0010] Optionally, pushing the data link diagram information related to the data link diagram included in the topic to the Web side for data link tracing by consuming the kafka topic, including: persisting the data link diagram information related to the data link diagram included in the topic by consuming the kafka topic; pushing the data link diagram information to the Web side subscribing to the monitoring event and rendering the data link diagram.

[0011] Optionally, the method further includes: managing the monitoring events published by the Web side and notifying each device to upload complete data content information.

[0012] Based on the same inventive concept, a data link tracking device based on an event bus mode is provided, including: a data link information collection module, used to collect data of multiple tracking points related to at least one monitoring event published by a Web terminal, the data including a unique identifier, a device ID, an endpoint ID, a link ID, and data content; a data link information processing module, used to pre-process the collected data through a flink job, and aggregate all the data to convert them into a data link graph; a data link information monitoring module, used to push the data link graph information related to the data link graph included in the topic to the Web terminal for data link tracking by consuming a kafka topic.

[0013] Based on the same inventive concept, an embodiment of the present invention further proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the aforementioned method when executing the program.

[0014] Based on the same inventive concept, an embodiment of the present invention further proposes a computer storage medium, in which at least one executable instruction is stored, and the executable instruction enables a processor to execute the aforementioned method.

[0015] The embodiment of the present invention collects data of multiple tracking points related to at least one monitoring event published by a Web terminal, wherein the data includes a unique identifier, a device ID, an endpoint ID, a link ID, and data content; pre-processes the collected data through a flinkjob, aggregates all the data, and converts them into a data link graph; and pushes the data link graph information related to the data link graph included in the topic to the Web terminal for data link tracking by consuming a kafka topic, thereby achieving real-time early warning of events and improving the timeliness and accuracy of the entire data link tracking system. It also has the characteristics of simple operation, visualization, real-time monitoring, and high reliability of data source information.

[0016] The above description is only an overview of the technical solution of the embodiment of the present invention. In order to more clearly understand the technical means of the embodiment of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiment of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0018] Figure 1 A schematic diagram of the structure of a data link tracking device based on an event bus mode provided by an embodiment of the present invention is shown;

[0019] Figure 2 An example diagram showing a data link tracking device based on an event bus mode according to an embodiment of the present invention;

[0020] Figure 3 An example diagram showing a data link diagram according to an embodiment of the present invention;

[0021] Figure 4 A schematic diagram of a flow chart of a data link tracing method based on an event bus mode provided by an embodiment of the present invention is shown;

[0022] Figure 5 A schematic diagram of an electronic device in an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0023] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.

[0024] Figure 1 FIG. 1 is a schematic diagram showing the structure of a data link tracking device based on an event bus mode provided by an embodiment of the present invention. Figure 1 As shown, the data link tracking device based on the event bus mode is applied to the server, including: a data link information acquisition module, a data link information processing module and a data link information monitoring module.

[0025] A data link information collection module is used to collect data of multiple tracking points related to at least one monitoring event published by the Web end, wherein the data includes a unique identifier, a device ID, an endpoint ID, a link ID, and data content;

[0026] The data link information processing module is used to pre-process the collected data through the flink job, aggregate all the data, and convert them into a data link graph;

[0027] The data link information monitoring module is used to push the data link diagram information related to the data link diagram included in the topic to the Web end for data link tracking by consuming the Kafka topic.

[0028] In the embodiment of the present invention, the data link information acquisition module first receives the monitoring events subscribed by the Web end, and sets each data point and the data to be collected by each data point according to the monitoring events. The Web end actively selects the monitored vehicle and publishes the monitoring event to the event bus; or sets the monitoring alarm rule, and when the monitoring alarm rule is reached, actively publishes the monitoring event to the event bus. As Figure 2 shown, data points are set at devices such as the gateway, flink job, and some servers. Each device uploads the data collected at the data point to the message middleware kafka. After aggregation, it is processed by the flink job to generate a data link diagram and transmitted to the message middleware kafka. Then, the kafka topic is consumed to obtain the link diagram information and trigger an alarm when displayed on the Web end.

[0029] The data link information acquisition module sets data points for the link of the data during the acquisition and transmission process, records the obtained information in accordance with a specific data point information format, including information such as the unique identifier id of the data, the device id to which it belongs, the endpoint id where it is located, the link id where it is located, and the data content. And send the data point record information to the data link information processing module. Among them, the data content is only added when an alarm event is triggered. When no alarm is triggered, the data content is empty. At the same time, the data link information processing module is a subscriber of the event-driven module. When the subscribed events, such as threshold events and active events, are triggered, the data point uploads the complete data information of the device id that triggers the event. For example: when the terminal device data passes through the gateway, the gateway will set data points at the entrance and the exit (that is, the entrance of the next link), and record that the entrance is the parent node of the exit. Taking Java as an example, data points are generally added to the gateway in the form of annotations using Aspect Oriented Programming (AOP) technology without modifying the business logic of the gateway. The data points send the information uploaded by the device to the gateway and the information processed by the gateway to the message middleware such as Kafka.

[0030] The data link information processing module constructs a big data real-time data warehouse model based on Apache flink + kafka, and performs operations such as cleaning, aggregating, and transforming the uploaded data link information through an ETL tool. Finally, it constructs the real-time data link diagram information of the device data and pushes the link diagram information to kafka. Specifically, by consuming the relevant kafka topic, the device upload data information collected by the flink job is cleaned, filtering out dirty data and duplicate data. Then, by setting a time window, the device data within a window is aggregated according to the parent node information included in the device upload data, and finally transformed into a data link diagram. And the transformed data link diagram information is transmitted to the message middleware such as kafka. Figure 3It is a schematic diagram of a data link diagram. Among them, it808-gateway is the gateway, 83ec371fc04b is the device ID, and Kafka, Redis, HBase, and Cassandra are databases for storing different data respectively.

[0031] The data link information monitoring module persists the data link diagram information contained therein by consuming Kafka-related topics. At the same time, as a pusher, it pushes the relevant information of the monitored device to its subscriber, the Web side, and renders the data link diagram. Specifically, the data link information monitoring module is responsible for consuming the link diagram information data in Kafka, and at the same time sets an alarm threshold for the feature information in the uploaded link diagram information. If it is found that the feature information reaches the alarm threshold, an alarm event is triggered, and the event bus module notifies the data link information collection module to upload the link information with the complete data content attached. Finally, the complete link diagram information will be persisted, and the data link information monitoring module will display the entire completed link information diagram that triggers the alarm on the Web side. When no alarm event is triggered, the user can actively trigger an event to monitor the device to be monitored. Web-side users can also actively select a monitored vehicle, publish a monitoring event to the event bus, and can also set monitoring alarm rules. When the monitoring alarm rules are met, a monitoring event is actively published to the time bus.

[0032] The data link tracing device based on the event bus mode in the embodiment of the present invention further includes an event bus module, which is used to manage the events and event processing methods registered by the components of the data link information collection module, implement the publish-subscribe mode, process the triggered monitoring events, and notify the components of the data link information collection module to upload the complete data content information.

[0033] The data link tracing device based on the event bus mode in the embodiment of the present invention can enable the components of the data link tracing to interact in a loosely coupled manner based on the event bus design mode of "event bus -> data collection -> data processing -> data monitoring", without defining specific interfaces and manually adding listeners. Each component can respond efficiently, view the data source information in real time, and ensure the high reliability and traceability of the data by setting the unique ID of the data information. Based on the custom buried point mechanism, without modifying the original business data processing, the data information of each link in the transmission process can be collected, custom alarm rules are supported, the data feature values are monitored in real time, and when the alarm threshold is reached, the system automatically records the data link diagram. Based on the big data real-time data warehouse, the data can be processed in real time, quickly aggregated, and a complete data link diagram can be formed, which can realize real-time early warning of events, improve the timeliness and accuracy of the entire data link tracing system, and at the same time has features such as simplification and lightweight.

[0034] For the convenience of description, when describing the above device, it is divided into various modules according to functions for separate description. Of course, when implementing the embodiments of the present invention, the functions of each module can be implemented in one or more software and / or hardware.

[0035] Based on the same concept, an embodiment of the present invention further provides a schematic flowchart of a data link tracing method based on the event bus mode. As Figure 2 shown, the data link tracing method based on the event bus mode is applied to a server and includes:

[0036] Step S11: Collect data of multiple buried points related to at least one monitoring event published by the Web side, where the data includes a unique identifier, the device id to which it belongs, the endpoint id where it is located, the link id where it is located, and the data content.

[0037] In the embodiment of the present invention, before step S11, receive the monitoring events subscribed by the Web side, and set each buried point and the data to be collected by each buried point according to the monitoring events. The Web side actively selects a monitoring vehicle and publishes a monitoring event to the event bus; or sets a monitoring alarm rule, and when the monitoring alarm rule is reached, actively publishes a monitoring event to the event bus. In step S11, buried points are respectively set at the entrance and exit of the device related to at least one monitoring event, and the entrance is the parent node of the exit; generally, the buried points are added to the device related to at least one monitoring event in the form of annotations using AOP technology. Then, the information uploaded by the device and the information processed by the device are sent to the message middleware kafka through the buried points. For example, taking Java as an example, the buried points are generally added to the gateway in the form of annotations using AOP technology without modifying the business logic of the gateway. The buried points send the information uploaded by the device to the gateway and the information processed by the gateway to the message middleware such as kafka.

[0038] Step S12: Preprocess the collected data through a flink job, and aggregate all the data to be transformed into a data link graph.

[0039] In step S12, optionally, by consuming relevant topics of kafka, the flink job is used to clean the device upload data information collected, filter out dirty data and duplicate data; by setting a time window, for the device data within one window, according to the parent node information included in the device upload data, aggregation is performed to be transformed into a data link graph; the information of the transformed data link graph is passed to the message middleware.

[0040] Build a big data real-time data warehouse model based on Apache flink + kafka, clean, aggregate, transform, etc. the uploaded data link information through an ETL tool, and finally build the real-time data link graph information of device data and push the link graph information to kafka.

[0041] Step S13: Push the data link graph information included in the topic related to the data link graph to the Web side for data link tracking by consuming the kafka topic.

[0042] In the embodiment of the present invention, by consuming the kafka topic, the data link graph information related to the data link graph included in the topic is persisted; the data link graph information is pushed to the Web side subscribing to the monitoring event, and the data link graph is rendered.

[0043] Set an alarm threshold for the feature information in the uploaded link graph information. If it is found that the feature information reaches the alarm threshold, an alarm event is triggered. The event bus module notifies the data link information collection module to upload the link information with the complete data content attached. Finally, the complete link graph information will be persisted. At the same time, the data link information monitoring module displays the entire completed link information graph triggering the alarm on the Web side. When the alarm event is not triggered, the user can actively trigger the event to monitor the device to be monitored. The Web side user can also actively select to monitor the vehicle, publish a monitoring event to the event bus, or set a monitoring alarm rule. When the monitoring alarm rule is reached, a monitoring event is actively published to the time bus.

[0044] In the embodiment of the present invention, the monitoring events published by the Web side are also managed to notify each device to upload the complete data content information. Specifically, manage the events and event processing methods registered by each component for data collection, implement the publish-subscribe mode, process the triggered monitoring events, and notify each component for data collection to upload the complete data content information.

[0045] In summary, the data link tracking method based on the event bus mode in the embodiment of the present invention collects the data of multiple buried points related to at least one monitoring event published by the Web side. The data includes a unique identifier, the device id to which it belongs, the endpoint id where it is located, the link id where it is located, and the data content; preprocesses the collected data through a flink job and aggregates all the data to be transformed into a data link graph; pushes the data link graph information related to the data link graph included in the topic to the Web side for data link tracking by consuming the kafka topic, which can realize real-time early warning of events, improve the timeliness and accuracy of the entire data link tracking system, and at the same time has the characteristics of simple operation, visualization, real-time monitoring, and high reliability of data source information.

[0046] The specific embodiments of the present invention have been described above. In some cases, the actions or steps recited in the embodiments of the present invention may be performed in an order different from that in the embodiments and still achieve the desired results. Additionally, the processes depicted in the drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0047] The method of the above embodiments is applied to the corresponding device in the foregoing embodiments and has the beneficial effects of the corresponding device embodiments, which will not be elaborated herein.

[0048] Based on the same inventive concept, an embodiment of the present invention further provides an electronic device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the method described in any one of the above embodiments.

[0049] An embodiment of the present invention provides a non-volatile computer storage medium, which stores at least one executable instruction, and the computer executable instruction can execute the method described in any one of the above embodiments.

[0050] Figure 5 A more specific schematic diagram of the hardware structure of the electronic device provided in this embodiment is shown. The device may include: a processor 501, a memory 502, an input / output interface 503, a communication interface 504, and a bus 505. Among them, the processor 501, the memory 502, the input / output interface 503, and the communication interface 504 are communicatively connected to each other inside the device through the bus 505.

[0051] The processor 501 may be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is used to execute relevant programs to implement the technical solutions provided by the method embodiments of the present invention.

[0052] The memory 502 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage devices, dynamic storage devices, etc. The memory 502 can store the operating system and other application programs. When implementing the technical solutions provided by the method embodiments of the present invention through software or firmware, the relevant program codes are stored in the memory 502 and called and executed by the processor 501.

[0053] The input / output interface 503 is used to connect to the input / output module to achieve information input and output. The input / output module can be configured as a component in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Among them, the input device can include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device can include a display, a speaker, a vibrator, an indicator light, etc.

[0054] The communication interface 504 is used to connect to the communication module (not shown in the figure) to achieve communication interaction between this device and other devices. Among them, the communication module can achieve communication through wired means (such as USB, network cable, etc.) or through wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0055] The bus 505 includes a path for transmitting information between various components of the device (such as the processor 501, the memory 502, the input / output interface 503, and the communication interface 504).

[0056] It should be noted that although the above device only shows the processor 501, the memory 502, the input / output interface 503, the communication interface 504, and the bus 505, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, those skilled in the art can understand that the above device may also only include the components necessary to implement the solution of the embodiments of the present invention, and does not necessarily include all the components shown in the figure.

[0057] Those of ordinary skill in the art should understand that: the discussion of any of the above embodiments is only exemplary and is not intended to imply that the scope of the present disclosure is limited to these examples; under the concept of the present disclosure, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations in different aspects of the embodiments of the present invention as described above, and they are not provided in detail for the sake of brevity.

[0058] This application is intended to cover all such substitutions, modifications, and variations that fall within the broad scope of all embodiments. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present invention shall be included within the scope of protection of this disclosure.

Claims

1. A data link tracing method based on the event bus mode, characterized in that, The method includes: Collecting data of multiple buried points related to at least one monitoring event published by the Web side, where the data includes a unique identifier, the ID of the affiliated device, the ID of the location endpoint, the ID of the location link, and the data content; Preprocessing the collected data through a Flink job, and aggregating all the data to be transformed into a data link graph; By consuming the Kafka topic, pushing the data link graph information included in the topic related to the data link graph to the Web side for data link tracing.

2. The method according to claim 1, characterized in that, Before collecting the data of multiple buried points related to the monitoring event, it includes: Receiving the monitoring events subscribed by the Web side, and setting each buried point and the data to be collected by each buried point according to the monitoring events.

3. The method according to claim 2, characterized in that, The Web side actively selects a monitoring vehicle and publishes a monitoring event to the event bus; or sets a monitoring alarm rule, and when the monitoring alarm rule is reached, actively publishes a monitoring event to the event bus.

4. The method according to claim 2, characterized in that, The setting of each buried point and the data to be collected by each buried point according to the monitoring event to be traced includes: Setting buried points at the entrance and exit of the device related to at least one monitoring event respectively, and the entrance is the parent node of the exit; The buried points are generally added to the device related to at least one monitoring event in the form of annotations using AOP technology.

5. The method according to claim 1, characterized in that, The preprocessing of the collected data through a Flink job and the aggregation of all the data to be transformed into a data link graph includes: By consuming the relevant Kafka topic, applying the Flink job to clean the device upload data information collected, filtering out dirty data and duplicate data; By setting a time window, aggregating the device data within one window according to the parent node information included in the device upload data to be transformed into a data link graph; Transmitting the transformed data link graph information to the message middleware.

6. The method according to claim 1, characterized in that, The pushing of the data link graph information included in the topic related to the data link graph to the Web side for data link tracing by consuming the Kafka topic includes: By consuming the Kafka topic, persisting the data link graph information included in the topic related to the data link graph; Pushing the data link graph information to the Web side subscribing to the monitoring event and rendering the data link graph.

7. The method according to claim 1, characterized in that, The method further includes: Managing the monitoring events published by the Web side and notifying each device to upload complete data content information.

8. A data link tracing device based on the event bus mode, characterized in that, The device includes: A data link information collection module, which is used to collect data of multiple buried points related to at least one monitoring event published by the Web side, where the data includes a unique identifier, the ID of the affiliated device, the ID of the location endpoint, the ID of the location link, and the data content; A data link information processing module, which is used to preprocess the collected data through a Flink job and aggregate all the data to be transformed into a data link graph; A data link information monitoring module, which is used to push the data link graph information included in the topic related to the data link graph to the Web side for data link tracing by consuming the Kafka topic.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, the method described in any one of claims 1-7 is implemented.

10. A computer storage medium, characterized in that, At least one executable instruction is stored in the storage medium, and the executable instruction causes the processor to execute the method described in any one of claims 1-7.

Images