Data operation and maintenance management system

Abstract

The embodiment of the invention discloses a data operation and maintenance management system. The system comprises a data operation and maintenance management platform, middleware and at least one data collection end, wherein for each data collection end, the data collection ends are used for executing data collection operation for market information nodes to obtain collected data and pushing the collected data to the middleware, and the data collection ends and the market information nodes are deployed in an associated mode; the data operation and maintenance management platform is used for querying target data and performing operation and maintenance management on the target data, and the target data is obtained according to the collected data received by the middleware. According to the technical scheme, unified operation and maintenance management of the target data can be achieved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of data management technology, and in particular to a data operation and maintenance management system. Background Art

[0002] With the launch and widespread use of streaming market information on various exchanges, related companies have gradually completed the upgrade and transformation of various market-related application systems, and the acceleration of the new round of market information has also tended to stabilize.

[0003] In order to improve the work efficiency of market operation and maintenance personnel, there is an increasingly urgent need for unified operation and maintenance management of the collected data obtained from market nodes and the target data obtained based on the collected data.

[0004] However, it is currently difficult to carry out unified operation and maintenance management of target data, which needs to be solved urgently. Summary of the invention

[0005] The embodiment of the present invention provides a data operation and maintenance management system to achieve unified operation and maintenance management of target data.

[0006] According to one aspect of the present invention, a data operation and maintenance management system is provided, which may include: a data operation and maintenance management platform, middleware, and at least one data collection terminal; wherein:

[0007] For each data collection terminal, the data collection terminal is used to perform data collection operations on the market node, obtain the collected data, and push the collected data to the middleware, wherein the data collection terminal is associated with the market node for deployment;

[0008] The data operation and maintenance management platform is used to query the target data and perform operation and maintenance management on the target data, wherein the target data is obtained based on the collected data received by the middleware.

[0009] The technical solution of the embodiment of the present invention is to perform data collection operations on each data collection terminal for the market node to obtain collected data and push the collected data to the middleware, wherein the data collection terminal is deployed in association with the market node, thereby realizing the pushing of the collected data to the unified middleware; the data operation and maintenance management platform is used to perform operation and maintenance management on the target data obtained according to the collected data received by the middleware, and realize the operation and maintenance management of the target data on the data operation and maintenance management platform. The above technical solution can realize the unified collection of each collected data by adopting at least one data collection terminal deployed in association with each market node, and can realize the unified operation and maintenance management of the target data by pushing the uniformly collected collected data to the middleware and adopting the unified operation and maintenance management platform to operate and manage the target data.

[0010] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present invention, nor is it used to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0012] Figure 1 is a structural block diagram of a data operation and maintenance management system provided according to an embodiment of the present invention;

[0013] Figure 2 is a flowchart of pushing collected data to middleware in a data operation and maintenance management system provided according to an embodiment of the present invention;

[0014] Figure 3 is a structural block diagram of a display large screen in a data operation and maintenance management system provided according to an embodiment of the present invention;

[0015] Figure 4 is a structural block diagram of another data operation and maintenance management system provided according to an embodiment of the present invention;

[0016] Figure 5 is a flowchart of storing decoded data into a collection database in another data operation and maintenance management system provided according to an embodiment of the present invention;

[0017] Figure 6 is a structural block diagram of yet another data operation and maintenance management system provided according to an embodiment of the present invention.

[0018] Figure 7 is a flowchart of data operation and maintenance management of an optional example in yet another data operation and maintenance management system provided according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0019] In order to enable those skilled in the art to better understand the solution of the present invention, the following clearly and completely describes the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0020] It should be noted that the terms "first", "second", etc. in the specification, claims and above-mentioned drawings of the present invention are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present invention described here can be implemented in an order other than those illustrated or described here. The same is true for "target", "original", etc., which will not be elaborated here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0021] Before introducing the embodiments of the present invention, an exemplary description will be given of the implementation process of the current solution for data operation and maintenance management and the reasons for the problem that it is difficult to perform unified operation and maintenance management of target data, so as to better understand the reasons why the solution proposed in the embodiments of the present invention can achieve unified operation and maintenance management of target data.

[0022] Exemplarily, in the process of market quotation distribution, the defects of using traditional data operation and maintenance management solutions are gradually exposed. Specifically, due to the wide use of market quotations, the scale of market quotation nodes of each enterprise is also getting larger and larger. The client / server (C / S) deployment method adopted by traditional data operation and maintenance management cannot meet the requirements of data operation and maintenance management. And the data operation and maintenance management method based on the browser / server (B / S) architecture on the market still focuses on the individual management of market quotation nodes, and there is a problem that multiple market quotation nodes cannot be associated. Therefore, it is currently difficult to perform unified operation and maintenance management of target data, and it is thus very difficult to meet the construction requirements of a modern market quotation center.

[0023] In addition, due to the large variety of suppliers related to market quotations, the data operation and maintenance management systems corresponding to different suppliers can only operate and maintain the target data corresponding to their respective market quotation nodes, and cannot be compatible and managed with each other. However, switching between various data operation and maintenance management systems back and forth will increase the management complexity, and there is also a phenomenon of timely response to problem services. Therefore, it is currently difficult to perform unified operation and maintenance management of target data.

[0024] In view of this, in the embodiments of the present invention, in order to improve the work efficiency of market operation and maintenance personnel and provide stable and reliable data support for various operation and maintenance business systems, by adopting at least one data collection end associated with each market node, unified collection of the collected data can be achieved. By pushing the collected data collected uniformly to the middleware and adopting a unified operation and maintenance management platform to perform operation and maintenance management on the target data, unified operation and maintenance management of the target data can be achieved. The following will elaborate on this in detail.

[0025] Figure 1 FIG. is a structural block diagram of a data operation and maintenance management system provided by an embodiment of the present invention. This embodiment is applicable to the situation of data operation and maintenance management, especially applicable to the situation of operation and maintenance management of target data.

[0026] See Figure 1 , the data operation and maintenance management system of the embodiment of the present invention includes: a data operation and maintenance management platform 110, a middleware 120, and at least one data collection end 130; wherein,

[0027] For each data collection end 130, the data collection end 130 is configured to perform a data collection operation on the market node to obtain collected data, and push the collected data to the middleware 120, wherein the data collection end 130 is deployed in association with the market node;

[0028] The data operation and maintenance management platform 110 is configured to query the target data and perform operation and maintenance management on the target data, wherein the target data is obtained according to the collected data received by the middleware 120.

[0029] Among them, the data collection end 130 can be understood as an end for collecting the collected data; the data collection end 130 can exist in the form of a market collection client; in the case where the middleware 120 is a Kafka message middleware, the data collection end 130 can exist as a Kafka producer.

[0030] The market node can be understood as a node that can output or generate collected data related to the market; the market node can be, for example, a data operation and maintenance management system corresponding to an enterprise or a supplier.

[0031] The collected data can be understood as data obtained by performing a data collection operation on the market node; the collected data can include, for example, at least one of market data, node logs, registration information, and node information, etc.

[0032] The middleware 120 can be understood as an intermediate bridge for receiving and collecting data; for example, the middleware 120 can be a Kafka message middleware, and a Kafka service can be deployed on the Kafka message middleware to receive and collect data; it can be understood that receiving and collecting data through the middleware 120 can effectively solve the data collection pressure of the data operation and maintenance management system.

[0033] In the embodiment of the present invention, a data collection end 130 can be pre-installed on a machine with a market condition node, so that the data collection end 130 located on the same machine is associated and deployed with the market condition node. After installing the data collection end 130, the middleware 120 configuration is performed on the data collection end 130, for example, Kafka service information is configured for the data collection end 130, so that the data collection end 130 can push the collected data to the middleware 120. In the embodiment of the present invention, the data collection end 130 can be installed only on the machine of the market condition node that serves as the primary node.

[0034] In the embodiment of the present invention, the data collection end 130 associated and deployed with the market condition node can perform data collection operations on the market condition node, obtain the collected data, and push the collected data to the middleware 120.

[0035] In the embodiment of the present invention, after the data collection end 130 is started, it can automatically perform data collection operations on the market condition node and push the obtained collected data to the middleware 120.

[0036] In the embodiment of the present invention, to meet business high availability and load balancing, the middleware 120 can be deployed in a cluster deployment manner.

[0037] Exemplarily, refer to Figure 2During the process of market data forwarding through the market data forwarding service, the market node will output transmission logs and alarm information in real time as node logs through the forwarding log service, and at the same time provide market ports for different market conditions externally; the data collection end 130, as a kafka producer, is configured with log collection specifications such as market node log monitoring and market ports; after the collection service of the data collection end 130 is started, it will generate unique node identifiers corresponding to each market node, automatically collect node logs through the node identifiers, and collect market data in real time, and send the node logs and node identifiers to the kafka message middleware through topic 1 (topic1), and send the node identifiers and market data to the kafka message middleware through topic 2 (topic2); the data collection end 130 can also generate registration data corresponding to each unregistered market node and send it to the kafka message middleware through topic 0 (topic0), where the kafka message middleware can be deployed in a cluster, for example, multiple middleware 120 (broker) such as middleware 0 and middleware 1 can be used for load balancing.

[0038] The data operation and maintenance management platform 110 can be understood as a unified platform for data operation and maintenance management.

[0039] The target data can be understood as the data to be operated and maintained; the target data can include, for example, at least one of market data, node logs, registration information, and node information, etc.; specifically, the target data can include, for example, at least one of the number of nodes in the sub-market, node type, node status statistics, total node data, statistics of the number of alarms in the node sub-market, node sub-market logs, and real-time market index, etc.; the target data can exist in the form of classified data, that is, the target data can include multiple different classified data, and the classification can be, for example, market data classification, node log classification, registration information classification, or node information classification, etc.

[0040] It should be noted that the target data is obtained based on the collected data received by the middleware 120. For example, the collected data received by the middleware 120 may be at least one of market data, node logs, registration information, and node information, etc. The data operation and maintenance management platform 110, the market decoding end, or the collection database can perform statistical analysis on the collected data to obtain at least one of the number of nodes in the sub-market, node type, node status statistics, total node data, statistics of the number of alarms in the node sub-market, node sub-market logs, and real-time market index, etc. as the target data.

[0041] In the embodiment of the present invention, the data operation and maintenance management platform 110 can query the target data and perform operation and maintenance management in aspects such as displaying, modifying, adding, deleting, or statistical analysis of the target data.

[0042] In an embodiment of the present invention, the data operation and maintenance management platform 110 can also manage the market nodes by managing the target data. For example, deleting the registration information of a certain market node as the target data, that is, realizing the deletion of the market node in the data operation and maintenance management platform 110. Therefore, by managing the target data, at least one of registration, display, modification, addition, deletion, and statistics of the market nodes can be achieved.

[0043] In an embodiment of the present invention, the data operation and maintenance management platform 110 can also query the market nodes and manage at least one aspect such as display, modification, addition, deletion, and statistics of the market nodes. For example, it can query the market nodes and perform display management on the node sub-market logs of the market nodes.

[0044] Exemplarily, the market nodes can be preset as the nodes to be managed in the data operation and maintenance management platform 110. The data operation and maintenance management platform 110 can periodically or in real time query the relevant data related to the nodes to be managed as the target data, and perform display management on the target data to achieve unified detection of the market nodes, thereby realizing the data operation and maintenance management platform 110 as the market detection center.

[0045] In an embodiment of the present invention, the data operation and maintenance management platform 110 can query the target data periodically or in real time, and perform operation and maintenance management on the target data. For example, the data operation and maintenance management platform 110 can query the target data periodically or in real time and perform display on the target data to achieve real-time scrolling of the target data.

[0046] In an embodiment of the present invention, the data operation and maintenance management platform 110 can query the target data in response to a data query instruction. For example, the data operation and maintenance management platform 110 can query the target data according to at least one of the market node, query type, query condition, and query keyword characterized by the data query instruction in response to the data query instruction.

[0047] It can be understood that in the related solutions, the target data is only displayed in the counter program, and there is a lack of intuitive display of the target data in places such as the market forwarding center. Therefore, a unified display large screen can be set for the data operation and maintenance management platform 110 in places such as the market forwarding center to display the target data for market operation and maintenance personnel to view, thereby realizing the operation and maintenance management of the target data.

[0048] Exemplarily, see Figure 3, the data operation and maintenance management platform 110 may include a display screen. The display screen can be configured with a corresponding data center for visually displaying the enterprise data center structure. The display screen may also include a market node type and status statistics module, and the content it displays can help market operation and maintenance personnel count market data. The display screen may also include an overall node statistics module, which contains an overview of all market nodes of the connected platform. The display screen may also include an alarm statistics module, which contains the abnormal information of all market nodes and performs real-time alarms. The display screen may also include a market node real-time log statistics module, in which the node logs of all market nodes can be scrolled and played in real time. The display screen may also include a market index display module, which can display the current market data in real time. The market data may include information such as five-level market data, trading volume, transaction price, latest price, and opening price. The above data displayed on the display screen is the target data to be displayed.

[0049] In the technical solution of the embodiment of the present invention, for each data collection end, the data collection end is used to perform data collection operations on market nodes to obtain collected data and push the collected data to the middleware. Among them, the data collection end is deployed in association with the market nodes, so as to realize pushing the collected data to a unified middleware; the data operation and maintenance management platform is used to perform operation and maintenance management on the target data obtained according to the collected data received by the middleware, so as to realize the operation and maintenance management of the target data on the data operation and maintenance management platform. In the above technical solution, by using at least one data collection end deployed in association with each market node, unified collection of each collected data can be realized. By pushing the unifiedly collected data to the middleware and using a unified operation and maintenance management platform to perform operation and maintenance management on the target data, unified operation and maintenance management of the target data can be realized.

[0050] To better understand the technical solutions of the above embodiments of the present invention, an optional example is provided herein. Exemplarily, for a market-related enterprise, the enterprise has multiple different suppliers, each supplier corresponds to a market node respectively, and each supplier can only perform data operation and maintenance management for its corresponding market node. A data collection end can be installed on each machine with a market node. Each data collection end performs data collection operations for the managed and deployed market nodes to obtain collected data, and pushes the collected data to the middleware. The data operation and maintenance management platform queries target data including real-time market indices and historical market data. The target data is determined based on the collected data and is displayed to achieve unified data operation and maintenance management. Specifically, it realizes the integration of the collected data corresponding to the market nodes respectively corresponding to different suppliers, realizes the same data collection for multiple market nodes on the market forwarding link, and at the same time provides a market index display function and a historical market data display function. Thus, in the process of operation and maintenance management, it can effectively help market operation and maintenance personnel troubleshoot problems or cooperate with tests to check data, so as to quickly discover and locate problems and debug errors, provide perfect data operation and maintenance management functions for market operation and maintenance personnel, and effectively improve the efficiency of operation and maintenance management.

[0051] An optional technical solution is that the number of market nodes corresponding to the data collection end is at least one; the data collection end is specifically used to perform data collection operations for each of the at least one market nodes, obtain collected data, and push the collected data to the middleware.

[0052] In the embodiments of the present invention, the number of market nodes corresponding to the data collection end is at least one. For example, on a machine with a market node, there may be at least one market node on the machine, and a data collection end can be installed on the machine so that the data collection end is deployed in association with the at least one market node.

[0053] It should be noted that the data operation and maintenance management platform can query the target data and perform operation and maintenance management on the target data. The target data can be obtained based on the collected data of the target node to be queried among the at least one market nodes received by the middleware, and the target data can also be obtained based on the collected data respectively corresponding to the at least one market nodes received by the middleware.

[0054] In the embodiments of the present invention, by performing data collection operations for each of the at least one market nodes through the data collection end, obtaining collected data, and pushing the collected data to the middleware, it is possible to achieve the unified collection of the collected data corresponding to the at least one market nodes respectively corresponding to the at least one data collection ends, thereby further realizing the unified operation and maintenance management of the target data.

[0055] Figure 4It is a structural block diagram of another data operation and maintenance management system provided by an embodiment of the present invention. This embodiment is optimized based on the above technical solutions. In this embodiment, optionally, the data operation and maintenance management system includes: a market quotation decoding end; wherein, the market quotation decoding end is used to read and collect data from the middleware, decode the collected data to obtain decoded data, and obtain target data according to the decoded data.

[0056] Among them, the explanations of the same or corresponding terms in the above embodiments will not be elaborated here.

[0057] Specifically, referring to Figure 4 , the data operation and maintenance management system of this embodiment includes: a data operation and maintenance management platform 210, a middleware 220, at least one data collection end 230, and a market quotation decoding end 240; wherein,

[0058] For each data collection end 230, the data collection end 230 is used to perform data collection operations on the market quotation nodes to obtain collected data, and push the collected data to the middleware 220, wherein the data collection end 230 is deployed in association with the market quotation nodes;

[0059] The market quotation decoding end 240 is used to read the collected data from the middleware 220, decode the collected data to obtain decoded data, and obtain target data according to the decoded data;

[0060] The data operation and maintenance management platform 210 is used to query the target data and perform operation and maintenance management on the target data, wherein the target data is obtained according to the collected data received by the middleware 220.

[0061] Among them, the market quotation decoding end 240 can be understood as the end for decoding the collected data; the market quotation decoding end 240 can exist in the form of a market quotation decoding client; in the case where the middleware 220 is a kafka message middleware, the market quotation decoding end 240 can exist as a kafka consumer; the number of market quotation decoding ends 240 can be one or more. In the case where the number of market quotation decoding ends 240 is multiple, the consumption efficiency can be improved through multiple market quotation decoding ends 240.

[0062] The decoded data can be understood as the data obtained by decoding the collected data read from the middleware 220.

[0063] In the embodiment of the present invention, the market quotation decoding end 240 can decode the collected data read from the middleware 220 to obtain decoded data, and obtain target data according to the decoded data.

[0064] It should be noted that decoding the collected data may include at least one of data decoding, data analysis, information extraction, data parsing, format conversion, etc. of the collected data. For example, for each type of data in different data types of the collected data, according to the data type, data decoding, data analysis, data extraction, data parsing or format conversion can be performed on the type of data to obtain decoded data; for another example, format conversion can be performed on the collected data so that the decoded data is in a unified format that can be managed by the data operation and maintenance management platform 210.

[0065] Exemplarily, referring to Figure 5 , the market quotation decoding end 240, as a kafka consumer, in order to improve efficiency, supports the deployment of multiple market quotation decoding ends 240; the market quotation decoding end 240 obtains registration data from the kafka message middleware, and stores the registration information obtained by parsing the registration data into the node list of the collection database for the data operation and maintenance management platform 210 to query and use; when the market quotation decoding end 240 obtains node logs from the kafka message middleware, through a pre-set log extraction specification, alarm information is extracted from the node logs, and the node logs and the alarm information are stored together in the log and alarm table of the collection database for the data operation and maintenance management platform 210 to monitor and use; when the market quotation client obtains real-time market quotation data from the kafka message middleware, according to the market quotation market identifier corresponding to the market quotation data, the market quotation data is decoded, and the decoded data including details such as market quotation snapshots, tick-by-tick, and market status is stored in the market quotation table of the collection database for the data operation and maintenance management platform 210 to display and use; it should be noted that considering that the amount of data stored in the collection database may become larger over time, the historical data stored in the collection database can be migrated for the data operation and maintenance management platform 210 to perform historical market quotation data playback.

[0066] The technical solution of the embodiment of the present invention reads the collected data from the middleware through the market quotation decoding end, decodes the collected data to obtain decoded data, and can obtain target data according to the decoded data, so that the target data obtained according to the decoded data can be applied to be managed by the data operation and maintenance management platform.

[0067] An alternative technical solution is that the collected data includes market quotation data, and the data collection end includes a market quotation data obtaining module and a market quotation data pushing module; among them, the market quotation data obtaining module is used to perform data collection operations on the market quotation node through the market quotation port corresponding to the market quotation node to obtain market quotation data; the market quotation data pushing module is used to push the market quotation data to the middleware.

[0068] Among them, the market quotation data can be understood as data related to the market quotation; the market quotation data can be, for example, trading information and other data of various assets in the market quotation market.

[0069] The market data acquisition module can be understood as a module for acquiring market data.

[0070] The market data push module can be understood as a module for pushing market data to the middleware.

[0071] The market port can be understood as a port that enables the market data acquisition module to perform data collection operations on market nodes; each market port corresponding to each data collection end can use the unified interface specification of the exchange, so as to more effectively implement the unified operation and maintenance management of target data.

[0072] In the embodiment of the present invention, the market data acquisition module can perform data collection operations on market nodes through the market ports corresponding to the market nodes to obtain market data, and the market data push module can push the market data to the middleware.

[0073] In the embodiment of the present invention, the number of market ports can be at least one. For example, in the case where the market nodes include multiple market markets, each market market can respectively correspond to a market port. Among them, the market market can be understood as a market related to the market such as a financial market. The market market can be, for example, a place market related to the market, or can be an overall market divided by regions such as Region A or Region B.

[0074] In the embodiment of the present invention, the node distribution service port configuration of the market node can be pre-configured as the market port and configured to the data collection end, so as to perform data collection operations on the market node through the market port corresponding to the market node.

[0075] In the embodiment of the present invention, when the market node is the primary node, the market data acquisition module can perform real-time data collection operations on the market node through the market port corresponding to the market node to obtain market data, so as to realize the real-time collection of market data; when the market node is not the primary node, market data can not be collected or data collection operations on the market node can be periodically performed at preset time intervals.

[0076] In the embodiment of the present invention, by performing data collection operations on the market node through the market port corresponding to the market node to obtain market data, relatively accurate market data collection can be realized.

[0077] Based on the above solution, in another alternative technical solution, the market data push module is specifically used to push market data to the middleware through the market topic corresponding to the market data; the market decoding end is specifically used to read market data from the middleware by subscribing to the market topic, and decode the market data according to the market topic to obtain decoded data, so as to obtain target data according to the decoded data.

[0078] Among them, the market theme can be understood as the theme corresponding to the market data; for example, the market theme can be the theme corresponding to the market category of the market data, that is, the market theme can represent the market category of the market data.

[0079] It can be understood that the middleware can be middleware adopting the Publish / Subscribe Pattern (pub / sub). Therefore, the market data push module can push the market data to the middleware through the market theme corresponding to the market data.

[0080] In the embodiment of the present invention, the market decoding end can read the market data from the middleware by subscribing to the market theme, and decode the market data according to the market theme to obtain decoded data. For example, the first decoding method corresponding to the market theme can be determined according to the market theme, and the market data can be decoded according to the first decoding method to obtain decoded data, so as to obtain target data according to the decoded data.

[0081] It should be noted that the market data can also be decoded according to the market identifier of the market corresponding to the market data to obtain decoded data.

[0082] In the embodiment of the present invention, by pushing the market data to the middleware through the market data push module through the market theme corresponding to the market data, the interaction logic of the market data push can be simplified and more reliable. The market decoding end can also read the market data from the middleware by subscribing to the market theme and decode the market data according to the market theme, so as to improve the accuracy of the obtained decoded data.

[0083] Another optional technical solution is that the collected data includes node logs, and the data collection end includes a node log obtaining module and a node log pushing module; among them, the node log obtaining module is used to perform data collection operations on the market nodes according to the configured log collection specifications to obtain node logs; the node log pushing module is used to push the node logs to the middleware; the market decoding end is specifically used to read the node logs from the middleware, extract information from the node logs to decode the collected data, obtain alarm information, and use the alarm information as decoded data, so as to obtain target data according to the node logs and the decoded data.

[0084] Among them, the node logs can be understood as the logs of the market nodes; the node logs can include alarm information.

[0085] The node log obtaining module can be understood as a module for obtaining node logs.

[0086] The node log push module can be understood as a module for pushing node logs to the middleware.

[0087] The log collection specification can be understood as a specification for collecting logs; the log collection specification can include, for example, at least one of a log collection directory, a log level keyword, and the like.

[0088] In an embodiment of the present invention, after installing the data collection end, the log collection specification of the data collection end can be configured.

[0089] The alarm information can be understood as information reflecting the abnormal situation of the market node.

[0090] In an embodiment of the present invention, the node log acquisition module can perform data collection operations on the market node through the configured log collection specification to obtain node logs; the node log push module can push the node logs to the middleware; the market decoding end can read the node logs from the middleware, extract information from the node logs to obtain alarm information, and use the alarm information as decoded data to obtain target data based on the node logs and the decoded data.

[0091] In an embodiment of the present invention, when the target data is obtained based on the node logs and the decoded data, the target data can be operationally and maintained, for example, through a unified display screen associated with the data operation and maintenance management platform, to display the node logs in real time and perform real-time abnormal alarms.

[0092] Compared with the related solutions where there is a problem of missing alarm information, especially when there is missing or overly delayed downstream data anomalies and market operation and maintenance personnel cannot receive alarm information and cannot quickly locate alarm problems, the solution of the embodiment of the present invention can avoid missing alarm information by collecting node logs and pushing them to the middleware, and enabling the market decoding end to extract alarm information from the node logs read from the middleware, thereby facilitating quick location of alarm problems.

[0093] On the basis of the above solution, in another optional technical solution, the node log push module is specifically used to push the node logs to the middleware through a log topic corresponding to the node logs; the market decoding end is further specifically used to read the node logs from the middleware by subscribing to the log topic, extract information from the node logs according to the log topic to obtain alarm information, and use the alarm information as decoded data to obtain target data based on the node logs and the decoded data.

[0094] Among them, the log topic is a topic corresponding to the node logs; the log topic can be, for example, a topic corresponding to the log category of the node logs, that is, the log topic can represent the log category of the node logs.

[0095] It can be understood that the middleware can be middleware adopting the publish / subscribe mode. Therefore, the node log push module can push the node log to the middleware through the log topic corresponding to the node log.

[0096] In the embodiment of the present invention, the market quotation decoding end can read the node log from the middleware by subscribing to the log topic, and perform information extraction on the node log according to the log topic to obtain alarm information. For example, the extraction method corresponding to the log topic can be determined according to the log topic, and information extraction is performed on the log topic according to the extraction method to obtain alarm information, and the alarm information is used as the decoded data to obtain the target data according to the node log and the decoded data.

[0097] In the embodiment of the present invention, by pushing the node log to the middleware through the node log push module through the log topic corresponding to the node log, the interaction logic of the node log push can be simplified and more reliable. Also, the market quotation decoding end can read the node log from the middleware by subscribing to the log topic, and perform information extraction on the node log according to the log topic to improve the accuracy of the obtained alarm information.

[0098] Another optional technical solution is that the data collection end is further configured to perform a data collection operation on the market quotation node through the market quotation port corresponding to the market quotation node to obtain registration data and push the registration data to the middleware when the market quotation node is not registered; the market quotation decoding end is further configured to read the registration data from the middleware and decode the registration data to obtain registration information; the data operation and maintenance management platform is further configured to query the registration information and perform a registration operation on the market quotation node according to the registration information.

[0099] Among them, the registration data can be understood as the data for registering the market quotation node that has not been decoded. Correspondingly, the registration information can be understood as the information for registering the market quotation node that has been decoded; the registration information can be, for example, the registration instruction of the market quotation node.

[0100] In the embodiment of the present invention, the data collection end can perform a data collection operation on the market quotation node through the market quotation port corresponding to the market quotation node to obtain registration data and push the registration data to the middleware when the market quotation node is not registered; the market quotation decoding end can read the registration data from the middleware and decode the registration data to obtain registration information; the data operation and maintenance management platform can query the registration information and perform a registration operation on the market quotation node according to the registration information.

[0101] In the solution of the embodiment of the present invention, by collecting the registration data and pushing it to the middleware, the market quotation decoding end reads the registration data from the middleware and decodes it to obtain the registration information. Furthermore, the data operation and maintenance management platform performs a registration operation on the market quotation node according to the queried registration information, realizing the registration of the market quotation node, so that the data operation and maintenance management platform can manage the target data related to the registered market quotation node.

[0102] On the basis of the above solution, in an alternative technical solution, the data collection end is further specifically configured to, when the market quotation node is not registered, perform a data collection operation on the market quotation node through the market quotation port corresponding to the market quotation node to obtain the registration data, and push the registration data to the middleware through the registration topic corresponding to the registration data; the market quotation decoding end is further specifically configured to read the registration data from the middleware by subscribing to the registration topic, and decode the registration data according to the registration topic to obtain the registration information.

[0103] Wherein, the registration topic is the topic corresponding to the registration data; the registration topic can be, for example, the topic corresponding to the node category of the market quotation node corresponding to the registration data, that is, the market quotation topic can represent the node category of the market quotation node.

[0104] It can be understood that the middleware can be a middleware adopting the publish / subscribe mode. Therefore, when the market quotation node is not registered, the data collection end can perform a data collection operation on the market quotation node through the market quotation port corresponding to the market quotation node to obtain the registration data, and push the registration data to the middleware through the registration topic.

[0105] In the embodiment of the present invention, the market quotation decoding end can be used to read the registration data from the middleware by subscribing to the registration topic, and decode the registration data according to the registration topic to obtain the registration information. For example, it can determine the second decoding method corresponding to the registration topic according to the registration topic, decode the registration data according to the second decoding method to obtain the registration information, and decode the registration data according to the registration topic to obtain the registration information.

[0106] In the embodiment of the present invention, by pushing the registration data to the middleware through the registration topic corresponding to the registration data by the data collection end, the interaction logic of the push of the registration data can be simplified and made more reliable. Moreover, the market quotation decoding end can read the registration data from the middleware by subscribing to the registration topic and decode the registration data according to the registration topic, improving the accuracy of the obtained registration information.

[0107] Figure 6It is a structural block diagram of another data operation and maintenance management system provided by an embodiment of the present invention. This embodiment is optimized based on the above technical solutions. In this embodiment, optionally, the data operation and maintenance management system further includes: a collection database; wherein, the market quotation decoding end is further configured to store the decoded data into the collection database; the data operation and maintenance management platform is specifically configured to query target data from the collection database and perform operation and maintenance management on the target data.

[0108] Among them, the explanations of the same or corresponding terms as those in the above embodiments will not be elaborated here.

[0109] Specifically, refer to Figure 6 , the data operation and maintenance management system of this embodiment includes: a data operation and maintenance management platform 310, a middleware 320, at least one data collection end 330, a market quotation decoding end 340, and a collection database 350; wherein,

[0110] For each data collection end 330, the data collection end 330 is configured to perform a data collection operation on the market quotation node to obtain collection data, and push the collection data to the middleware 320, wherein the data collection end 330 is deployed in association with the market quotation node;

[0111] The market quotation decoding end 340 is configured to read the collection data from the middleware 320 and decode the collection data to obtain decoded data, so as to obtain target data according to the decoded data;

[0112] The market quotation decoding end 340 is further configured to store the decoded data into the collection database 350;

[0113] The data operation and maintenance management platform 310 is specifically configured to query target data from the collection database 350 and perform operation and maintenance management on the target data, wherein the target data is obtained according to the collection data received by the middleware 320.

[0114] Among them, the collection database 350 can be understood as a database for storing collection data; since the collection data may be collected and stored in the collection database 350 in real time, and the data operation and maintenance management platform 310 may query the target data from the collection database 350 in real time, that is, the data throughput of the collection database 350 may be relatively large and the real-time requirement for data throughput is relatively high. Therefore, the collection database 350 can be a memory database or other databases that can meet the requirements of the data operation and maintenance management platform 310 and the middleware 320.

[0115] In an embodiment of the present invention, a market quotation decoding terminal 340 may be pre-installed, and database configuration and middleware 320 configuration may be performed on the market quotation decoding terminal 340. For example, market quotation database information and Kafka service information may be configured for the market quotation decoding terminal 340, so that the market quotation decoding terminal 340 can read and collect data from the middleware 320 and store the collected data in the collection database 350.

[0116] In an embodiment of the present invention, database configuration may be pre-performed on the data operation and maintenance management platform 310, so that the data operation and maintenance management platform 310 can query target data from the collection database 350.

[0117] In an embodiment of the present invention, the market quotation decoding terminal 340 may store the collected data in the collection database 350. For example, when the collected data includes market quotation data, the market quotation decoding terminal 340 may store the market quotation data in the market quotation database; for another example, when the collected data includes node logs, the market quotation decoding terminal 340 may store the node logs in the corresponding log table in the market quotation database. It should be noted that when the market quotation decoding terminal 340 obtains registration information, the market quotation decoding terminal 340 may store the registration information in the node list in the market quotation database.

[0118] In an embodiment of the present invention, the data operation and maintenance management platform 310 may query target data from the collection database 350 and perform operation and maintenance management on the target data.

[0119] In an embodiment of the present invention, the data operation and maintenance management system may also implement historical recording and playback to quickly query historical collection data for a specified historical time period or historical time point. For example, the collection database 350 may store each piece of collected data deposited by the historical market quotation decoding terminal 340. The data operation and maintenance management platform 310 may query the collected data corresponding to the specified historical time period or historical time point from the collection database 350 according to the specified historical time period or historical time point as historical collection data, and perform operation and maintenance management such as displaying the historical collection data.

[0120] In an embodiment of the present invention, by using the market quotation decoding terminal to store the decoded data in the collection database and the data operation and maintenance management platform to query the target data from the collection database, it is possible to avoid the problem of loss of decoded data due to limited data storage capacity of the market quotation decoding terminal or other terminals when the amount of decoded data decoded by the market quotation decoding terminal is large.

[0121] To better understand the technical solutions of the above embodiments of the present invention, an optional example is provided here. Exemplarily, refer to Figure 7, the data operation and maintenance management system includes: a data operation and maintenance management platform, a collection database, a market quotation decoding terminal, a kafka message middleware, and a data collection terminal; the collected data can be distributed from the market data gateway (Market Data Gateway / Vendor Data Engine, MDGW / VDUE) to the data collection terminal for secondary or multi-level forwarding; a data collection terminal is deployed on each machine with a market quotation node, and the collection service on each data collection terminal performs real-time data collection operations on the managed market quotation nodes to obtain the collected data, and pushes the collected data to the kafka message middleware. The market quotation decoding terminal, as a kafka consumer, reads the collected data from the kafka message middleware, decodes the collected data, and stores the obtained decoded data in the collection database. The data operation and maintenance management platform queries the target data including node information, node logs, and market quotation data, and classifies and displays the target data through a display screen. This classified display is to display according to the classification of the target data. For example, the node information, node logs, and market quotation data are respectively used as one category and displayed in different areas of the display screen, and historical market quotations can also be supported for display.

[0122] The above specific implementation manners do not constitute a limitation to the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A data operation and maintenance management system, characterized in that, including: a data operation and maintenance management platform, middleware, and at least one data collection end; wherein, for each of the data collection ends, the data collection end is configured to perform a data collection operation on a market situation node to obtain collected data, and push the collected data to the middleware, wherein the data collection end is deployed in association with the market situation node; the data operation and maintenance management platform is configured to query target data and perform operation and maintenance management on the target data, wherein the target data is obtained based on the collected data received by the middleware.

2. The system according to claim 1, wherein further including: a market situation decoding end; wherein, the market situation decoding end is configured to read the collected data from the middleware and decode the collected data to obtain decoded data, so as to obtain the target data based on the decoded data.

3. The system according to claim 2, wherein the collected data includes market situation data, and the data collection end includes a market situation data obtaining module and a market situation data pushing module; wherein, the market situation data obtaining module is configured to perform a data collection operation on a market situation node through a market situation port corresponding to the market situation node to obtain the market situation data; the market situation data pushing module is configured to push the market situation data to the middleware.

4. The system according to claim 3, wherein: the market situation data pushing module is specifically configured to push the market situation data to the middleware through a market situation topic corresponding to the market situation data; the market situation decoding end is specifically configured to read the market situation data from the middleware by subscribing to the market situation topic, and decode the market situation data according to the market situation topic to obtain decoded data, so as to obtain the target data based on the decoded data.

5. The system according to claim 2, wherein the collected data includes node logs, and the data collection end includes a node log obtaining module and a node log pushing module; wherein, the node log obtaining module is configured to perform a data collection operation on a market situation node through a configured log collection specification to obtain the node logs; the node log pushing module is configured to push the node logs to the middleware; the market situation decoding end is specifically configured to read the node logs from the middleware, perform information extraction on the node logs to decode the collected data, obtain alarm information, and use the alarm information as the decoded data, so as to obtain the target data based on the node logs and the decoded data.

6. The system according to claim 5, wherein: the node log pushing module is specifically configured to push the node logs to the middleware through a log topic corresponding to the node logs; the market situation decoding end is further specifically configured to read the node logs from the middleware by subscribing to the log topic, perform information extraction on the node logs according to the log topic to obtain alarm information, and use the alarm information as the decoded data, so as to obtain the target data based on the node logs and the decoded data.

7. The system according to claim 2, wherein: The data collection end is further configured to, when the market node is not registered, perform a data collection operation on the market node through the market port corresponding to the market node to obtain registration data, and push the registration data to the middleware; The market decoding end is further configured to read the registration data from the middleware and decode the registration data to obtain registration information; The data operation and maintenance management platform is further configured to query the registration information and perform a registration operation on the market node according to the registration information.

8. The system according to claim 7, wherein: The data collection end is further specifically configured to, when the market node is not registered, perform a data collection operation on the market node through the market port corresponding to the market node to obtain registration data, and push the registration data to the middleware through the registration topic corresponding to the registration data; The market decoding end is further specifically configured to read the registration data from the middleware by subscribing to the registration topic, and decode the registration data according to the registration topic to obtain registration information.

9. The system according to claim 2, wherein Further included: A collection database; wherein, The market decoding end is further configured to store the decoded data in the collection database; The data operation and maintenance management platform is specifically configured to query target data from the collection database and perform operation and maintenance management on the target data.

10. The system according to claim 1, wherein The number of market nodes corresponding to the data collection end is at least one; The data collection end is specifically configured to perform a data collection operation on each of at least one market node to obtain collection data, and push the collection data to the middleware.

Images