Data processing method and device, data processing system

Abstract

This application discloses a data processing method, apparatus, and system. The method includes: receiving a notification message indicating that a distributed server has written a target file, the target file including multiple time-series index data from multiple communication devices, and the notification message carrying multiple codes corresponding to the multiple time-series index data; obtaining an index information table, determining the type of time-series index data corresponding to each code in the index information table; determining a parsing method for parsing the target file based on the type of each time-series index data; parsing the target file according to the parsing method and the index information table, obtaining parsing results, and storing the parsing results in a database in chronological order. This application solves the technical problem of slow file parsing speed and inability to guarantee the timeliness of time-series data caused by the manual intervention method used in related technologies to determine the file for parsing and storing time-series data.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and more specifically, to a data processing method and apparatus, and a data processing system. Background Technology

[0002] The daily operation of operator network equipment requires a large amount of real-time communication equipment operation data to support business scenarios such as equipment inspection, fault analysis, and visualization. Related technologies decouple data acquisition and data parsing, collecting large amounts of communication data simultaneously from various types of devices with different functions within a single acquisition cycle; and manually determining the parsing method based on the data type results in slow parsing speeds and an inability to guarantee the timeliness of time-series data.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a data processing method and apparatus, and a data processing system, to at least solve the technical problem that the slow file parsing speed caused by the method of manually determining the file for parsing and storing time-series data in related technologies makes it impossible to guarantee the timeliness of time-series data.

[0005] According to one aspect of the embodiments of this application, a data processing method is provided, comprising: receiving a notification message, wherein the notification message indicates that a distributed server has written a target file, the target file including multiple time-series indicator data of multiple communication devices, and the notification message carrying multiple codes corresponding to the multiple time-series indicator data; obtaining an indicator information table, determining the type of time-series indicator data corresponding to each code in the indicator information table; determining a parsing method for parsing the target file based on the type of each time-series indicator data; parsing the target file according to the parsing method and the indicator information table, obtaining a parsing result, and storing the parsing result in a database in chronological order, wherein the chronological order is determined based on the generation time of the multiple time-series indicator data in the target file.

[0006] Optionally, the indicator information table records multiple codes, model types, model contents, model types associated with each code, and model contents associated with each code for various types of time-series indicator data. The model type indicates the type of data model used to parse the time-series indicator data corresponding to each code, and the model contents indicate the parsing result of the time-series indicator data corresponding to each code. Determining the type of time-series indicator data corresponding to each code in the indicator information table includes: determining the target code in the indicator information table, where the target code is the same as the code of the time-series indicator data in the target file; and determining the model type associated with the target code in the indicator information table, where the model type is the type of time-series data corresponding to the target code, and the model type includes: single-indicator model and multi-indicator model.

[0007] Optionally, the target file is parsed according to the parsing method and the indicator information table, including: if the target file contains only one type of time series indicator data, the single indicator model is used to parse the target file; the first type of model content associated with the single indicator model is determined in the indicator information table, wherein the first type of model content includes: indicator code and port; the target file is parsed using the single indicator model to obtain a first type of parsing result that conforms to the first type of model content, wherein the first type of parsing result includes: the code of each time series indicator data in the target file and the port corresponding to each time series indicator data; if the target file contains multiple types of time series indicator data, the multi-indicator model is used to parse the target file.

[0008] Optionally, the target file is parsed using a multi-indicator model, including: determining the data level of the time-series indicator data; when the data level of the time-series indicator data is 0, determining the second type of model content associated with the multi-indicator model in the indicator information table, wherein the second type of model content includes: encoding; parsing the target file using the multi-indicator model to obtain a second type of parsing result that conforms to the second type of model content, wherein the second type of parsing result includes: the encoding of each time-series indicator data.

[0009] Optionally, parsing the target file using the multi-indicator model further includes: if the data level of the time-series indicator data is greater than 0 levels, determining the third type of model content associated with the multi-indicator model in the indicator information table, wherein the third type of model content includes: port, virtual local area network identifier and encoding; parsing the target file using the multi-indicator model to obtain a third type of parsing result that conforms to the second type of model content, wherein the third type of parsing result includes: a port corresponding to multiple time-series indicator data in the target file, a virtual local area network identifier corresponding to each time-series indicator data in the multiple time-series indicator data, and the encoding of each time-series indicator data.

[0010] Optionally, the data processing method further includes: monitoring the data processing process and generating notification information when an anomaly is detected in the data processing; identifying the process that is experiencing the anomaly, wherein the process includes: a notification message process, a file writing process, a file parsing process, and a data storage process; and determining the target object to receive the notification message based on the process that is experiencing the anomaly.

[0011] Optionally, the target object for receiving the notification message is determined based on the process in which the exception occurred. This includes: if the process in which the exception occurred is one of the following: a notification message process, a first type of exception in a file writing process, or a second type of exception in a file parsing process, the notification message is sent to a first target object, where the first target object is the platform sending the target file; the first type of exception includes: abnormal file size, missing file, and empty file content; the second type of exception includes: absence of time series indicator data and missing parsing model. If the process in which the exception occurred is one of the following: a third type of exception in a file writing process or a data entry process, the notification message is sent to a second target object, where the second target object is the platform processing the target file; the third type of exception includes: file download exception and distributed server connection exception. If the process in which the exception occurred is a fourth type of exception in a file parsing process, the notification message is sent to both the first target object and the second target object, where the fourth type of exception is an exception that occurred during file parsing.

[0012] Optionally, the data processing method further includes: receiving update information, wherein the update information is used to update the type of time series indicator data and the type of data model for parsing time series indicator data, and the update method corresponding to the update information includes full update and incremental update.

[0013] According to another aspect of the embodiments of this application, a data processing system is also provided, including: a distributed server, a model management module, a file parsing module, and a monitoring and alarm module. The model management module is used to receive update information, update the type of time-series data and the data model for parsing the time-series data according to the update information, and record the type of time-series data and the data model for parsing the time-series data in an indicator information table. The distributed server is used to receive a target file to be written and send the target file to the file parsing module. The target file includes multiple time-series indicator data from multiple communication devices. The file parsing module is used to receive a notification message and the target file. The notification message indicates that the distributed server has been written to the target file, and the notification message includes multiple time-series indicator data... The system includes a coding module; a file parsing module, which also retrieves an indicator information table, determines the type of time-series indicator data corresponding to each coding in the indicator information table, and determines the parsing method for parsing the target file based on the type of each time-series indicator data; the file parsing module also parses the target file according to the parsing method and the indicator information table, obtains the parsing results of the target file, and stores the parsing results in the database in chronological order, where the chronological order is determined based on the generation time of multiple time-series indicator data in the target file; and a monitoring and alarm module, which monitors the running processes of the distributed server, model management module, and file parsing module, generates notification information when abnormal data processing is detected, and determines the target object to receive the notification message based on the process where the abnormality occurred.

[0014] According to another aspect of the embodiments of this application, a data processing apparatus is also provided, comprising: a receiving module, configured to receive a notification message, wherein the notification message indicates that a distributed server has been written to a target file, the target file including multiple time-series index data of multiple communication devices, and the notification message including multiple codes corresponding to the multiple time-series index data; an obtaining module, configured to obtain an index information table, and determine the type of time-series index data corresponding to each code in the index information table; a determining module, configured to determine a parsing method for parsing the target file based on the type of each time-series index data; and a parsing module, configured to parse the target file according to the parsing method and the index information table, obtain parsing results, and store the parsing results in a database in chronological order, wherein the chronological order is determined based on the generation time of the time-series data in the target file.

[0015] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, which stores a computer program, wherein the device where the non-volatile storage medium is located executes the above-described data processing method by running the computer program.

[0016] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to execute the above-described data processing method through the computer program.

[0017] In this embodiment, a notification message is received, indicating that a distributed server has written a target file. The target file includes multiple time-series indicator data from multiple communication devices, and the notification message carries multiple codes corresponding to the multiple time-series indicator data. An indicator information table is obtained, and the type of time-series indicator data corresponding to each code is determined within the indicator information table. A parsing method for parsing the target file is determined based on the type of each time-series indicator data. The target file is parsed according to the parsing method and the indicator information table to obtain the parsing result, which is then stored in a database in chronological order. The chronological order is determined based on the generation time of the multiple time-series indicator data in the target file, through a decentralized distribution. The Ceph server stores files concurrently, ensuring the timeliness of data from communication equipment. By automatically updating indicator information and parsing models, it automatically identifies and parses the indicator content in different data collection tasks, improving data parsing efficiency. Through full-process monitoring of data processing, it promptly detects abnormal data processing and sends notification messages. This achieves the technical effects of improving data parsing speed, ensuring the timeliness of time-series data, and quickly locating faults. It also solves the technical problem of slow file parsing speed and inability to guarantee the timeliness of time-series data caused by manually determining the files storing time-series data in related technologies. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0019] Figure 1 This is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a data processing method according to an embodiment of this application;

[0020] Figure 2 This is a flowchart of a data processing method according to an embodiment of this application;

[0021] Figure 3 This is a flowchart illustrating an interruption alarm when sending a notification message, according to an embodiment of this application.

[0022] Figure 4 This is a schematic diagram of a data processing system according to an embodiment of this application;

[0023] Figure 5 This is a flowchart of a data processing system according to an embodiment of this application;

[0024] Figure 6 This is a structural diagram of a data processing apparatus according to an embodiment of this application. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] To better understand the embodiments of this application, the technical terms involved in the embodiments of this application are explained below:

[0028] Timing metrics data: Metrics data that measure the timing performance of a communication system, such as transmission delay and processing delay.

[0029] In related technologies, data acquisition and data parsing are decoupled, and periodic batch files are used to transmit device operation data. However, since data acquisition and parsing involve multiple systems, these technologies suffer from the following problems when acquiring and parsing communication device operation data: 1) Manual intervention is required to parse data from multiple types of communication devices simultaneously, as the indicator types are not fixed, making the manual intervention process cumbersome; 2) Different indicators have different acquisition instructions and significantly different data formats, resulting in long data parsing times; 3) The number of files in a single acquisition cycle can reach tens of thousands, and using periodic batch files to transmit device operation data cannot guarantee data timeliness; 4) The entire data parsing process includes multiple stages such as data reading, model parsing, and data storage. If data anomalies or missing data occur, it is impossible to quickly locate the fault and perform fault repair. To solve the above problems, this application provides relevant solutions in its embodiments, which are described in detail below.

[0030] According to an embodiment of this application, a method embodiment for processing data is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0031] The methods and embodiments provided in this application can be executed on mobile terminals, computer terminals, or similar computing devices. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing a data processing method is shown. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0032] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0033] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the data processing method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the aforementioned data processing method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0034] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0035] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0036] Under the above operating environment, this application provides a data processing method. Figure 2 This is a flowchart of a data processing method provided according to an embodiment of this application, such as... Figure 2 As shown, the method includes the following steps:

[0037] Step S202: Receive a notification message, wherein the notification message indicates that the distributed server has written to the target file, the target file includes multiple time-series index data of multiple communication devices, and the notification message carries multiple codes corresponding to the multiple time-series index data.

[0038] The method provided in this application embodiment is used to process time-series indicator data in the operation data of various communication devices of operators. In step S202, after the time-series indicator data is written in batches to the distributed server (Ceph server) in the form of (target) files, the data sharing platform that processes the data will receive a file writing notification (i.e., notification information). The file writing notification (i.e., notification information) includes a virtual instruction code for recording each time-series indicator data in the (target) file written this time. Each type of time-series indicator data has a unique virtual instruction code. Therefore, the data sharing platform can determine the type of time-series indicator data and other relevant information of the time-series indicator data through the virtual instruction code.

[0039] In addition, the above-mentioned file writing notification (i.e., notification information) also includes: the name of the (target) file, the path of the (target) file, the generation time of the (target) file, the number of bytes of the (target) file, the identifier of the acquisition device (sampling source ID) for each time series index data in the (target) file, the identifier of the device that generated each time series index data (device ID), the type of device that generated each time series index data, and other relevant information about the time series index data in the (target) file.

[0040] Step S204: Obtain the indicator information table and determine the type of time series indicator data corresponding to each code among multiple codes in the indicator information table.

[0041] In step S204, after receiving the file write notification (i.e., notification information), the data sharing platform obtains the indicator information table stored in the sharing platform, queries the indicator list carried in the file write notification (i.e., notification information) in the indicator information table to determine the type of time series indicator data in the (target) file.

[0042] The aforementioned indicator information table is used to record relevant information of time-series indicator data received by the data sharing platform in the historical process. Optionally, the indicator information table is used to record multiple codes, model types, model contents, model types associated with each code among multiple types of time-series indicator data, and model contents associated with each code. The model type indicates the type of data model used to parse the time-series indicator data corresponding to each code, and the model content indicates the parsing result of the time-series indicator data corresponding to each code. Determining the type of time-series indicator data corresponding to each code in the indicator information table includes: determining the target code in the indicator information table, wherein the target code is the same as the code of the time-series indicator data in the target file; and determining the model type associated with the target code in the indicator information table, wherein the model type is the type of time-series data corresponding to the target code, and the model type includes: single-indicator model and multi-indicator model.

[0043] In this embodiment, the indicator information table records the virtual instruction code of time series indicator data, the type of data model for parsing time series indicator data, and the result (i.e., model content) that should be obtained from parsing time series indicator data in an associated form. Among them, the virtual instruction code of a type of time series indicator data is recorded only once, the type of data model associated with the virtual instruction code is the model type for parsing this type of time series indicator data (e.g., single indicator model, multi-indicator model), and the model content associated with the virtual instruction code is the fields that should be included in the parsing result obtained from parsing this type of time series indicator data (e.g., data format field, port field, etc.). Therefore, the data model of the file where the time series indicator data is parsed can be found in the indicator information table through the virtual instruction code of the time series indicator data. When determining the parsing method for the (target) file, the first step is to obtain the indicator information table stored in the data sharing platform. The encoding of the time series indicator data in the (target) file is then queried in the indicator information table. When an encoding identical to the encoding of the time series indicator data in the (target) file is found in the indicator information table (i.e., the target encoding), the data model associated with the target encoding is determined as the data model used to parse the (target) file. In this embodiment, the data model for parsing the file includes two types: a single-indicator model and a multi-indicator model.

[0044] Step S206: Determine the parsing method for parsing the target file based on the type of each time series index data.

[0045] In the method provided in this application embodiment, different parsing methods are provided for different types of time series index data. Therefore, after the type of time series index data is determined in step S204, the parsing method of the (target) file is determined in step S206 according to the type of time series index data.

[0046] Step S208: The target file is parsed according to the parsing method and the indicator information table to obtain the parsing results, and the parsing results are stored in the database in chronological order. The chronological order is determined based on the generation time of multiple time-series indicator data in the target file.

[0047] In step S208, the target file is parsed using a parsing method corresponding to the type of time series index data in the (target) file, and the parsed data (i.e., parsing results) is stored in a shared platform database such as an open-source time series database (OpenTime Series Database, OpenTSDB) or a key-value database (Redis) in order of the generation time of the time series index data corresponding to the parsing results from earliest to latest (i.e., time order).

[0048] According to an optional embodiment of this application, parsing a target file based on a parsing method and an indicator information table includes: if the target file contains only one type of time-series indicator data, determining to use a single-indicator model to parse the target file; determining a first type of model content associated with the single-indicator model in the indicator information table, wherein the first type of model content includes: indicator code and port; parsing the target file using the single-indicator model to obtain a first type of parsing result that conforms to the first type of model content, wherein the first type of parsing result includes: the code of each time-series indicator data in the target file and the port corresponding to each time-series indicator data; if the target file contains multiple types of time-series indicator data, determining to use a multi-indicator model to parse the target file.

[0049] In the method provided in this application embodiment, data is collected only through single indicator command or multi-indicator command. The single indicator command collects the same type of time series indicator data from multiple ports, while the multi-indicator command collects multiple types of time series indicator data from one port. In this embodiment, determining the parsing method based on the time series index data in the (target) file includes the following two cases: If all the time series index data in the (target) file are of the same type, it indicates that the time series index data in the (target) file is time series index data collected through single index instructions. At the same time, the data parsing model associated with the virtual instruction code of the data collected by the single index instruction can be found in the index information table as the single index model. Therefore, it is determined that the single index model is used to parse the (target) file. Before using the single index model to parse the (target) file, the content of the (first type) model associated with the single index model can also be found in the index information table, which includes an index code field and a port field. Therefore, the parsing result obtained by using the single index model to parse the (target) file includes the code of each time series index data and the port number (such as 8080) and / or port name of the port that transmits each time series index data when collecting the time series index data. If the time series indicator data in the (target) file is of a different type, it means that the time series indicator data in the (target) file is time series indicator data collected through multi-indicator instructions. At the same time, it can be found in the indicator information table that the data parsing model associated with the virtual instruction code of the data collected by the multi-indicator instructions is a multi-indicator model. Therefore, it is determined that the multi-indicator model will be used to parse the (target) file.

[0050] Optionally, the target file is parsed using a multi-indicator model, including: determining the data level of the time-series indicator data; when the data level of the time-series indicator data is 0, determining the second type of model content associated with the multi-indicator model in the indicator information table, wherein the second type of model content includes: encoding; parsing the target file using the multi-indicator model to obtain a second type of parsing result that conforms to the second type of model content, wherein the second type of parsing result includes: the encoding of each time-series indicator data.

[0051] In this embodiment, when using a multi-indicator model to parse a (target) file composed of multiple types of time-series indicator data collected from the same port, different parsing methods are adopted depending on the data level of the time-series indicator data. Specifically, when the data level of the time-series indicator data is level 0, the (second type) model content associated with the multi-indicator model is an encoded field. Therefore, the parsing result obtained by using the multi-indicator model to parse the (target) file is the virtual instruction code of each time-series indicator data in the (target) file. In addition, since the time-series indicator data is a lightweight data format (JSON data) composed of key-value pairs, each virtual instruction code can also include the attributes of the time-series indicator data corresponding to the virtual instruction code. For example, it includes the memory usage rate and the central processing unit (CPU) usage rate of the device corresponding to the time-series indicator data of the virtual instruction code.

[0052] According to some optional embodiments of this application, parsing the target file using a multi-index model further includes: if the data level of the time-series index data is greater than 0 levels, determining the third type of model content associated with the multi-index model in the index information table, wherein the third type of model content includes: port, virtual local area network identifier and encoding; parsing the target file using the multi-index model to obtain a third type of parsing result that conforms to the second type of model content, wherein the third type of parsing result includes: a port corresponding to multiple time-series index data in the target file, a virtual local area network identifier corresponding to each time-series index data in the multiple time-series index data, and the encoding of each time-series index data.

[0053] In some embodiments, when the data level of the time series indicator data is greater than 0, the (third type) model content associated with the multi-indicator model includes: the virtual instruction code of each time series indicator data, the virtual local area network identifier (VLAN) of the device that generates each time series indicator data, and port-related information such as the port number (e.g., 0808) and / or port name of the port that transmits each time series indicator data during data acquisition. For example, if the data level of the time series indicator data is 3 layers, then when using the multi-indicator model to parse a (target) file composed of different types of time series indicator data collected from the same port, the parsing result includes 3 layers, where the first layer is the port name of the port that transmits the time series indicator data, the second layer is the virtual local area network identifier (VLAN number) and Internet Protocol Address (IP) address of the device that generates each time series indicator data, and the third layer is the virtual instruction code of each time series indicator data.

[0054] According to an optional embodiment of this application, the data processing method further includes: monitoring the data processing process and generating notification information when an anomaly is detected in the data processing; determining the process that is experiencing the anomaly, wherein the process includes: a notification message process, a file writing process, a file parsing process, and a data storage process; and determining the target object for receiving the notification message based on the process that is experiencing the anomaly.

[0055] The method provided in this application embodiment also performs real-time monitoring of data processing. In this embodiment, data processing includes: a notification message process for receiving file writing notifications (i.e., notification messages), a file writing process for writing (target) files in a distributed server (ceph server), and a data entry process for writing parsing results into a database. The above data processing processes are monitored in their entirety, and when an anomaly is detected, the process in which the anomaly occurs is determined. Based on the process in which the anomaly occurs, the platform (i.e., target object) that receives the alarm message (i.e., notification message) is determined.

[0056] Optionally, the method of determining the target object for receiving the notification message based on the process that experiences an anomaly, as mentioned in the previous embodiment, includes: when the process that experiences an anomaly is one of the following: a notification message process, a first type of anomaly in a file writing process, or a second type of anomaly in a file parsing process, the notification message is sent to a first target object, wherein the first target object is the platform that sends the target file; the first type of anomaly includes: abnormal file size, missing file, and empty file content; the second type of anomaly includes: absence of time series indicator data and missing parsing model; when the process that experiences an anomaly is one of the following: a third type of anomaly in a file writing process or a data entry process, the notification message is sent to a second target object, wherein the second target object is the platform that processes the target file; the third type of anomaly includes: file download anomaly and distributed server connection anomaly; when the process that experiences an anomaly is one of the following: a fourth type of anomaly in a file parsing process, the notification message is sent to both the first target object and the second target object, wherein the fourth type of anomaly is an anomaly in file parsing.

[0057] In the application embodiment, the platform (i.e., the target object) receiving the alarm message (i.e., the notification message) is determined according to the following rules based on the process where the anomaly occurs. When the abnormal process is any of the following: notification message process anomaly (e.g., notification message transmission interruption, notification message format anomaly, etc.), file size anomaly, missing file, or empty file content in the file writing process (i.e., first type of anomaly), or the absence of time series indicator data or missing parsing model in the parsing result of the file parsing process (i.e., third type of anomaly), the alarm information (i.e., the notification message) is sent to the acquisition and control platform (i.e., the first target object) that sent the (target) file. When the abnormal process is any of the following: distributed server (ceph server) connection anomaly, file download anomaly (i.e., fourth type of anomaly), or data entry process anomaly (e.g., OpenTSDB connection anomaly, OpenTSDB insertion failure, Redis connection anomaly, Redis insertion failure, etc.), the alarm information (i.e., the notification message) is sent to the data sharing platform (i.e., the second target object) that processed the (target) file. When the abnormal process is a parsing process error in the file parsing process (i.e., the fourth type of abnormality), the alarm information (i.e. the notification message) will be sent to both the acquisition and control platform (i.e. the first target object) and the data sharing platform (i.e. the second target object). Figure 3 This is a flowchart illustrating the interruption of alarm notification message transmission, such as... Figure 3 As shown, starting from receiving a file write notification (i.e., notification message) from the data sharing platform, the status of the logs (ES logs) generated during the data transmission processing and the notification information are monitored. If no new records are generated in the logs and there is no backlog of notification information, it is determined that the notification message is interrupted. At this time, to avoid continuously sending duplicate alarm messages, it is checked whether there are identical, unrecovered alarm records. If so, the ES logs and notification information continue to be monitored without sending alarm messages; if not, the platform receiving the interrupted notification message is determined to be the data acquisition and control platform according to the above rules, and an alarm message of notification message interruption is sent to the data acquisition and control platform. This action of sending the alarm message of message interruption is recorded. The alarm message includes: monitoring time, name of the abnormal process, time-series index data affected by the abnormality, number of communication devices affected by the abnormality, and Internet Protocol addresses of the top 5 devices most affected by the abnormality. The alarm message can be in the form of email, SMS, etc. Figure 3 As shown, after sending the alarm message, the system continues to monitor the ES logs and notification information until an ES log is generated or a notification message is detected. Then, it determines whether there are any unrecovered alarm records. If not, it continues to monitor the ES logs and notification information. If so, it resets the status of the alarm record to recovered and sends an abnormal recovery notification message to the acquisition and control platform.

[0058] According to some optional embodiments of this application, the data processing method further includes: receiving update information, wherein the update information is used to update the type of time series indicator data and the type of data model for parsing time series indicator data, and the update method corresponding to the update information includes full update and incremental update.

[0059] In other embodiments, the data sharing platform periodically updates the type of time-series indicator data stored in the indicator information table and the type of data model used to parse files composed of time-series indicator data, so that the information stored in the indicator information table is the latest and most comprehensive. Specifically, the data sharing platform in this application embodiment supports two update methods: full update and incremental update. When updating the indicator information table using the full update method, the application programming interface (API) is called to transmit the data used to update the indicator information table to the data sharing platform. When updating the indicator information table using the incremental update method, the distributed data stream processing platform (Kafka) is used to transmit the data used to update the indicator information table.

[0060] Through the above steps, a distributed server (Ceph server) is used to concurrently store files, reducing the latency generated during data transmission. It is also compatible with different data structures and provides different parsing methods for time-series indicator data with different data structures. It automatically identifies and parses time-series indicator data of different formats, improving the speed of data parsing. The entire data processing process is monitored, and alarm information is sent promptly when abnormal processes are detected, providing technical support for rapid fault location and handling, and effectively ensuring the timeliness of time-series data.

[0061] Figure 4 This is a schematic diagram of a data processing system provided according to an embodiment of this application, such as... Figure 4As shown, the data processing system includes: a distributed server 40, a model management module 42, a file parsing module 44, and a monitoring and alarm module 46. The model management module 42 receives update information, updates the type of time-series data and the data model for parsing the time-series data based on the update information, and records the type of time-series data and the data model for parsing the time-series data in an indicator information table. The distributed server 40 receives the target file to be written and sends the target file to the file parsing module 44. The target file includes multiple time-series indicator data from multiple communication devices. The file parsing module 44 receives a notification message and the target file. The notification message indicates that the distributed server 40 has been written to the target file, and the notification message includes multiple codes for multiple time-series indicator data; the file... The parsing module 44 is also used to obtain an indicator information table, determine the type of time-series indicator data corresponding to each code in the indicator information table, and determine the parsing method for parsing the target file based on the type of each time-series indicator data. The file parsing module 44 is also used to parse the target file according to the parsing method and the indicator information table, obtain the parsing result of the target file, and store the parsing result in the database in chronological order, wherein the chronological order is determined according to the generation time of multiple time-series indicator data in the target file. The monitoring and alarm module 46 is used to monitor the running process of the distributed server 40, the model management module 42, and the file parsing module 44, generate notification information when an abnormality is detected in the data processing, and determine the target object to receive the notification message based on the process in which the abnormality occurred.

[0062] Figure 5 It is a workflow diagram of a data processing system, such as Figure 5 As shown, the data sharing platform that processes (target) files composed of time-series index data interacts with the data acquisition and control platform that collects time-series index data. Users pre-define analytical models corresponding to different structures of time-series index data in the data acquisition and control platform, and synchronize the type of analytical model, the analytical results of different types of data (i.e., model content), and the structure of the time-series index data with the data information and data sharing platform. After the distributed server 40 receives a file write notification (i.e., a notification message), it transmits the notification message to the file parsing module 44. The file parsing module 44 retrieves the index information table from the model management module 42 based on the content of the notification message, and then determines the type of time-series index data based on the index information table. The content of the file write notification (i.e., the notification message) includes: the name of the file being written, the file's write path, and the encoding of each time-series index data in the file. Next, the file parsing module 44 further determines the analytical model for parsing the received file based on the content of the notification message in the index information table, such as... Figure 5As shown, when parsing a received file using the parsing model, there are two scenarios: When the parsing model is a single-index model, the received file is parsed according to the parsing result being virtual instruction encoding and port information; when the parsing model is a multi-index model, the data level of the time-series index data is determined. For time-series index data at data level 0, the received file is parsed according to the parsing result being virtual instruction encoding; for time-series index data at data levels greater than 0, the received file is parsed according to the parsing result being virtual instruction encoding, port information, and virtual local area network information. Figure 5 As shown, regardless of the parsing model used, the process of parsing and storing the data is ultimately executed, and the parsing results are stored in ascending order of the generation time of their corresponding time-series indicator data. For example... Figure 5 As shown, the monitoring and alarm module 46 in the data sharing platform performs full-process monitoring of the above data processing. Specifically, when receiving a file write notification (i.e., notification message), it performs notification message monitoring; during the process of determining the parsing model and parsing file, it performs file write monitoring and file parsing monitoring; during the process of storing the parsing result in the database, it performs data entry monitoring; and when any of the above processes is detected to be abnormal, it sends an alarm message to the platform corresponding to the abnormal process.

[0063] Figure 6 This is a structural diagram of a data processing apparatus provided according to an embodiment of this application, such as... Figure 6 As shown, the data processing device includes: a receiving module 60, used to receive a notification message, wherein the notification message indicates that a distributed server has been written to a target file, the target file includes multiple time-series index data of multiple communication devices, and the notification message includes multiple codes corresponding to the multiple time-series index data; an acquisition module 62, used to acquire an index information table, and determine the type of time-series index data corresponding to each code in the index information table; a determination module 64, used to determine the parsing method for parsing the target file based on the type of each time-series index data; and a parsing module 66, used to parse the target file according to the parsing method and the index information table, obtain the parsing result, and store the parsing result in a database in chronological order, wherein the chronological order is determined according to the generation time of the time-series data in the target file.

[0064] When the data processing device is working, the receiving module 60 receives the (target) file writing notification (i.e., notification message), the acquiring module 62 obtains the indicator information table, and searches for the code in the indicator information table that is the same as the virtual instruction code in the notification message; the determining module 64 determines the parsing method of the received (target) file according to the virtual instruction code of the time series indicator data recorded in the indicator information table; and the parsing module 66 uses the interest calculation method output by the determining module 64 to parse the received (target) file.

[0065] It should be noted that, Figure 6 Preferred embodiments of the shown examples can be found in [reference needed]. Figure 2 The relevant descriptions of the embodiments shown will not be repeated here.

[0066] This application also provides a non-volatile storage medium storing a computer program, and the above data processing method is executed by running the computer program on the device where the non-volatile storage medium is located.

[0067] The aforementioned non-volatile storage medium is used to store a program that performs the following functions: receiving a notification message, wherein the notification message indicates that a distributed server has written a target file, the target file including multiple time-series index data from multiple communication devices, and the notification message carrying multiple codes corresponding to the multiple time-series index data; obtaining an index information table, determining the type of time-series index data corresponding to each code in the index information table; determining a parsing method for parsing the target file based on the type of each time-series index data; parsing the target file according to the parsing method and the index information table, obtaining the parsing result, and storing the parsing result in a database in chronological order, wherein the chronological order is determined based on the generation time of the multiple time-series index data in the target file.

[0068] This application also provides an electronic device, including a memory and a processor. The memory stores a computer program, and the processor is configured to execute the above data processing method through the computer program.

[0069] The processor in the aforementioned electronic device is used to run a program that performs the following functions: receiving a notification message, wherein the notification message indicates that a distributed server has written a target file, the target file including multiple time-series index data of multiple communication devices, and the notification message carrying multiple codes corresponding to the multiple time-series index data; obtaining an index information table, determining the type of time-series index data corresponding to each code in the index information table; determining a parsing method for parsing the target file based on the type of each time-series index data; parsing the target file according to the parsing method and the index information table, obtaining the parsing result, and storing the parsing result in a database in chronological order, wherein the chronological order is determined based on the generation time of the multiple time-series index data in the target file.

[0070] It should be noted that each module in the above data processing device can be a program module (e.g., a set of program instructions to implement a certain function) or a hardware module. For the latter, it can be manifested in the following forms, but is not limited to them: each of the above modules is manifested as a processor, or the functions of each of the above modules are implemented by a processor.

[0071] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0072] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0073] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0074] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0075] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to related technologies, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0076] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, include: Receive a notification message, wherein the notification message indicates that the distributed server has written to the target file, the target file includes multiple time-series index data of multiple communication devices, and the notification message carries multiple codes corresponding to the multiple time-series index data; Obtain the indicator information table, and determine the type of time-series indicator data corresponding to each of the plurality of codes in the indicator information table; The parsing method for parsing the target file is determined based on the type of each of the time-series index data. The target file is parsed according to the parsing method and the indicator information table to obtain the parsing results, and the parsing results are stored in the database in chronological order, wherein the chronological order is determined according to the generation time of multiple time-series indicator data in the target file; The step of parsing the target file based on the parsing method and the indicator information table includes: when the target file contains multiple types of time-series indicator data, using a multi-indicator model to parse the target file, including: The data level of the time-series indicator data is determined, wherein the data level is used to indicate the model content associated with the time-series indicator data and the multi-indicator model. When the data level of the time-series indicator data is level 0, the model content associated with the time-series indicator data and the multi-indicator model is an encoded field; when the data level is greater than level 0, the model content associated with the time-series indicator data and the multi-indicator model includes: the virtual instruction code of each time-series indicator data, the virtual local area network identifier of the device that generates each time-series indicator data, and the port number of the port that transmits each time-series indicator data during data acquisition. When the data level of the time series indicator data is 0, the second type of model content associated with the multi-indicator model is determined in the indicator information table, wherein the second type of model content includes: encoding; the target file is parsed using the multi-indicator model to obtain a second type of parsing result that conforms to the second type of model content, wherein the second type of parsing result includes: the encoding of each time series indicator data.

2. The method according to claim 1, characterized in that, The indicator information table is used to record multiple codes, model types, model contents, model types associated with each of the multiple codes, and model contents associated with each code of various types of time series indicator data. The model type is used to indicate the type of data model used to parse the time series indicator data corresponding to each code, and the model content is used to indicate the parsing result of the time series indicator data corresponding to each code. Determining the type of time-series indicator data corresponding to each of the plurality of codes in the indicator information table includes: A target code is determined in the indicator information table, wherein the target code is the same as the code of the time series indicator data in the target file; The model type associated with the target code is determined in the indicator information table, wherein the model type is the type of time series data corresponding to the target code, and the model type includes: single indicator model and multi indicator model.

3. The method according to claim 2, characterized in that, The target file is parsed according to the parsing method and the aforementioned indicator information table, including: If the target file contains only one type of time series indicator data, then the single indicator model will be used to parse the target file. In the indicator information table, determine the first type of model content associated with the single indicator model, wherein the first type of model content includes: indicator code and port; The target file is parsed using the single-index model to obtain a first type of parsing result that conforms to the content of the first type of model. The first type of parsing result includes: the encoding of each time series index data in the target file and the port corresponding to each time series index data. If the target file contains multiple types of time-series indicator data, a multi-indicator model will be used to parse the target file.

4. The method according to claim 1, characterized in that, The parsing of the target file using a multi-index model also includes: If the data level of the time series indicator data is greater than 0, the third type of model content associated with the multi-indicator model is determined in the indicator information table, wherein the third type of model content includes: port, virtual local area network identifier and encoding; The target file is parsed using the multi-index model to obtain a third type of parsing result that conforms to the content of the second type of model. The third type of parsing result includes: a port corresponding to multiple time-series index data in the target file, a virtual local area network identifier corresponding to each time-series index data, and the encoding of each time-series index data.

5. The method according to claim 1, characterized in that, The method further includes: monitoring the data processing process and generating a notification message when an anomaly is detected in the data processing; Identify the processes that have encountered an anomaly, including: the notification message process, the file writing process, the file parsing process, and the data import process; The target object to receive the notification message is determined based on the process in which the anomaly occurred.

6. The method according to claim 5, characterized in that, Determining the target object to receive the notification message based on the process where the anomaly occurred includes: If the abnormal process is one of the following: the notification message process, the first type of abnormality in the file writing process, or the second type of abnormality in the file parsing process, the notification message is sent to the first target object, wherein the first target object is the platform that sends the target file. The first type of abnormality includes: abnormal file size, missing file, and empty file content. The second type of abnormality includes: absence of time series indicator data and missing parsing model. When the abnormal process is either a third type of abnormality in the file writing process or one of the data entry processes, the notification message is sent to the second target object, wherein the second target object is the platform that processes the target file, and the third type of abnormality includes: file download abnormality and distributed server connection abnormality; When the abnormal process is the fourth type of abnormality in the file parsing process, the notification message is sent to the first target object and the second target object, wherein the fourth type of abnormality is an abnormality in file parsing.

7. The method according to claim 1, characterized in that, The method further includes: receiving update information, wherein the update information is used to update the type of time series indicator data and the type of data model for parsing the time series indicator data, and the update method corresponding to the update information includes full update and incremental update.

8. A data processing system, characterized in that, include: The system comprises a distributed server, a model management module, a file parsing module, and a monitoring and alarm module. The model management module is used to receive update information, update the type of time series data and the data model for parsing the time series data according to the update information, and record the type of time series data and the data model for parsing the time series data in the indicator information table. The distributed server is used to receive the written target file and send the target file to the file parsing module, wherein the target file includes multiple time-series indicator data of multiple communication devices; The file parsing module is used to receive a notification message and the target file, wherein the notification message indicates that the distributed server has been written to the target file, and the notification message includes: multiple codes of the multiple time-series index data; The file parsing module is further configured to obtain the indicator information table, determine the type of time-series indicator data corresponding to each of the plurality of codes in the indicator information table, and determine the parsing method for parsing the target file based on the type of each of the time-series indicator data. The file parsing module is further configured to parse the target file according to the parsing method and the indicator information table, obtain the parsing result of the target file, and store the parsing result in a database in chronological order, wherein the chronological order is determined based on the generation time of multiple time-series indicator data in the target file; the parsing of the target file according to the parsing method and the indicator information table includes: when there are multiple types of time-series indicator data in the target file, using a multi-indicator model to parse the target file, including: determining the data level of the time-series indicator data, wherein the data level is used to indicate the model content associated with the time-series indicator data and the multi-indicator model, and when the data level of the time-series indicator data is level 0, the time... The model content associated with the time-series indicator data and the multi-indicator model is an encoded field; when the data level is greater than 0, the model content associated with the time-series indicator data and the multi-indicator model includes: the virtual instruction code of each time-series indicator data, the virtual local area network identifier of the device that generates each time-series indicator data, and the port number of the port that transmits each time-series indicator data during data acquisition; when the data level of the time-series indicator data is 0, the second type of model content associated with the multi-indicator model is determined in the indicator information table, wherein the second type of model content includes: encoding; the target file is parsed using the multi-indicator model to obtain a second type of parsing result that conforms to the second type of model content, wherein the second type of parsing result includes: the encoding of each time-series indicator data; The monitoring and alarm module is used to monitor the running processes of the distributed server, the model management module, and the file parsing module. When an anomaly is detected in the data processing, a notification message is generated, and the target object to receive the notification message is determined based on the process that has an anomaly.

9. A data processing apparatus, characterized in that, include: A receiving module is used to receive a notification message, wherein the notification message indicates that a distributed server has been written to a target file, the target file includes multiple time-series index data of multiple communication devices, and the notification message includes multiple codes corresponding to the multiple time-series index data; The acquisition module is used to acquire an indicator information table and determine the type of time-series indicator data corresponding to each of the multiple codes in the indicator information table; The determination module is used to determine the parsing method for parsing the target file based on the type of each time-series index data. The parsing module is used to parse the target file according to the parsing method and the indicator information table, obtain the parsing results, and store the parsing results in a database in chronological order, wherein the chronological order is determined according to the generation time of the time-series data in the target file; the parsing of the target file according to the parsing method and the indicator information table includes: when there are multiple types of time-series indicator data in the target file, using a multi-indicator model to parse the target file, including: determining the data level of the time-series indicator data, wherein the data level is used to indicate the model content associated with the time-series indicator data and the multi-indicator model; when the data level of the time-series indicator data is level 0, the time-series indicator data and the multi-indicator model are associated with the multi-indicator model. The model content associated with the indicator model is an encoded field; when the data level is greater than 0, the model content associated with the time series indicator data and the multi-indicator model includes: the virtual instruction code of each time series indicator data, the virtual local area network identifier of the device that generates each time series indicator data, and the port number of the port that transmits each time series indicator data during data acquisition; when the data level of the time series indicator data is 0, the second type of model content associated with the multi-indicator model is determined in the indicator information table, wherein the second type of model content includes: encoding; the target file is parsed using the multi-indicator model to obtain a second type of parsing result that conforms to the second type of model content, wherein the second type of parsing result includes: the encoding of each time series indicator data.

10. A non-volatile storage medium, characterized in that, The non-volatile storage medium stores a computer program, wherein the device containing the non-volatile storage medium executes the data processing method according to any one of claims 1 to 7 by running the computer program.

11. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the data processing method according to any one of claims 1 to 7 through the computer program.

Images