Data analysis method, device and equipment

Abstract

The present application relates to the technical field of data processing, and provides a data analysis method, device and equipment, which comprises the following steps: creating a distributed cluster, and mounting nodes in the distributed cluster to a distributed file system; storing data files downloaded from network management servers of all manufacturers in the distributed file system, and the data files are checked by cyclic redundancy check; reading and analyzing the data files corresponding to the network management servers of all manufacturers from the distributed file system based on the nodes in the distributed cluster, to obtain analysis result data corresponding to each network management server of each manufacturer; and storing the analysis result data corresponding to each network management server of each manufacturer to a local database, and performing data continuity check and cell coverage rate comparison check on the analysis result data. The present application combines the distributed file system and the distributed cluster, and combines a three-level data integrity check mechanism, so that the whole data analysis process is more efficient and reliable.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data parsing method, apparatus, and device. Background Technology

[0002] In the operation and optimization of modern communication networks, data parsing is a crucial step in achieving network performance monitoring, fault diagnosis, and optimization adjustments. Northbound servers are a key component of modern communication networks, responsible for providing data and management interfaces for network devices to higher-level systems (such as big data platforms and network optimization systems). The data files on the network optimization side of operators are diverse in type. These data files are generated by base station equipment from numerous manufacturers, distributed across a large number of network management servers, resulting in massive data volumes, high storage costs, poor data consistency, and complex management. With the continuous expansion of network scale, the diversification of base station equipment manufacturers, and the explosive growth of data volume, existing data parsing methods face numerous challenges.

[0003] The current data acquisition and parsing process typically relies on northbound servers downloading data files one by one from these network management servers, then decompressing the data files, and finally parsing them. Because northbound servers use a distributed data acquisition method, the rate of missed data collection is high, and it is difficult to pinpoint the source of these missed data issues. Furthermore, due to the lack of an effective load balancing mechanism, some servers may be overloaded while others remain idle, resulting in uneven load distribution and low resource utilization. Summary of the Invention

[0004] This invention provides a data parsing method, apparatus, and device to solve the technical problems in the prior art where the northbound server adopts a distributed data collection method, resulting in a high rate of missed data collection and difficulty in locating missed data collection, as well as the lack of an effective load balancing mechanism, leading to uneven load.

[0005] This invention provides a data parsing method, comprising the following steps: Create a distributed cluster and mount the nodes in the distributed cluster to a distributed file system; wherein the distributed file system stores data files downloaded from network management servers of all manufacturers, and the data files are verified by cyclic redundancy check; The nodes in the distributed cluster are invoked to execute the parsing tasks of the data files corresponding to the network management servers of each manufacturer in the distributed file system in parallel, so as to obtain the parsing result data corresponding to the network management server of each manufacturer. The parsing result data corresponding to each manufacturer's network management server is stored in a local database, and the parsing result data is verified for data continuity and month-on-month cell coverage.

[0006] According to a data parsing method provided by the present invention, nodes in the distributed cluster are invoked to perform parsing tasks of data files corresponding to network management servers of various manufacturers in a distributed file system in parallel, including: Upon detecting a file write close event at a file path in the distributed file system, perform file integrity verification on the data file corresponding to the file write close event; If the data file corresponding to the file write closed event passes the file integrity check, a parsing task for the data file corresponding to the file write closed event is created.

[0007] According to a data parsing method provided by the present invention, before detecting a file write close event under a file path in a distributed file system, the method further includes: Define a file monitor class, which is used to listen for file write close events in the distributed file system; Initialize the file monitor class to create observer objects and event handlers, and associate the event handlers with file paths in the distributed file system.

[0008] According to a data parsing method provided by the present invention, nodes in the distributed cluster are invoked to perform parsing tasks of data files corresponding to network management servers of various manufacturers in a distributed file system in parallel, including: After creating the parsing task for the corresponding data file, the target node corresponding to the parsing task is determined based on the load status of the nodes in the distributed cluster. The parsing task is assigned to the thread pool of the target node, and the parsing task is processed based on the threads in the thread pool of the target node. After the parsing task is detected to be completed, the task status corresponding to the parsing task is updated to the completed status.

[0009] According to a data parsing method provided by the present invention, the load status includes the length of the pending task queue and the number of parsing tasks completed per unit time; determining the target node corresponding to the parsing task based on the load status of the nodes in the distributed cluster includes: Determine the length of the pending task queue and the number of parsing tasks completed per unit time for each node in the current distributed cluster; Based on the length of the task queue to be processed and the number of parsed tasks completed, a weighted scoring strategy is adopted to determine the load status score of the nodes in the distributed cluster; the weight coefficient corresponding to the length of the task queue to be processed is positive, and the weight coefficient corresponding to the number of parsed tasks completed is negative. The node with the lowest load status score is determined as the target node corresponding to the parsing task.

[0010] According to a data parsing method provided by the present invention, the load status includes the length of the queue of tasks to be processed; after determining the target node corresponding to the parsing task based on the load status of the nodes in the distributed cluster, the method further includes: If the length of the pending task queue of the target node is detected to be greater than the first task queue length threshold, the expansion quantity corresponding to each node in the distributed cluster is determined based on the length of the pending task queue of each node in the distributed cluster. Create a corresponding number of Pod instances for each node in the distributed cluster, and distribute the created Pod instances to the corresponding nodes in the distributed cluster; If the length of the pending task queue of the target node is less than the second task queue length threshold, all nodes in the distributed cluster that need to be scaled down are determined; the length of the pending task queue of the nodes to be scaled down is less than the second task queue length threshold. Based on the length of the pending task queue for each node to be scaled down, determine the number of Pod instances to be scaled down for each node, and release the corresponding number of Pod instances for each node to be scaled down.

[0011] According to a data parsing method provided by the present invention, data continuity verification is performed on the parsed result data, including: For each manufacturer's gateway server, the local database is checked every preset time interval to see if the parsing result data corresponding to that manufacturer's gateway server is stored within that preset time interval. If the local database is found to contain the parsing result data corresponding to the manufacturer's gateway server within the preset time period, the data continuity check is deemed to have passed.

[0012] According to a data parsing method provided by the present invention, performing a cell coverage rate comparison verification on the parsed result data includes: After the parsing result data of all manufacturers' network management servers is stored each day, the cell coverage rate of the parsing result data of all manufacturers' network management servers on the current day is determined compared with the parsing result data of all manufacturers' network management servers on the previous day. If the month-on-month change in cell coverage rate is within the preset range, the cell coverage rate month-on-month change verification is passed.

[0013] The present invention also provides a data parsing apparatus, comprising: The first data parsing module is used to create a distributed cluster and mount the nodes in the distributed cluster to a distributed file system; wherein, the distributed file system stores data files downloaded from network management servers of all manufacturers, and the data files are verified by cyclic redundancy check; The second data parsing module is used to call the nodes in the distributed cluster to execute the parsing tasks of the data files corresponding to the network management servers of each manufacturer in the distributed file system in parallel, so as to obtain the parsing result data corresponding to the network management server of each manufacturer. The third data parsing module is used to store the parsing result data corresponding to each manufacturer's network management server into the local database, and to perform data continuity verification and cell coverage rate comparison verification on the parsing result data.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the program to implement the data parsing method as described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data parsing method as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the data parsing methods described above.

[0017] The data parsing method, apparatus, and device provided by this invention achieve centralized storage and management of massive data files by creating a distributed cluster and mounting nodes to a distributed file system, and ensure data integrity using cyclic redundancy check. Furthermore, the distributed cluster nodes read and parse data from the distributed file system in parallel, replacing the traditional decentralized collection mode, reducing the missed sampling rate and simplifying problem localization. In addition, the distributed cluster can dynamically allocate parsing tasks, balance node load, improve resource utilization, and perform data continuity verification and cell coverage ratio verification again after parsing. This invention, through a distributed file system and distributed cluster, combined with a three-level data integrity verification mechanism, makes the entire data parsing process more efficient and reliable. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is one of the flowcharts illustrating the data parsing method provided by the present invention.

[0020] Figure 2 This is the second flowchart of the data parsing method provided by the present invention.

[0021] Figure 3 This is a schematic diagram of the data parsing device provided by the present invention.

[0022] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0024] The data parsing method of this invention, as shown in the embodiments, is as follows: Figure 1 As shown, it includes steps 110, 120 and 130.

[0025] Step 110: Create a distributed cluster and mount the nodes in the distributed cluster to a distributed file system; wherein, the distributed file system stores data files downloaded from network management servers of all manufacturers, and the data files pass cyclic redundancy check.

[0026] In this embodiment, a distributed cluster is used as the parsing orchestration platform, responsible for scheduling and managing parsing tasks, and a distributed file system (Ceph File System, CephFS) is used as the file storage container. In the distributed cluster, each node (such as a server) in the cluster is connected to CephFS via a network, and CephFS is accessed and managed as a unified storage resource.

[0027] It should be understood here that each manufacturer of network management servers is different, therefore the base station data types involved in the data files of each manufacturer's network management server are also different.

[0028] In practical applications, a Kubernetes (K8S) cluster can be created. A K8S cluster includes a Master node (the control center of the K8S cluster, responsible for managing the entire cluster) and Worker nodes (nodes running application containers). Nodes are connected via a high-speed network to achieve message passing and data exchange. Specifically, Worker nodes in the K8S cluster can be mounted to CephFS.

[0029] Here, mounting is an operating system operation that attaches a file system (such as CephFS) to a directory (mount point), making the files and directories under that directory accessible. In a distributed cluster, each node attaches CephFS to a directory on its local file system via a mount operation.

[0030] For example, nodes in a distributed cluster can be mounted to CephFS by executing the command "mount -t ceph : / / mnt / cephfs -oname=admin,secret=...", ensuring that all nodes can directly access the unified storage resource.

[0031] It should be understood that CephFS is a component of the Ceph storage cluster, built on Ceph's distributed object storage (RADOS), and provides a POSIX-compliant file system interface. Data stored in CephFS is distributed across multiple nodes, supporting horizontal scaling and allowing multiple clients to mount and access the same file system simultaneously.

[0032] In this embodiment, a corresponding CephFS volume is pre-created in the Ceph cluster for each vendor's network management server. The CephFS volume can be mounted to the network management server for storing data files. Additionally, the CephFS volume can be mounted to nodes (servers) in the distributed cluster, allowing nodes in the distributed cluster to access the stored data files. After creating the corresponding CephFS volume, the storage pool it uses is specified.

[0033] Based on the above configuration, data files are periodically downloaded from the network management servers of all manufacturers and stored in CephFS. It should be noted that after the downloaded data files are stored in the corresponding CephFS directory, the file mapping information can be viewed using CephFS tools to determine the file path and storage directory of the data files.

[0034] Furthermore, to ensure the integrity and consistency of data storage, cyclic redundancy check (CRC) is performed during the process of storing data files into CephFS. Here, CRC is an error detection technique used to detect whether errors have occurred during data transmission or storage. For example, a CRC32 check mechanism can be used; when a data file is written from the network management server to CephFS, the CRC32 check mechanism verifies whether the data blocks have been corrupted during transmission or storage.

[0035] Step 120: Invoke the nodes in the distributed cluster to execute the parsing tasks of the data files corresponding to the network management servers of each manufacturer in CEPHFS in parallel, and obtain the parsing result data corresponding to the network management servers of each manufacturer.

[0036] Here, we will continue to use a Kubernetes (K8S) cluster as an example for explanation. Specifically, the Master node in a K8S cluster can monitor the load status of each Worker node in real time. Based on the load status of each Worker node, it calls the Worker nodes in the K8S cluster to access and resolve the file paths and data files in the storage directories mounted by the Worker nodes.

[0037] In this embodiment, the data parsing method can be flexibly adjusted based on the actual data parsing requirements, and there are no restrictions on it.

[0038] Step 130: Store the parsing result data corresponding to each manufacturer's network management server into the local database, and perform data continuity verification and cell coverage rate comparison verification on the parsing result data.

[0039] After obtaining the parsing result data corresponding to each manufacturer's network management server, the parsing result data is loaded into a local database, such as the StarRocks MPP database or the Doris MPP database.

[0040] For ease of understanding, please refer to Figure 2 , Figure 2 This is the second flowchart illustrating the data parsing method provided in this embodiment of the invention. As shown in the figure, a unified acquisition program downloads data files from network management servers of different manufacturers and stores them in CephFS. The parsing server (i.e., the worker node) in the K8S cluster mounts a unified CephFS directory. The parsing server directly parses the data files stored in CephFS through a download-free process and stores the parsed result data in a local analytical database.

[0041] Furthermore, after storing the data in the local database, a dual verification mechanism is used for the data in the local database to reduce the data loss rate.

[0042] Here, data continuity verification refers to checking the timestamps or sampling points in the data records to ensure that the data is continuous over time. Cell coverage rate month-on-month verification refers to performing a daily month-on-month comparison of the cell coverage area or performance indicators to detect whether there are abnormal changes in the cell coverage area or performance indicators at a macro level, and thus check whether data omissions have occurred.

[0043] The data parsing method in this embodiment achieves centralized storage and management of massive data files by creating a distributed cluster and mounting nodes to a distributed file system, and ensures data integrity using cyclic redundancy check. Furthermore, the distributed cluster nodes read and parse data from the distributed file system in parallel, replacing the traditional decentralized collection mode, reducing the missed sampling rate and simplifying problem localization. In addition, the distributed cluster can dynamically allocate parsing tasks, balance node load, improve resource utilization, and perform data continuity verification and cell coverage ratio verification again after parsing. This invention, through a distributed file system and a distributed cluster, combined with a three-level data integrity verification mechanism, makes the entire data parsing process more efficient and reliable.

[0044] It should be noted that each implementation method of this application can be freely combined, rearranged, or executed individually, and does not need to rely on or depend on a fixed execution order.

[0045] In some embodiments, performing data continuity verification on the parsed result data includes: For each manufacturer's gateway server, the local database is checked every preset time interval to see if the parsing result data corresponding to that manufacturer's gateway server is stored within that preset time interval. If the local database is found to contain the parsing result data corresponding to the manufacturer's gateway server within the preset time period, the data continuity check is deemed to have passed.

[0046] Specifically, this involves verifying the continuity and integrity of the time-series data on each manufacturer's network management server.

[0047] It should be understood that the data on the network management server is continuously generated. Therefore, in this embodiment, data files are continuously downloaded from the network management server of each manufacturer and stored in CephFS at preset intervals. Based on this, this embodiment performs data continuity verification on the data parsed by the nodes in the distributed cluster.

[0048] For example, every 15 minutes, the system checks whether the local database has again stored the parsing result data corresponding to the manufacturer's gateway server. If it detects that the local database has again stored the parsing result data corresponding to the manufacturer's gateway server within that 15-minute period, it is determined that the current parsing result data corresponding to the manufacturer's gateway server is not missing; otherwise, it is marked that there is missing data during this period.

[0049] The data parsing method in this embodiment verifies the continuity and integrity of the time-series data of each manufacturer's network management server, enabling timely detection of possible interruptions or anomalies during data acquisition, and thus accurately locating data loss.

[0050] In some embodiments, performing a cell coverage rate comparison check on the parsed result data includes: After the parsing result data of all manufacturers' network management servers is stored each day, the cell coverage rate of the parsing result data of all manufacturers' network management servers on the current day is determined compared with the parsing result data of all manufacturers' network management servers on the previous day. If the month-on-month change in cell coverage rate is within the preset range, the cell coverage rate month-on-month change verification is passed.

[0051] Specifically, at the community business logic level, the data integrity of each manufacturer's network management server is checked once a day.

[0052] It should be understood that by parsing the raw data reported by the network management server, cell coverage rate can be generated. Cell coverage rate is a key performance indicator in communication networks (such as 4G / 5G) and is used to measure the signal coverage quality of a specific base station (cell).

[0053] In a normally operating mobile network, cell coverage typically fluctuates slowly rather than abruptly. Therefore, in this embodiment, after the parsing results data for all network management servers from all manufacturers are stored daily, a month-on-month verification is performed based on the cell coverage rate in the parsing results data. If the month-on-month cell coverage rate is within the preset range, the current data collection is considered normal, and no data omissions have occurred. If the month-on-month cell coverage rate exceeds the preset range, for example, if the minimum coverage rate within the preset range is 90% and the current day's cell coverage rate is 70%, the data collection is considered abnormal (e.g., missed data).

[0054] In some embodiments, the nodes in the distributed cluster are invoked to perform parallel parsing tasks of data files corresponding to the network management servers of various manufacturers in the distributed file system, including: Upon detecting a file write close event at a file path in the distributed file system, perform file integrity verification on the data file corresponding to the file write close event; If the data file corresponding to the file write closed event passes the file integrity check, a parsing task for the data file corresponding to the file write closed event is created.

[0055] In this embodiment, the file monitoring mechanism and file integrity verification can be combined to call the nodes in the distributed cluster to perform the parsing task.

[0056] In this step, when a file write close event is detected under a file path in CephFS, the kernel adds the file write close event to the queue and calls the nodes in the distributed cluster to execute the file write close events in the queue in parallel.

[0057] It should be understood that in CephFS, a file path refers to the complete path of a file within CephFS. The file path starts from the mount point and points to the specific file or directory within CephFS. A file write close event is an event triggered when a file is closed after a write operation is completed.

[0058] In CephFS, the inotify mechanism can be used to listen for file write and close events at a file path. Inotify is a file system event monitoring mechanism provided by the Linux kernel, allowing programs to monitor various events in the file system. Inotify maintains an event queue in the kernel; when a specified event occurs in the file system, the kernel adds the event information to the queue.

[0059] Upon detecting a file write close event, the system first performs a file integrity check on the data file corresponding to that event. Specifically, it verifies the file's integrity and consistency to ensure the file was not corrupted during the write process. For example, it uses the MD5 (Message-Digest Algorithm 5) algorithm to verify data integrity; if the MD5 check passes, it indicates that the data file is intact and undamaged.

[0060] After the data file corresponding to the file write close event passes the file integrity check, the download-free parsing process is triggered. A parsing task for the data file corresponding to this file write close event is created, and the file is parsed and processed directly without downloading the file first.

[0061] In some embodiments, before detecting the file write close event at the file path in the distributed file system, the method further includes: Define a file monitor class, which is used to listen for file write close events in the distributed file system; Initialize the file monitor class to create observer objects and event handlers, and associate the event handlers with file paths in the distributed file system.

[0062] Specifically, first define a custom file watcher class to encapsulate the file monitoring logic based on inotify, such as a custom FileWatcher class.

[0063] Next, the file monitor class is initialized. In the file monitor class's initialization method, an Observer object is first created to listen for changes in the file system. Then, a CephEventHandler event handler is created to define the specific event handling logic. Finally, the `schedule` method of the Observer object is used to associate the CephEventHandler with a specified file path in CephFS. Here, the specified file path in CephFS refers to the file path of the data file to be monitored.

[0064] In this embodiment, the above configuration enables the `start` method of the file monitor class to launch the Observer object and begin monitoring file system changes under the specified file path. When a file is closed and a write operation has previously occurred, a file write close event is triggered. The Observer object captures the event and notifies the CephEventHandler. In the `on_close_write` method of the CephEventHandler, an MD5 checksum is performed on the data file corresponding to the file write close event. If the MD5 checksum passes, a download-free parsing process is triggered, and the file is directly parsed and processed.

[0065] The data parsing method in this embodiment uses a file monitoring mechanism to detect changes in the data file in a timely manner and performs a file integrity check after the file is written to ensure that the data file being parsed subsequently is intact and undamaged. Finally, after the data file passes the file integrity check, a download-free parsing process is triggered, improving file processing efficiency and reducing unnecessary file download operations.

[0066] In some embodiments, the nodes in the distributed cluster are invoked to perform parallel parsing tasks of data files corresponding to the network management servers of various manufacturers in the distributed file system, including: After creating the parsing task for the corresponding data file, the target node corresponding to the parsing task is determined based on the load status of the nodes in the distributed cluster. The parsing task is assigned to the thread pool of the target node, and the parsing task is processed based on the threads in the thread pool of the target node. After the parsing task is detected to be completed, the task status corresponding to the parsing task is updated to the completed status.

[0067] In this embodiment, after creating the parsing task for the corresponding data file, the target node corresponding to the parsing task can be determined based on the load status of the nodes in the distributed cluster and a pre-configured load balancing strategy.

[0068] After the target node is determined, the parsing task is assigned to the target node's thread pool, and the target node processes the parsing task through threads in the thread pool.

[0069] Here, you can use `Executors.newFixedThreadPool(custom thread count)` to create a thread pool with a dynamically adjustable thread count via a configuration file. This ensures sufficient thread resources for parsing tasks while avoiding the overhead of frequently creating threads. When a new data file is detected in the file path in CephFS, a parsing task is automatically generated and submitted to the thread pool on the target node. Each thread in the thread pool is responsible for parsing one data file. Through thread pool management, multiple data files can be parsed simultaneously without blocking each other.

[0070] Furthermore, once the data file has been parsed, its parsing task status will be marked as processed, so that subsequent repetitive processing requests can be ignored or skipped directly.

[0071] The data parsing method in this embodiment first determines the target node corresponding to the parsing task based on the load status of the nodes in the distributed cluster to ensure system load balancing. Then, the parsing task is allocated to the thread pool of the target node to achieve efficient parallel processing of the task. Finally, after the file parsing is completed, the task status corresponding to this parsing task is updated to the completed state to ensure the idempotency of the task and avoid data inconsistency caused by repeated processing.

[0072] In some embodiments, the load status includes the length of the pending task queue and the number of parsing tasks completed per unit time; determining the target node corresponding to the parsing task based on the load status of the nodes in the distributed cluster includes: Determine the length of the pending task queue and the number of parsing tasks completed per unit time for each node in the current distributed cluster; Based on the length of the task queue to be processed and the number of parsed tasks completed, a weighted scoring strategy is adopted to determine the load status score of the nodes in the distributed cluster; the weight coefficient corresponding to the length of the task queue to be processed is positive, and the weight coefficient corresponding to the number of parsed tasks completed is negative. The node with the lowest load status score is determined as the target node corresponding to the parsing task.

[0073] In this embodiment, load balancing task allocation is performed by considering two dimensions: the length of the task queue to be processed and the number of parsed tasks completed per unit time.

[0074] The length of the pending task queue reflects the number of tasks a node currently needs to process and is an important indicator of node load. The longer the pending task queue, the heavier the node's task load.

[0075] The number of parsing tasks completed per unit of time reflects the processing efficiency of a node and is another important indicator for measuring node load. The higher the number of parsing tasks completed per unit of time, the stronger the node's processing capacity or the more abundant its resources.

[0076] Specifically, the load status score of a node is comprehensively quantified by combining the length of the pending task queue, which positively reflects load pressure, and the number of parsing tasks completed, which negatively offsets load pressure. By assigning a positive weight to the length of the pending task queue and a negative weight to the number of parsing tasks completed, an increase in the pending task queue length (positive weight) increases the score, indicating a heavier load, while a decrease in the number of parsing tasks completed (negative weight) decreases the score, reflecting strong processing capacity. The higher the final score, the heavier the node's load. Therefore, in this embodiment, the node with the lowest load status score is selected as the target node corresponding to the current parsing task.

[0077] The data parsing method in this embodiment accurately identifies the node load status by combining the length of the unprocessed task queue that positively reflects the load pressure and the number of parsing tasks completed that negatively offsets the load pressure. Then, it prioritizes the allocation of parsing tasks to target nodes with low load, thereby achieving dynamic balance of system load.

[0078] In some embodiments, the load status includes the length of the queue of tasks to be processed; after determining the target node corresponding to the parsing task based on the load status of the nodes in the distributed cluster, the method further includes: If the length of the pending task queue of the target node is detected to be greater than the first task queue length threshold, the expansion quantity corresponding to each node in the distributed cluster is determined based on the length of the pending task queue of each node in the distributed cluster. Create a corresponding number of Pod instances for each node in the distributed cluster, and distribute the created Pod instances to the corresponding nodes in the distributed cluster; If the length of the pending task queue of the target node is less than the second task queue length threshold, all nodes in the distributed cluster that need to be scaled down are determined; the length of the pending task queue of the nodes to be scaled down is less than the second task queue length threshold. Based on the length of the pending task queue for each node to be scaled down, determine the number of Pod instances to be scaled down for each node, and release the corresponding number of Pod instances for each node to be scaled down.

[0079] In this embodiment, the load status of each node is monitored in real time, and dynamic scaling operations are performed based on the load status. Here, after the parsing task is assigned to the target node with the least load, the target node can be used as a reference standard to determine whether the entire cluster needs dynamic scaling operations.

[0080] Specifically, when the length of the pending task queue of the target node is detected to be greater than the first task queue length threshold, since the target node is the node with the least load in the current distributed cluster, this indicates that all nodes in the current distributed cluster have insufficient resources. Therefore, based on the length of the pending task queue of each node in the current distributed cluster, the number of expansions required for each node is determined. Then, the corresponding number of Pod instances are created for each node, and the created Pod instances are allocated to the corresponding nodes for their use.

[0081] Furthermore, when the length of the pending task queue of a target node is detected to be less than the second task queue length threshold, it indicates that there is at least one node with surplus resources in the current distributed cluster. Then, all nodes with pending task queue lengths less than the second task queue length threshold are selected from the current distributed cluster to be scaled down. Based on the pending task queue length of each node to be scaled down, the corresponding number of Pod instances to be scaled down for each node is determined, and the corresponding number of Pod instances to be scaled down are released for each node, thus achieving reasonable resource allocation.

[0082] The data parsing method in this embodiment dynamically expands and shrinks based on the length of the node's pending task queue, ensuring efficient system operation and rational resource utilization.

[0083] The data parsing apparatus provided by the present invention will be described below. The data parsing apparatus described below can be referred to in correspondence with the data parsing method described above.

[0084] The data parsing device of this invention embodiment, such as Figure 3 As shown, it includes the following modules: a first data parsing module 310, a second data parsing module 320, and a third data parsing module 330.

[0085] The first data parsing module 310 is used to create a distributed cluster and mount the nodes in the distributed cluster to a distributed file system; wherein, the distributed file system stores data files downloaded from network management servers of all manufacturers, and the data files pass cyclic redundancy check.

[0086] The second data parsing module 320 is used to call the nodes in the distributed cluster to execute the parsing tasks of the data files corresponding to the network management servers of each manufacturer in the distributed file system in parallel, so as to obtain the parsing result data corresponding to the network management server of each manufacturer.

[0087] The third data parsing module 330 is used to store the parsing result data corresponding to each manufacturer's network management server into a local database, and to perform data continuity verification and cell coverage rate comparison verification on the parsing result data.

[0088] The data parsing device in this embodiment achieves centralized storage and management of massive data files by creating a distributed cluster and mounting nodes to a distributed file system, and ensures data integrity using cyclic redundancy check. Furthermore, the distributed cluster nodes read and parse data from the distributed file system in parallel, replacing the traditional decentralized collection mode, reducing the missed sampling rate and simplifying problem localization. In addition, the distributed cluster can dynamically allocate parsing tasks, balance node load, improve resource utilization, and perform data continuity verification and cell coverage ratio verification again after parsing. This invention, through a distributed file system and distributed cluster, combined with a three-level data integrity verification mechanism, makes the entire data parsing process more efficient and reliable.

[0089] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a data parsing method, which includes: Create a distributed cluster and mount the nodes in the distributed cluster to a distributed file system; wherein the distributed file system stores data files downloaded from network management servers of all manufacturers, and the data files are verified by cyclic redundancy check; The nodes in the distributed cluster are invoked to execute the parsing tasks of the data files corresponding to the network management servers of each manufacturer in the distributed file system in parallel, so as to obtain the parsing result data corresponding to the network management server of each manufacturer. The parsing result data corresponding to each manufacturer's network management server is stored in a local database, and the parsing result data is verified for data continuity and month-on-month cell coverage.

[0090] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., each of which can store program code.

[0091] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is capable of executing the data parsing method provided by each of the above methods, the method comprising: Create a distributed cluster and mount the nodes in the distributed cluster to a distributed file system; wherein the distributed file system stores data files downloaded from network management servers of all manufacturers, and the data files are verified by cyclic redundancy check; The nodes in the distributed cluster are invoked to execute the parsing tasks of the data files corresponding to the network management servers of each manufacturer in the distributed file system in parallel, so as to obtain the parsing result data corresponding to the network management server of each manufacturer. The parsing result data corresponding to each manufacturer's network management server is stored in a local database, and the parsing result data is verified for data continuity and month-on-month cell coverage.

[0092] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the data parsing method provided by each of the above methods, the method comprising: Create a distributed cluster and mount the nodes in the distributed cluster to a distributed file system; wherein the distributed file system stores data files downloaded from network management servers of all manufacturers, and the data files are verified by cyclic redundancy check; The nodes in the distributed cluster are invoked to execute the parsing tasks of the data files corresponding to the network management servers of each manufacturer in the distributed file system in parallel, so as to obtain the parsing result data corresponding to the network management server of each manufacturer. The parsing result data corresponding to each manufacturer's network management server is stored in a local database, and the parsing result data is verified for data continuity and month-on-month cell coverage.

[0093] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0094] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in each of the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of each embodiment of the present invention.

Claims

1. A data parsing method, characterized in that, include: Create a distributed cluster and mount the nodes in the distributed cluster to a distributed file system; wherein the distributed file system stores data files downloaded from network management servers of all manufacturers, and the data files are verified by cyclic redundancy check; The nodes in the distributed cluster are invoked to execute the parsing tasks of the data files corresponding to the network management servers of each manufacturer in the distributed file system in parallel, so as to obtain the parsing result data corresponding to the network management server of each manufacturer. The parsing result data corresponding to each manufacturer's network management server is stored in a local database. For each manufacturer's gateway server, the parsing result data corresponding to that manufacturer's gateway server is checked every preset time interval to see if it has been stored in the local database within that preset time interval. If the parsing result data corresponding to that manufacturer's gateway server is detected to have been stored in the local database within that preset time interval, the data continuity check is determined to have passed. Furthermore, after the parsing result data corresponding to all manufacturers' network management servers is stored each day, the cell coverage rate of the parsing result data corresponding to all manufacturers' network management servers on that day is compared with the cell coverage rate of the parsing result data corresponding to all manufacturers' network management servers on the previous day. If the cell coverage rate comparison is within a preset cell coverage rate comparison range, the cell coverage rate comparison check is determined to have passed. This includes invoking nodes in the distributed cluster to perform parallel parsing tasks for the data files corresponding to the network management servers of various manufacturers in the distributed file system, including: After creating the parsing task for the corresponding data file, the target node corresponding to the parsing task is determined based on the load status of the nodes in the distributed cluster. The parsing task is assigned to the thread pool of the target node, and the parsing task is processed based on the threads in the thread pool of the target node. After the parsing task is detected to be completed, the task status corresponding to the parsing task is updated to the completed status. The load status includes the length of the pending task queue and the number of parsing tasks completed per unit time; based on the load status of the nodes in the distributed cluster, the target node corresponding to the parsing task is determined, including: Determine the length of the pending task queue and the number of parsing tasks completed per unit time for each node in the current distributed cluster; Based on the length of the task queue to be processed and the number of parsed tasks completed, a weighted scoring strategy is adopted to determine the load status score of the nodes in the distributed cluster; the weight coefficient corresponding to the length of the task queue to be processed is positive, and the weight coefficient corresponding to the number of parsed tasks completed is negative. The node with the lowest load status score is determined as the target node corresponding to the parsing task.

2. The data parsing method according to claim 1, characterized in that, The task of parsing data files corresponding to the network management servers of various manufacturers in the distributed file system is executed in parallel by calling nodes in the distributed cluster, including: Upon detecting a file write close event at a file path in the distributed file system, perform file integrity verification on the data file corresponding to the file write close event. If the data file corresponding to the file write closed event passes the file integrity check, a parsing task for the data file corresponding to the file write closed event is created.

3. The data parsing method according to claim 2, characterized in that, Before detecting the file write close event at the file path in the distributed file system, the process also includes: Define a file monitor class, which is used to listen for file write close events in the distributed file system; Initialize the file monitor class to create observer objects and event handlers, and associate the event handlers with file paths in the distributed file system.

4. The data parsing method according to claim 1, characterized in that, The load status includes the length of the queue of tasks to be processed; After determining the target node corresponding to the parsing task based on the load status of the nodes in the distributed cluster, the process further includes: If the length of the pending task queue of the target node is detected to be greater than the first task queue length threshold, the expansion quantity corresponding to each node in the distributed cluster is determined based on the length of the pending task queue of each node in the distributed cluster. Create a corresponding number of Pod instances for each node in the distributed cluster, and distribute the created Pod instances to the corresponding nodes in the distributed cluster; If the length of the pending task queue of the target node is less than the second task queue length threshold, all nodes in the distributed cluster that need to be scaled down are determined; the length of the pending task queue of the nodes to be scaled down is less than the second task queue length threshold. Based on the length of the pending task queue for each node to be scaled down, determine the number of Pod instances to be scaled down for each node, and release the corresponding number of Pod instances for each node to be scaled down.

5. A data parsing device, characterized in that, include: The first data parsing module is used to create a distributed cluster and mount the nodes in the distributed cluster to a distributed file system; wherein, the distributed file system stores data files downloaded from network management servers of all manufacturers, and the data files are verified by cyclic redundancy check; The second data parsing module is used to call the nodes in the distributed cluster to execute the parsing tasks of the data files corresponding to the network management servers of each manufacturer in the distributed file system in parallel, so as to obtain the parsing result data corresponding to the network management server of each manufacturer. The third data parsing module is used to store the parsing result data corresponding to each manufacturer's network management server into a local database. For each manufacturer's gateway server, it checks at preset intervals whether the parsing result data corresponding to that manufacturer's gateway server has been stored in the local database within that preset interval. If the parsing result data corresponding to that manufacturer's gateway server is detected to have been stored in the local database within that preset interval, it determines that the data continuity check has passed. Furthermore, after the parsing result data corresponding to all manufacturers' network management servers has been stored each day, it determines the cell coverage rate comparison between the parsing result data corresponding to all manufacturers' network management servers on that day and the parsing result data corresponding to all manufacturers' network management servers on the previous day. If the cell coverage rate comparison is within a preset cell coverage rate comparison range, it determines that the cell coverage rate comparison check has passed. This includes invoking nodes in the distributed cluster to perform parallel parsing tasks for the data files corresponding to the network management servers of various manufacturers in the distributed file system, including: After creating the parsing task for the corresponding data file, the target node corresponding to the parsing task is determined based on the load status of the nodes in the distributed cluster. The parsing task is assigned to the thread pool of the target node, and the parsing task is processed based on the threads in the thread pool of the target node. After the parsing task is detected to be completed, the task status corresponding to the parsing task is updated to the completed status. The load status includes the length of the pending task queue and the number of parsing tasks completed per unit time; based on the load status of the nodes in the distributed cluster, the target node corresponding to the parsing task is determined, including: Determine the length of the pending task queue and the number of parsing tasks completed per unit time for each node in the current distributed cluster; Based on the length of the task queue to be processed and the number of parsed tasks completed, a weighted scoring strategy is adopted to determine the load status score of the nodes in the distributed cluster; the weight coefficient corresponding to the length of the task queue to be processed is positive, and the weight coefficient corresponding to the number of parsed tasks completed is negative. The node with the lowest load status score is determined as the target node corresponding to the parsing task.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the data parsing method as described in any one of claims 1 to 4.

Images