Adaptive data acquisition method, device, equipment, medium and computer program product

Abstract

The present invention discloses an adaptive data acquisition method, apparatus, device, medium, and computer program product. The method comprises: traversing OMC list information one by one to obtain target data file information; packaging the target data file information into task packages according to the priority of the target data file information, and writing the task packages into corresponding priority queues; adaptively correcting the target number of task packages pulled according to the processing time of data parsing; and pulling a corresponding number of task packages from the priority queue according to the target number to parse the target data in the task packages. The present invention divides data acquisition and parsing into two independent links, adaptively correcting the amount of data to be collected according to the duration of data parsing, thereby balancing the resource utilization of data acquisition and parsing servers, minimizing data processing delays, and avoiding data backlogs or losses. At the same time, by configuring different data delay priority queues, it can flexibly adapt to the needs of various different delay priorities and improve resource utilization.

Description

Technical Field

[0001] The present invention relates to the field of data acquisition technology, and in particular to an adaptive data acquisition method, device, equipment, medium and computer program product. Background Art

[0002] In existing networks, 4G and 5G MR and MDT data from different equipment vendors typically adhere to wireless network management specifications and are exported in XML or CSV file formats. The smallest file unit typically contains 15 minutes of measurement data for a single base station (4G eNodeB or 5G gNodeB). A single OMC (network management center) typically manages hundreds or thousands of 4G or 5G base stations. Some equipment vendors aggregate MR and MDT data from multiple OMCs onto a single OMC FTP server. Single and double compression of MR and MDT data occurs simultaneously, resulting in individual file sizes on a single OMC ranging from tens or hundreds of kilobytes to over 1GB. Furthermore, the number of data files in a single folder on the server can range from tens or hundreds to over 10,000. This makes it difficult to evenly distribute the collection and parsing tasks to the servers running the collection and parsing tasks concurrently during the distributed / parallel collection / parsing of 4G, 5G MR and MDT data. This results in an uneven distribution of collection tasks and inefficient resource utilization. Severely unbalanced task distribution can also cause some collection servers to fail to meet the latency requirements for each batch of data processing, leading to excessive data delays or even loss. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide an adaptive data collection method, device, equipment, medium and computer program product, which divides data collection and analysis into two independent links, and adaptively corrects the amount of data to be collected according to the duration of data analysis, thereby balancing the resource utilization of data collection and analysis servers, minimizing data processing delay, and avoiding data backlog or loss; at the same time, by configuring different data delay priority queues, it can flexibly adapt to the needs of various different delay priorities and improve resource utilization.

[0004] To achieve the above objectives, an embodiment of the present invention provides an adaptive data acquisition method, comprising:

[0005] Traverse the OMC list information one by one to obtain the target data file information;

[0006] Packing the target data file information into a task package according to the priority of the target data file information, and writing the task package into a corresponding priority queue;

[0007] Adaptively modifying the target number of task packets pulled from the priority queue according to the processing time of data analysis;

[0008] A corresponding number of task packets are pulled from the priority queue according to the target number to parse the target data in the task packets.

[0009] As an improvement to the above solution, the number of the priority queue is at least one, and the method further includes:

[0010] The number of priority queues, the priority of each priority queue, and the data object corresponding to each priority are determined according to a preset delay requirement.

[0011] As an improvement to the above solution, the OMC list information includes the server's IP address, port, data type, data storage file root path, login account, and login password. The step of traversing the OMC list information one by one to obtain target data file information includes:

[0012] The OMC list information is traversed one by one according to a preset period, the data storage file root path of the target data is found, and the target data file information is obtained; wherein the target data file information includes the name and size of the file.

[0013] As an improvement to the above solution, the target data file information is packaged into a task package according to the priority of the target data file information, and the task package is written into the corresponding priority queue, including:

[0014] According to the priority of each target data file information, at least one target data file information of the same priority is packaged into a task package;

[0015] Write task packets of different priorities into the corresponding priority queues respectively.

[0016] As an improvement to the above solution, the priority queue includes a first priority queue and a second priority queue, and the priority of the first priority queue is higher than the priority of the second priority queue. Then, according to the target quantity, a corresponding number of task packets are pulled from the priority queue to parse the target data in the task packet, including:

[0017] Pulling a corresponding number of task packets from the first priority queue according to the target number to parse the target data in the task packets;

[0018] If the first priority queue is empty, a corresponding number of task packets are pulled from the second priority queue to parse the target data in the task packets.

[0019] As an improvement to the above solution, the method further includes:

[0020] One or more threads are configured to run in parallel according to the number of OMCs and the amount of target data.

[0021] An embodiment of the present invention further provides an adaptive data acquisition device, comprising:

[0022] The data detection module is used to traverse the OMC list information one by one to obtain the target data file information;

[0023] A data packaging module, configured to package the target data file information into a task package according to the priority of the target data file information, and write the task package into a corresponding priority queue;

[0024] A task correction module, configured to adaptively correct a target number of task packets pulled from the priority queue according to a processing time of data analysis;

[0025] The data acquisition module is used to pull a corresponding number of task packets from the priority queue according to the target number, so as to parse the target data in the task packets.

[0026] An embodiment of the present invention further provides a terminal device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements any of the above-mentioned adaptive data collection methods when executing the computer program.

[0027] An embodiment of the present invention further provides a computer-readable storage medium, which includes a stored computer program. When the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the above-mentioned adaptive data acquisition methods.

[0028] An embodiment of the present invention further provides a computer program product, which includes a computer program or computer instructions. When the computer program or the computer instructions are executed by a processor, any one of the above-mentioned adaptive data acquisition methods is implemented.

[0029] Compared with the prior art, the embodiments of the present invention provide an adaptive data acquisition method, device, equipment, medium and computer program product with the following advantages: obtaining target data file information by traversing OMC list information one by one; packaging the target data file information into task packages according to the priority of the target data file information, and writing the task packages into corresponding priority queues; adaptively adjusting the target number of task packages pulled from the priority queue according to the processing time of data parsing; pulling a corresponding number of task packages from the priority queue according to the target number to parse the target data in the task packages. The embodiments of the present invention distinguish data acquisition and parsing into two independent steps, adaptively adjusting the amount of data to be collected according to the time of data parsing, thereby minimizing the imbalance of data collection and parsing tasks caused by the uneven distribution of source data in the data acquisition and parsing steps, thereby causing excessive data processing delays, data backlogs, data loss, etc.; at the same time, by configuring different data delay priority queues, it can flexibly adapt to the needs of various different delay priorities, so that resource utilization is maintained at the highest reuse rate, that is, the highest resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a flow chart of a preferred embodiment of an adaptive data acquisition method provided by the present invention;

[0031] Figure 2 This is a flow chart of task adaptive balancing configuration in an adaptive data acquisition method provided by the present invention;

[0032] Figure 3 It is a logic diagram of a configurable priority data source information queue in an adaptive data acquisition method provided by the present invention;

[0033] Figure 4 It is a structural diagram of a preferred embodiment of an adaptive data acquisition device provided by the present invention;

[0034] Figure 5 It is a structural diagram of a preferred embodiment of a terminal device provided by the present invention. DETAILED DESCRIPTION

[0035] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0036] See also Figure 1 , Figure 11 is a flow chart of a preferred embodiment of an adaptive data acquisition method provided by the present invention. The adaptive data acquisition method includes:

[0037] S1, traverse the OMC list information one by one to obtain the target data file information;

[0038] S2, packaging the target data file information into a task package according to the priority of the target data file information, and writing the task package into a corresponding priority queue;

[0039] S3, adaptively modifying the target number of task packets pulled from the priority queue according to the processing time of the data analysis;

[0040] S4, pulling a corresponding number of task packets from the priority queue according to the target number, to parse the target data in the task packets.

[0041] It's important to note that network optimization software application systems or platforms often require the collection, analysis, and presentation of data from live networks (mobile communication networks, including access and core networks) to assist in assessing network operational status or locating network equipment faults, network planning / design solutions, and equipment parameter configuration flaws. Live network data includes operational parameters, test data, and performance data. These data types are diverse, with complex data fields and large data volumes. In addition to operational parameters, the data sources required for mobile network optimization applications primarily include performance data (often referred to as PM data), measurement reports (often referred to as MR data), and minimized drive tests (often referred to as MDT data). Operators typically define enterprise-level specifications for PM, MR, and MDT data, clarifying information such as data names, data types, data meanings, data value ranges, and output file formats. These specifications undergo an iterative update process. However, different equipment vendors in live networks vary in their implementation progress and compliance (consistency) with these enterprise-level specifications on their equipment. When equipment vendors output data, especially PM, MR, and MDT data, while generally complying with enterprise-level specifications, various output file formats coexist, including XML and CSV file formats; file compression encoding methods include zip and gzip; and secondary compression / packaging may occur. For example, a gzip file contains hundreds or even thousands of gzip files.

[0042] Another point worth noting is that even within the same province, different network management systems (OMCs) from the same equipment vendor may output different file formats. PM, MR, and MDT data are generated in batches every 15 minutes. To avoid data backlogs and loss, the latency for collecting and parsing each batch of PM, MR, and MDT data should not exceed 15 minutes. Typical upper-layer applications in wireless network optimization require varying data granularity and latency. Granularities include 15 minutes, hours, and days. Some applications are more sensitive to data latency (for example, requiring a processing latency of 20-30 minutes for 15-minute granularity data), while others are less sensitive. For collection and parsing tasks, the only available prior knowledge is the OMC FTP information, including the server's IP address, port number, data type, and the root path for data storage files. There is no prior knowledge of the number of files to be processed under this file path or their size or scale. Moreover, in the existing network, the number of OMC FTPs, the base stations managed (or accommodated) by each OMC, and the number of base stations are constantly changing. It is difficult to obtain prior knowledge such as the files to be processed and the size of the files to be processed on the OMC FTP through a one-time trial.

[0043] Based on this, an embodiment of the present invention provides an adaptive data acquisition method for use in an adaptive data acquisition system. The system includes a source data metadata detection module, a source data acquisition module, and a data parsing module. This embodiment of the present invention clearly distinguishes data acquisition and parsing into two independent stages, further subdividing the data acquisition phase into a data metadata detection process and a data acquisition (pull or download) process. The data metadata detection process and the data acquisition process are connected and isolated by a data metadata detection result queue, i.e., a priority queue. The source data metadata detection process traverses the OMC list information one by one to obtain target data file information. The source data metadata detection process can be configured to run one or more threads in parallel based on the number of OMCs and the total amount of data to be collected. After obtaining the target data file information, the source data metadata detection process packages the target data file information into a task package based on its priority and writes the task package to the corresponding priority queue. The source data acquisition process periodically (which can coincide with the downstream data parsing cycle) reads the source data metadata detection result queue to obtain complete information about the target data to be processed. Based on this information, it collects / downloads the corresponding file for use in the data parsing stage. When the source data collection process reads the source data metadata detection result queue, the number of task packages to be downloaded is determined by the processing time value dynamically corrected and fed back by the data parsing phase. The data parsing process dynamically adjusts the number of files to be pulled for the next batch (or batches over the next period) based on the processing time of the current batch of files, aiming to maintain (nearly) the same dynamic average execution time for tasks across distributed parsing nodes. The data collection process executes batch data pulls from the OMC FTP server based on the dynamically fed-back number of files to be pulled, handing the collected data for this batch to the data parsing process for processing. The source data collection process can configure one or more threads to run in parallel based on the number of OMCs and the total amount of data to be collected. After successfully parsing and processing this batch of data, the data parsing process feeds metadata for this batch of processed data to the source data metadata detection result queue. The source data metadata detection result queue marks the metadata of the successfully processed data files as successfully processed. To prevent queue overflow, the source data metadata detection result queue can periodically delete successfully processed cloud file information while ensuring that unparsed files are not processed repeatedly. The data parsing process dynamically adjusts the parameter value of the number of files pulled in each batch according to the processing time of this batch of files, and feeds this value back to the source data collection process.

[0044] The embodiment of the present invention adapts to the common imbalanced distribution of upstream data in the process of data collection and analysis, automatically learns and obtains the imbalance of data distribution, balances the resource utilization of data collection and analysis servers, minimizes data processing delay, and avoids data backlog or loss; at the same time, by configuring different data delay priority queues, it meets the differences in the delay requirements of different applications, can flexibly adapt to the needs of a variety of different delay priorities, and at the same time ensure that resource utilization remains at the highest multiplexing degree, that is, the highest resource utilization. The embodiment of the present invention eliminates to the greatest extent the imbalanced distribution of tasks in the collection and analysis links caused by the uneven distribution of source data in the data collection and analysis links, which in turn leads to phenomena such as excessive data processing delay, data backlog, and data loss.

[0045] In another preferred embodiment, the number of the priority queue is at least one, and the method further includes:

[0046] The number of priority queues, the priority of each priority queue, and the data object corresponding to each priority are determined according to a preset delay requirement.

[0047] Specifically, in this embodiment of the present invention, the number of priority queues is at least one. The priority configuration module can configure two or more priority queues based on the application's requirements for latency and other characteristics. The priority configuration module can flexibly configure the basis and criteria for determining priority and send them to the source data metadata detection module.

[0048] The embodiment of the present invention introduces different priority queues into the source data metadata detection result queue, thereby effectively guaranteeing the delay requirements of delay-sensitive applications while ensuring high utilization of software and hardware resources.

[0049] In another preferred embodiment, the OMC list information includes the server's IP address, port, data type, data storage file root path, login account, and login password, and traversing the OMC list information one by one to obtain target data file information includes:

[0050] The OMC list information is traversed one by one according to a preset period, the data storage file root path of the target data is found, and the target data file information is obtained; wherein the target data file information includes the name and size of the file.

[0051] Specifically, in this embodiment of the present invention, the OMC list information includes the server's IP address, port number, data type, data storage file root path, login account, and password. When traversing the OMC list information to obtain target data file information, the OMC list information is traversed one by one according to a preset period, the data storage file root path of the target data (e.g., MR and MDT data) is searched, and the target data file information is obtained. The target data file information includes the file name and size. The preset period is no longer than the data generation period. For example, for MR data, this period is no longer than 15 minutes.

[0052] In another preferred embodiment, the step of packaging the target data file information into a task package according to the priority of the target data file information and writing the task package into a corresponding priority queue includes:

[0053] According to the priority of each target data file information, at least one target data file information of the same priority is packaged into a task package;

[0054] Write task packets of different priorities into the corresponding priority queues respectively.

[0055] Specifically, after the source data metadata detection process acquires target data file information, at least one target data file of the same priority level is packaged into a task package based on the priority level of each target data file. If there is only one priority queue, all task packages are written to that priority queue. If there are multiple priority queues of different priorities, task packages of different priorities are written to their corresponding priority queues.

[0056] In another preferred embodiment, the priority queue includes a first priority queue and a second priority queue, and the priority of the first priority queue is higher than the priority of the second priority queue, then pulling a corresponding number of task packets from the priority queue according to the target number to parse the target data in the task packets includes:

[0057] Pulling a corresponding number of task packets from the first priority queue according to the target number to parse the target data in the task packets;

[0058] If the first priority queue is empty, a corresponding number of task packets are pulled from the second priority queue to parse the target data in the task packets.

[0059] Specifically, in an embodiment of the present invention, the priority queue includes a first priority queue and a second priority queue, and the priority of the first priority queue is higher than the priority of the second priority queue, that is, the first priority queue is a high priority queue and the second priority queue is a low priority queue. When the source data acquisition process reads the source data metadata detection result queue, a corresponding number of task packets are preferentially pulled from the first priority queue to parse the target data in the task packets. Only when the first priority queue is empty, a corresponding number of task packets are pulled from the second priority queue to parse the target data in the task packets.

[0060] In another preferred embodiment, the method further comprises:

[0061] One or more threads are configured to run in parallel according to the number of OMCs and the amount of target data.

[0062] Specifically, in the embodiment of the present invention, in both the source data metadata detection process and the source data collection process, one or more threads may be configured to run in parallel according to the number of OMCs and the total amount of data to be collected.

[0063] The embodiment of the present invention adopts an adaptive closed-loop feedback mechanism, which avoids the narrow adaptation scenarios and inability to adapt to the dynamic fluctuations of business volume caused by artificially determining the number of files to be processed in a single batch; it can maximize the balancing of distributed, multi-threaded data collection and analysis tasks, and achieve the lowest data processing delay while maximizing resource utilization; it can flexibly configure different levels of priority queues according to the different delay requirements of upper-layer applications to meet the delay requirements of each application; it avoids the idleness of some resources and the backlog of some resource tasks caused by task imbalance, which leads to excessive data processing delay or even data loss; it has natural horizontal scalability and is adaptable to various data processing links and scenarios.

[0064] For example, see Figure 2 and Figure 3 , Figure 2 This is a flow chart of task adaptive balancing configuration in an adaptive data acquisition method provided by the present invention. Figure 3 This is a logical diagram of a configurable priority data source information queue in an adaptive data collection method provided by the present invention. The operating steps of the embodiment of the present invention are as follows:

[0065] Step 0A: The priority configuration module defines the number of priority queues and the priority of each priority queue based on the different requirements of applications for characteristics such as latency.

[0066] Step 0B: The priority configuration module defines the number of priority queues and corresponding data objects based on the different requirements of applications for characteristics such as latency, and synchronizes the corresponding information with the source data metadata detection module;

[0067] Step 1: The source data metadata detection process periodically traverses the given OMC list one by one to find the complete path where the target data is stored and obtains the name and size information of the target data file.

[0068] Step 2: After the source data metadata detection process obtains the target data file information, one or more original file information is packaged into a (collection / parsing) task package, and the task package information is written into the source data metadata detection result queue.

[0069] Step 2A: After the source data metadata detection process obtains the target data file information, it packages one or more original file information of object files of different priorities into one (acquisition / parsing) task package, and writes the task package information into different priority queues in the source data metadata detection result queue (Step 2 can be regarded as a special case of Step 2A, that is, the case where there is only one priority queue).

[0070] Step 3: The source data collection process periodically reads the source data metadata detection result queue to obtain complete information about the data to be processed. When the source data collection process reads the source data metadata detection result queue, the number of task packages to be downloaded is determined by the processing time value dynamically corrected and fed back by the data analysis stage (Step 7).

[0071] Step 3A: The difference from Step 3 is that the information in the high-priority queue in the source data metadata detection result queue is obtained first for processing. The metadata in the low-priority queue is processed only when the high-priority queue is empty (Step 3 can be regarded as a special case of Step 3A, that is, the case with only one priority queue).

[0072] Step 4: Based on this information, collect / download the corresponding files for use in the data analysis phase.

[0073] Step 5: The data collection process delivers the data collected in this batch to the data analysis process for processing.

[0074] Step 6: After successfully parsing the batch of data, the data parsing process sends the metadata for this batch of data to the source data metadata detection result queue. The source data metadata detection result queue marks the successfully processed data file metadata as successfully processed. To prevent queue overflow, the source data metadata detection result queue periodically deletes successfully processed cloud file information while ensuring that unparsed files are not processed repeatedly.

[0075] Step 7: The data parsing process dynamically adjusts the parameter value of the number of files pulled in each batch according to the processing time of this batch of files, and feeds this value back to the source data collection process.

[0076] Correspondingly, the present invention also provides an adaptive data acquisition device that can implement all the processes of the adaptive data acquisition method in the above embodiment.

[0077] See also Figure 4 , Figure 4 1 is a schematic structural diagram of a preferred embodiment of an adaptive data acquisition device provided by the present invention. The adaptive data acquisition device comprises:

[0078] The data detection module 401 is used to traverse the OMC list information one by one to obtain the target data file information;

[0079] A data packaging module 402 is configured to package the target data file information into a task package according to the priority of the target data file information, and write the task package into a corresponding priority queue;

[0080] The task correction module 403 is used to adaptively correct the target number of task packets pulled from the priority queue according to the processing time of the data analysis;

[0081] The data collection module 404 is configured to pull a corresponding number of task packets from the priority queue according to the target quantity, so as to parse the target data in the task packets.

[0082] Preferably, the number of the priority queue is at least one, and the apparatus further comprises a priority configuration module, configured to:

[0083] The number of priority queues, the priority of each priority queue, and the data object corresponding to each priority are determined according to a preset delay requirement.

[0084] Preferably, the OMC list information includes the server's IP address, port, data type, data storage file root path, login account, and login password. The data detection module 401 is specifically configured to:

[0085] The OMC list information is traversed one by one according to a preset period, the data storage file root path of the target data is found, and the target data file information is obtained; wherein the target data file information includes the name and size of the file.

[0086] Preferably, the data packaging module 402 includes:

[0087] The packaging unit 412 is configured to package at least one target data file information of the same priority into a task package according to the priority of each target data file information;

[0088] The writing unit 422 is used to write task packets of different priorities into corresponding priority queues respectively.

[0089] Preferably, the priority queue includes a first priority queue and a second priority queue, and the priority of the first priority queue is higher than the priority of the second priority queue, then the data acquisition module is specifically used to:

[0090] Pulling a corresponding number of task packets from the first priority queue according to the target number to parse the target data in the task packets;

[0091] If the first priority queue is empty, a corresponding number of task packets are pulled from the second priority queue to parse the target data in the task packets.

[0092] Preferably, the device is further used for:

[0093] One or more threads are configured to run in parallel according to the number of OMCs and the amount of target data.

[0094] In specific implementation, the working principle, control process and technical effects achieved by the adaptive data acquisition device provided in the embodiment of the present invention are the same as those of the adaptive data acquisition method in the above embodiment, and will not be repeated here.

[0095] See also Figure 5 , Figure 5 1 is a schematic diagram of the structure of a preferred embodiment of a terminal device provided by the present invention. The terminal device includes a processor 501, a memory 502, and a computer program stored in the memory 502 and configured to be executed by the processor 501. When the processor 501 executes the computer program, it implements the adaptive data transmission method described in any of the above embodiments.

[0096] Preferably, the computer program can be divided into one or more modules / units (e.g., computer program 1, computer program 2, ...), which are stored in the memory 502 and executed by the processor 501 to implement the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program in the terminal device.

[0097] The processor 501 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor, or the processor 501 can be any conventional processor. The processor 501 is the control center of the terminal device, and uses various interfaces and lines to connect the various parts of the terminal device.

[0098] The memory 502 mainly includes a program storage area and a data storage area. The program storage area can store an operating system, at least one application required for a function, and the data storage area can store related data. In addition, the memory 502 can be a high-speed random access memory or a non-volatile memory, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, and a Flash Card. Alternatively, the memory 502 can be other volatile solid-state memory devices.

[0099] It should be noted that the above terminal device may include, but is not limited to, a processor and a memory. Those skilled in the art will understand that Figure 5 The structural diagram is only an example of the above-mentioned terminal device and does not constitute a limitation on the above-mentioned terminal device. It may include more or fewer components than shown in the figure, or combine certain components, or different components.

[0100] An embodiment of the present invention further provides a computer-readable storage medium, which includes a stored computer program. When the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the adaptive data acquisition method described in any of the above embodiments.

[0101] An embodiment of the present invention further provides a computer program product, which includes a computer program or computer instructions. When the computer program or computer instructions are executed by a processor, the adaptive data acquisition method described in any of the above embodiments is implemented.

[0102] Embodiments of the present invention provide an adaptive data acquisition method, apparatus, device, medium, and computer program product. These methods obtain target data file information by traversing OMC list information one by one; package the target data file information into task packets based on the priority of the target data file information, and write the task packets into corresponding priority queues; adaptively adjust the target number of task packets pulled from the priority queue based on the processing time of data parsing; and pull a corresponding number of task packets from the priority queue based on the target number to parse the target data in the task packets. Embodiments of the present invention divide data acquisition and parsing into two independent steps, adaptively adjusting the amount of data to be collected based on the duration of data parsing, thereby minimizing the imbalance in the distribution of tasks in the data acquisition and parsing steps caused by the uneven distribution of source data, which in turn leads to excessive data processing delays, data backlogs, data loss, and the like. Furthermore, by configuring different data delay priority queues, the method can flexibly adapt to the needs of various different delay priorities, maintaining resource utilization at the highest reuse rate, i.e., the highest resource utilization.

[0103] It should be noted that the system embodiment described above is merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the system embodiment provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art can understand and implement the present invention without inventive work.

[0104] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. An adaptive data acquisition method, characterized in that: include: Traverse the OMC list information of the network management equipment one by one to obtain the target data file information; Packing the target data file information into a task package according to the priority of the target data file information, and writing the task package into a corresponding priority queue; Adaptively modifying the target number of task packets pulled from the priority queue according to the processing time of data analysis; Pulling a corresponding number of task packets from the priority queue according to the target number to parse the target data in the task packets; The step of packaging the target data file information into a task package according to the priority of the target data file information and writing the task package into a corresponding priority queue includes: According to the priority of each target data file information, at least one target data file information of the same priority is packaged into a task package; Write task packets of different priorities into the corresponding priority queues respectively.

2. The adaptive data acquisition method according to claim 1, wherein: The number of the priority queue is at least one, and the method further includes: The number of priority queues, the priority of each priority queue, and the data object corresponding to each priority are determined according to a preset delay requirement.

3. The adaptive data acquisition method according to claim 2, wherein: The OMC list information includes the server's IP address, port, data type, data storage file root path, login account, and login password. The step of traversing the OMC list information one by one to obtain target data file information includes: The OMC list information is traversed one by one according to a preset period, the data storage file root path of the target data is found, and the target data file information is obtained; wherein the target data file information includes the name and size of the file.

4. The adaptive data acquisition method according to claim 3, wherein: The priority queue includes a first priority queue and a second priority queue, and the priority of the first priority queue is higher than the priority of the second priority queue, then pulling a corresponding number of task packets from the priority queue according to the target number to parse the target data in the task packets includes: Pulling a corresponding number of task packets from the first priority queue according to the target number to parse the target data in the task packets; If the first priority queue is empty, a corresponding number of task packets are pulled from the second priority queue to parse the target data in the task packets.

5. The adaptive data acquisition method according to claim 4, wherein: The method further comprises: One or more threads are configured to run in parallel according to the number of OMCs and the amount of target data.

6. An adaptive data acquisition device, characterized in that: include: The data detection module is used to traverse the OMC list information one by one to obtain the target data file information; A data packaging module, configured to package the target data file information into a task package according to the priority of the target data file information, and write the task package into a corresponding priority queue; A task correction module, configured to adaptively correct a target number of task packets pulled from the priority queue according to a processing time of data analysis; A data acquisition module is used to pull a corresponding number of task packets from the priority queue according to the target number, so as to parse the target data in the task packets; Wherein, the data packaging module includes: a packaging unit, configured to package at least one target data file information of the same priority into a task package according to the priority of each target data file information; The writing unit is used to write task packets of different priorities into corresponding priority queues respectively.

7. A terminal device, characterized in that: The device comprises a processor and a memory, wherein a computer program is stored in the memory and the computer program is configured to be executed by the processor, and when the processor executes the computer program, the adaptive data acquisition method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the device where the computer-readable storage medium is located executes the computer program, the adaptive data acquisition method according to any one of claims 1 to 5 is implemented.

9. A computer program product, characterized in that The computer program product comprises a computer program or computer instructions, and when the computer program or the computer instructions are executed by a processor, the adaptive data acquisition method according to any one of claims 1 to 5 is implemented.

Images