Self-adaptive data acquisition method, device, equipment, medium and computer program product

Abstract

The invention discloses a self-adaptive data acquisition method, device and equipment, a medium and a computer program product. The method comprises the following steps: traversing OMC list information one by one to obtain target data file information; packaging the target data file information into a task package according to the priority of the target data file information, and writing the task package into a corresponding priority queue; adaptively correcting the target number of the pulled task packets according to the processing duration of data analysis; and according to the target number, pulling a corresponding number of task packets from the priority queue to analyze the target data in the task packets. According to the method, data acquisition and analysis are divided into two mutually independent links, and the amount of data to be acquired is adaptively corrected according to the duration of data analysis, so that the resource utilization rate of a data acquisition and analysis server is balanced, the data processing delay is minimized, and the phenomenon of data backlog or loss is avoided; and meanwhile, different data delay priority queues are configured, so that the requirements of various different delay priorities can be flexibly met, and the resource utilization rate is improved.

Description

Technical Field

[0001] The present invention relates to the field of data acquisition technology, and in particular to an adaptive data acquisition method, device, equipment, medium and computer program product. Background Art

[0002] The 4G, 5G MR and MDT data of different equipment vendors in the existing network usually follow the wireless network management specifications and are output in XML or CSV file format. The smallest file unit usually contains 15 minutes of measurement data of a base station (4G eNodeB or 5G gNodeB). One OMC (network management equipment) usually manages hundreds or thousands of 4G or 5G base stations. Some equipment vendors will aggregate the MR and MDT data of multiple OMCs on an OMC FTP server. There is a phenomenon of 1-time and 2-time compression of MR and MDT data at the same time, resulting in a single file size on a single OMC ranging from tens or hundreds of kB to 1GB. The number of data files under a single folder path on the server can range from tens or hundreds to more than 10,000. This makes it difficult to evenly distribute the collection and analysis tasks to the servers that run the collection and analysis tasks concurrently when collecting and analyzing 4G, 5G MR and MDT data in a distributed / parallel manner, resulting in uneven distribution of collection tasks and inefficient resource utilization. When the task distribution is severely unbalanced, some collection servers will not be able to meet the requirements of the processing delay for each batch of data, which will cause excessive data delay or even loss. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide an adaptive data collection method, device, equipment, medium and computer program product, which divides data collection and analysis into two independent links, and adaptively corrects the amount of data to be collected according to the duration of data analysis, so as to balance the resource utilization of data collection and analysis servers, minimize data processing delay, and avoid data backlog or loss; at the same time, by configuring different data delay priority queues, it can flexibly adapt to the needs of various delay priorities and improve resource utilization.

[0004] In order to achieve the above object, an embodiment of the present invention provides an adaptive data collection method, comprising:

[0005] Traverse the OMC list information one by one to obtain the target data file information;

[0006] According to the priority of the target data file information, the target data file information is packaged into a task package, and the task package is written into a corresponding priority queue;

[0007] Adaptively modifying the target number of task packages pulled from the priority queue according to the processing time of the data analysis;

[0008] A corresponding number of task packets are pulled from the priority queue according to the target number to parse the target data in the task packets.

[0009] As an improvement of the above solution, the number of the priority queue is at least one, and the method further includes:

[0010] The number of priority queues, the priority of each priority queue, and the data object corresponding to each priority are determined according to a preset delay requirement.

[0011] As an improvement of the above solution, the OMC list information includes the server's IP address, port, data type, data storage file root path, login account, and login password. The step of traversing the OMC list information one by one to obtain the target data file information includes:

[0012] The OMC list information is traversed one by one according to a preset period, the data storage file root path of the target data is found, and the target data file information is obtained; wherein the target data file information includes the name and size of the file.

[0013] As an improvement of the above solution, the target data file information is packaged into a task package according to the priority of the target data file information, and the task package is written into a corresponding priority queue, including:

[0014] According to the priority of each target data file information, packing at least one target data file information of the same priority into a task package;

[0015] Write task packets of different priorities into the corresponding priority queues respectively.

[0016] As an improvement of the above scheme, the priority queue includes a first priority queue and a second priority queue, and the priority of the first priority queue is higher than the priority of the second priority queue, then pulling a corresponding number of task packets from the priority queue according to the target number to parse the target data in the task packet includes:

[0017] Pulling a corresponding number of task packets from the first priority queue according to the target number to parse the target data in the task packets;

[0018] If the first priority queue is empty, a corresponding number of task packets are pulled from the second priority queue to parse the target data in the task packets.

[0019] As an improvement of the above solution, the method further includes:

[0020] One or more threads are configured to run in parallel according to the number of OMCs and the number of target data.

[0021] The embodiment of the present invention further provides an adaptive data acquisition device, comprising:

[0022] A data detection module is used to traverse the OMC list information one by one to obtain the target data file information;

[0023] A data packaging module, used for packaging the target data file information into a task package according to the priority of the target data file information, and writing the task package into a corresponding priority queue;

[0024] A task correction module, used for adaptively correcting the target number of task packets pulled from the priority queue according to the processing time of data analysis;

[0025] The data acquisition module is used to pull a corresponding number of task packets from the priority queue according to the target number, so as to parse the target data in the task packets.

[0026] An embodiment of the present invention further provides a terminal device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements any of the above-mentioned adaptive data collection methods when executing the computer program.

[0027] An embodiment of the present invention further provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the above-mentioned adaptive data collection methods.

[0028] An embodiment of the present invention further provides a computer program product, which includes a computer program or a computer instruction. When the computer program or the computer instruction is executed by a processor, the adaptive data collection method described in any one of the above items is implemented.

[0029] Compared with the prior art, the beneficial effects of an adaptive data acquisition method, device, equipment, medium and computer program product provided by the embodiment of the present invention are: obtaining target data file information by traversing the OMC list information one by one; packaging the target data file information into a task package according to the priority of the target data file information, and writing the task package into the corresponding priority queue; adaptively correcting the target number of task packages pulled from the priority queue according to the processing time of data analysis; pulling a corresponding number of task packages from the priority queue according to the target number to parse the target data in the task package. The embodiment of the present invention divides data acquisition and analysis into two independent links, and adaptively corrects the amount of data to be collected according to the time of data analysis, thereby eliminating the imbalance of task distribution in the acquisition and analysis links due to the imbalance of source data distribution in the data acquisition and analysis links to the greatest extent, thereby causing the phenomenon of excessive data processing delay, data backlog, data loss, etc.; at the same time, by configuring different data delay priority queues, it can flexibly adapt to the needs of various different delay priorities, so that resource utilization is maintained at the highest reuse degree, that is, the highest resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a flow chart of a preferred embodiment of an adaptive data collection method provided by the present invention;

[0031] Figure 2 It is a task adaptive balancing configuration flow chart in an adaptive data acquisition method provided by the present invention;

[0032] Figure 3 It is a logic diagram of a configurable priority data source information queue in an adaptive data collection method provided by the present invention;

[0033] Figure 4 It is a structural schematic diagram of a preferred embodiment of an adaptive data acquisition device provided by the present invention;

[0034] Figure 5 It is a structural schematic diagram of a preferred embodiment of a terminal device provided by the present invention. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] See also Figure 1 , Figure 11 is a flow chart of a preferred embodiment of an adaptive data collection method provided by the present invention. The adaptive data collection method comprises:

[0037] S1, traverse the OMC list information one by one to obtain the target data file information;

[0038] S2, packaging the target data file information into a task package according to the priority of the target data file information, and writing the task package into a corresponding priority queue;

[0039] S3, adaptively modifying the target number of task packets pulled from the priority queue according to the processing time of the data analysis;

[0040] S4, pulling a corresponding number of task packets from the priority queue according to the target number to parse the target data in the task packets.

[0041] It should be noted that in network optimization software application systems or platforms, it is usually necessary to collect, analyze and present data from the existing network (mobile communication network, including access network and core network) to assist in evaluating the network operation status or locating network equipment failures, network planning / design solutions / equipment parameter configuration defects. Existing network data includes engineering parameters, tests, performance, etc. These data have the characteristics of many types, complex data fields, and huge data volume. In addition to engineering parameters, the data sources involved in mobile network network optimization applications mainly include performance data (usually referred to as PM data), measurement reports (usually referred to as MR data), and minimized drive tests (usually referred to as MDT data). For PM, MR and MDT data, operators usually define enterprise-level specifications to clarify the name, data type, data meaning, data value range, output file format and other information of the data. The above specifications also have an iterative update process. In the existing network, different equipment vendors have different progress in implementing the above enterprise-level specifications on their equipment and their compliance (consistency) with the specifications. When equipment vendors output data, especially PM, MR and MDT data, while basically complying with enterprise-level specifications, there are various coexisting output file formats, including xml and csv file formats; file compression encoding methods include zip and gzip; and there is secondary compression / packaging, for example: a gzip file includes hundreds or even thousands of gzip files.

[0042] Another point that needs to be explained is that the same equipment vendor has different output file formats on different network management devices (OMC) in the existing network (even in the same province). PM, MR and MDT data are generated in batches every 15 minutes. To avoid data backlog and loss, the delay of each batch of PM, MR and MDT data collection and analysis should not exceed 15 minutes. Typical upper-layer applications in the field of wireless network optimization require different data time granularity and data delay. The time granularity is: 15 minutes, hours and days. As for data delay, some applications are more sensitive (such as requiring the data processing delay of 15-minute time granularity data to be within 20 to 30 minutes), while some applications are not sensitive to data delay. For collection and analysis tasks, the only prior knowledge available is OMC FTP information, including the server's IP address, port, data type, and data storage file root path. There is no prior knowledge about how many files to be processed under this file path and the size / scale of the files to be processed. Moreover, in the existing network, the number of OMC FTPs, the base stations managed (or accommodated) by each OMC, and the number of base stations are in constant change, making it difficult to obtain prior knowledge such as the files to be processed and the size of the files to be processed on the OMC FTP through a one-time trial.

[0043] Based on this, an embodiment of the present invention provides an adaptive data acquisition method, which is applied to an adaptive data acquisition system, and the system includes a source data meta information detection module, a source data acquisition module, and a data parsing module. The embodiment of the present invention clearly distinguishes data acquisition and parsing into two independent links, and then subdivides the data acquisition stage into a data meta information detection process and a data acquisition (pulling or downloading) process. The data meta information detection process and the data acquisition process are connected and isolated through a data meta information detection result queue, that is, a priority queue. The source data meta information detection process traverses the OMC list information one by one to obtain the target data file information. Among them, the source data meta information detection process can configure one or more threads to run in parallel according to the number of OMCs and the total amount of data to be collected. After the source data meta information detection process obtains the target data file information, the target data file information is packaged into a task package according to the priority of the target data file information, and the task package is written to the corresponding priority queue. The source data acquisition process periodically (which can be consistent with the downstream data parsing cycle) reads the source data meta information detection result queue to obtain the complete information of the target data to be processed, and collects / downloads the corresponding file for use in the data parsing link based on this information. When the source data collection process reads the source data metadata detection result queue, the number of task packages to be downloaded is determined by the processing time value dynamically corrected and fed back by the data analysis link. The data analysis process dynamically adjusts the number of files pulled by the next batch (or batches in the next period of time) of the collection link according to the processing time of the current batch of files, with the goal of keeping the dynamic average time of task execution of each distributed analysis node (close to) consistent. The data collection process executes the task of pulling data in batches from the OMC FTP according to the number of files pulled in a single time dynamically fed back, and delivers the data collected in this batch to the data analysis process for processing. Among them, the source data collection process can configure one or more threads to run in parallel according to the number of OMCs and the total amount of data to be collected. After the data analysis process successfully parses and processes the current batch of data, it feeds back the metadata of the processed data in this batch to the source data metadata detection result queue. The source data metadata detection result queue marks the metadata of the data files that have been successfully obtained as successfully processed. To prevent queue overflow, the source data metadata detection result queue can regularly delete the cloud file information that has been successfully processed on the basis of ensuring that the files to be analyzed are not processed repeatedly. The data parsing process dynamically adjusts the parameter value of the number of files pulled in each batch according to the processing time of this batch of files, and feeds this value back to the source data collection process.

[0044] The embodiment of the present invention adapts to the common imbalanced distribution of upstream data in the process of data collection and analysis, automatically learns and obtains the imbalance of data distribution, balances the resource utilization of data collection and analysis servers, minimizes data processing delays, and avoids data backlogs or losses; at the same time, by configuring different data delay priority queues, it meets the differences in the requirements of different applications in terms of delay, can flexibly adapt to the requirements of a variety of different delay priorities, and at the same time ensure that resource utilization is maintained at the highest reuse degree, that is, the highest resource utilization. The embodiment of the present invention eliminates to the greatest extent the imbalanced distribution of tasks in the data collection and analysis links caused by the imbalanced distribution of source data in the data collection and analysis links, which in turn leads to excessive data processing delays, data backlogs, data losses, and other phenomena.

[0045] In another preferred embodiment, the number of the priority queue is at least one, and the method further comprises:

[0046] The number of priority queues, the priority of each priority queue, and the data object corresponding to each priority are determined according to a preset delay requirement.

[0047] Specifically, in the embodiment of the present invention, the number of priority queues is at least one, and the number of priority queues can be configured by the priority configuration module to be 2 or more according to different requirements of applications for characteristics such as latency. The basis and criteria for judging the priority level can be flexibly configured by the priority configuration module and sent to the source data metadata detection module.

[0048] The embodiment of the present invention introduces queues of different priorities into the source data metadata detection result queue, thereby effectively guaranteeing the delay requirements of delay-sensitive applications while ensuring high utilization rates of software and hardware resources.

[0049] In another preferred embodiment, the OMC list information includes the IP address, port, data type, data storage file root path, login account and login password of the server, and the step of traversing the OMC list information one by one to obtain the target data file information includes:

[0050] The OMC list information is traversed one by one according to a preset period, the data storage file root path of the target data is found, and the target data file information is obtained; wherein the target data file information includes the name and size of the file.

[0051] Specifically, in the embodiment of the present invention, the OMC list information includes the IP address, port, data type, data storage file root path, login account and login password of the server. When traversing the OMC list information one by one to obtain the target data file information, the OMC list information is traversed one by one according to the preset period, the data storage file root path of the target data (such as MR and MDT data) is searched, and the target data file information is obtained. Among them, the target data file information includes the name and size of the file. The preset period is not greater than the data generation period, for example: for MR data, this period is not greater than 15 minutes.

[0052] In another preferred embodiment, the step of packaging the target data file information into a task package according to the priority of the target data file information, and writing the task package into a corresponding priority queue includes:

[0053] According to the priority of each target data file information, packing at least one target data file information of the same priority into a task package;

[0054] Write task packets of different priorities into the corresponding priority queues respectively.

[0055] Specifically, after the source data metadata detection process in the embodiment of the present invention obtains the target data file information, at least one target data file information of the same priority is packaged into a task package according to the priority of each target data file information. If there is only one priority queue, all task packages are written into the priority queue; if there are multiple priority queues with different priorities, task packages of different priorities are written into corresponding priority queues respectively.

[0056] In another preferred embodiment, the priority queue includes a first priority queue and a second priority queue, and the priority of the first priority queue is higher than the priority of the second priority queue, then pulling a corresponding number of task packets from the priority queue according to the target number to parse the target data in the task packet includes:

[0057] Pulling a corresponding number of task packets from the first priority queue according to the target number to parse the target data in the task packets;

[0058] If the first priority queue is empty, a corresponding number of task packets are pulled from the second priority queue to parse the target data in the task packets.

[0059] Specifically, in the embodiment of the present invention, the priority queue includes a first priority queue and a second priority queue, and the priority of the first priority queue is higher than the priority of the second priority queue, that is, the first priority queue is a high priority queue, and the second priority queue is a low priority queue. When the source data acquisition process reads the source data meta information detection result queue, the corresponding number of task packets are preferentially pulled from the first priority queue to parse the target data in the task packet. Only when the first priority queue is empty, the corresponding number of task packets are pulled from the second priority queue to parse the target data in the task packet.

[0060] In another preferred embodiment, the method further comprises:

[0061] One or more threads are configured to run in parallel according to the number of OMCs and the number of target data.

[0062] Specifically, in the embodiment of the present invention, in both the source data metadata detection process and the source data collection process, one or more threads may be configured to run in parallel according to the number of OMCs and the total amount of data to be collected.

[0063] The embodiment of the present invention adopts an adaptive closed-loop feedback mechanism, which avoids the situation where the number of files processed in a single batch is artificially determined, which leads to narrow adaptation scenarios and inability to adapt to scenarios with dynamic fluctuations in business volume; it can maximize the balancing of distributed, multi-threaded data acquisition and analysis tasks, and achieve the lowest data processing delay while maximizing resource utilization; it can flexibly configure priority queues of different levels according to the different delay requirements of upper-layer applications to meet the delay requirements of each application; it avoids the idleness of some resources and the backlog of some resource tasks caused by task imbalance, which leads to excessive data processing delay or even data loss; it has natural horizontal scalability and can adapt to a variety of data processing links and scenarios.

[0064] For example, see Figure 2 and Figure 3 , Figure 2 is a task adaptive balancing configuration flow chart in an adaptive data acquisition method provided by the present invention, Figure 3 This is a logical diagram of a configurable priority data source information queue in an adaptive data collection method provided by the present invention. The operation steps of the embodiment of the present invention are as follows:

[0065] Step 0A: The priority configuration module defines the number of priority queues and the priority of each priority queue according to the different requirements of applications for latency and other characteristics;

[0066] Step 0B: The priority configuration module defines the number of priority queues and corresponding data objects according to the different requirements of applications for characteristics such as latency, and synchronizes the corresponding information to the source data metadata detection module;

[0067] Step 1: The source data metadata detection process periodically traverses the given OMC list one by one to find the complete path where the target data is stored and obtain the name and size information of the target data file.

[0068] Step 2: After the source data metadata detection process obtains the target data file information, one or more original file information is packaged into one (collection / analysis) task package, and the task package information is written into the source data metadata detection result queue.

[0069] Step 2A: After the source data metadata detection process obtains the target data file information, one or more original file information of object files of different priorities are packaged into one (acquisition / parsing) task package, and the task package information is written into different priority queues in the source data metadata detection result queue (Step 2 can be regarded as a special case of Step 2A, that is, there is only one priority queue).

[0070] Step 3: The source data collection process periodically reads the source data metadata detection result queue to obtain complete information about the data to be processed. When the source data collection process reads the source data metadata detection result queue, the number of task packages to be downloaded is determined by the processing time value dynamically corrected and fed back by the data analysis link (Step 7).

[0071] Step 3A: The difference from Step 3 is that the information of the high priority queue in the source data meta information detection result queue is obtained first for processing, and the meta information in the low priority queue is processed only when the high priority queue is empty (Step 3 can be regarded as a special case of Step 3A, that is, there is only one priority queue).

[0072] Step 4: Based on this information, collect / download corresponding files for use in the data analysis phase.

[0073] Step 5: The data collection process delivers the data collected in this batch to the data analysis process for processing.

[0074] Step 6: After the data parsing process successfully parses and processes the batch of data, the metadata of the processed data in this batch is fed back to the source data metadata detection result queue. The source data metadata detection result queue marks the successfully processed data file metadata as successfully processed. To prevent queue overflow, the source data metadata detection result queue can regularly delete the successfully processed cloud file information on the basis of ensuring that the files to be parsed are not processed repeatedly.

[0075] Step 7: The data parsing process dynamically adjusts the parameter value of the number of files pulled in each batch according to the processing time of this batch of files, and feeds this value back to the source data collection process.

[0076] Correspondingly, the present invention also provides an adaptive data acquisition device, which can implement all the processes of the adaptive data acquisition method in the above embodiment.

[0077] See also Figure 4 , Figure 4 1 is a schematic diagram of a preferred embodiment of an adaptive data acquisition device provided by the present invention. The adaptive data acquisition device comprises:

[0078] The data detection module 401 is used to traverse the OMC list information one by one to obtain the target data file information;

[0079] A data packaging module 402 is used to package the target data file information into a task package according to the priority of the target data file information, and write the task package into a corresponding priority queue;

[0080] A task correction module 403 is used to adaptively correct the target number of task packets pulled from the priority queue according to the processing time of data analysis;

[0081] The data collection module 404 is used to pull a corresponding number of task packets from the priority queue according to the target number, so as to parse the target data in the task packets.

[0082] Preferably, the number of the priority queue is at least one, and the device further comprises a priority configuration module, which is used to:

[0083] The number of priority queues, the priority of each priority queue, and the data object corresponding to each priority are determined according to a preset delay requirement.

[0084] Preferably, the OMC list information includes the server's IP address, port, data type, data storage file root path, login account and login password. The data detection module 401 is specifically used for:

[0085] The OMC list information is traversed one by one according to a preset period, the data storage file root path of the target data is found, and the target data file information is obtained; wherein the target data file information includes the name and size of the file.

[0086] Preferably, the data packaging module 402 includes:

[0087] A packing unit 412 is used to pack at least one target data file information of the same priority into a task package according to the priority of each target data file information;

[0088] The writing unit 422 is used to write task packets of different priorities into corresponding priority queues respectively.

[0089] Preferably, the priority queue includes a first priority queue and a second priority queue, and the priority of the first priority queue is higher than the priority of the second priority queue, then the data acquisition module is specifically used to:

[0090] Pulling a corresponding number of task packets from the first priority queue according to the target number to parse the target data in the task packets;

[0091] If the first priority queue is empty, a corresponding number of task packets are pulled from the second priority queue to parse the target data in the task packets.

[0092] Preferably, the device is also used for:

[0093] One or more threads are configured to run in parallel according to the number of OMCs and the number of target data.

[0094] In specific implementation, the working principle, control process and technical effect achieved by the adaptive data acquisition device provided in the embodiment of the present invention are the same as those of the adaptive data acquisition method in the above embodiment, and will not be repeated here.

[0095] See also Figure 5 , Figure 5 1 is a schematic diagram of a preferred embodiment of a terminal device provided by the present invention. The terminal device includes a processor 501, a memory 502, and a computer program stored in the memory 502 and configured to be executed by the processor 501. When the processor 501 executes the computer program, the adaptive data transmission method described in any of the above embodiments is implemented.

[0096] Preferably, the computer program can be divided into one or more modules / units (such as computer program 1, computer program 2, ...), and the one or more modules / units are stored in the memory 502 and executed by the processor 501 to complete the present invention. The one or more modules / units can be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program in the terminal device.

[0097] The processor 501 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor, or the processor 501 can be any conventional processor. The processor 501 is the control center of the terminal device, and various parts of the terminal device are connected using various interfaces and lines.

[0098] The memory 502 mainly includes a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function, etc., and the data storage area can store related data, etc. In addition, the memory 502 can be a high-speed random access memory, or a non-volatile memory, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, and a flash card (Flash Card), etc., or the memory 502 can also be other volatile solid-state storage devices.

[0099] It should be noted that the above terminal device may include, but is not limited to, a processor and a memory. Those skilled in the art will understand that Figure 5 The structural diagram is only an example of the above-mentioned terminal device and does not constitute a limitation of the above-mentioned terminal device. It may include more or less components than shown in the figure, or combine certain components, or different components.

[0100] An embodiment of the present invention further provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the adaptive data collection method described in any of the above embodiments.

[0101] An embodiment of the present invention further provides a computer program product, which includes a computer program or computer instructions. When the computer program or the computer instructions are executed by a processor, the adaptive data collection method described in any of the above embodiments is implemented.

[0102] The embodiment of the present invention provides an adaptive data acquisition method, device, equipment, medium and computer program product, which obtains target data file information by traversing OMC list information one by one; according to the priority of the target data file information, the target data file information is packaged into a task package, and the task package is written into the corresponding priority queue; according to the processing time of data analysis, the target number of task packages pulled from the priority queue is adaptively corrected; according to the target number, a corresponding number of task packages are pulled from the priority queue to parse the target data in the task package. The embodiment of the present invention divides data collection and analysis into two independent links, and adaptively corrects the amount of data to be collected according to the time of data analysis, so as to eliminate the imbalance of task distribution in the data collection and analysis links caused by the imbalance of source data distribution in the data collection and analysis links to the greatest extent, thereby causing the phenomenon of excessive data processing delay, data backlog, data loss, etc.; at the same time, by configuring different data delay priority queues, it can flexibly adapt to the needs of various different delay priorities, so that resource utilization is maintained at the highest multiplexing degree, that is, the highest resource utilization.

[0103] It should be noted that the system embodiment described above is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the system embodiment provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art may understand and implement it without paying any creative effort.

[0104] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. An adaptive data collection method, characterized in that: include: Traverse the OMC list information of the network management equipment one by one to obtain the target data file information; According to the priority of the target data file information, the target data file information is packaged into a task package, and the task package is written into a corresponding priority queue; Adaptively modifying the target number of task packages pulled from the priority queue according to the processing time of the data analysis; A corresponding number of task packets are pulled from the priority queue according to the target number to parse the target data in the task packets.

2. The adaptive data collection method according to claim 1, characterized in that: The number of the priority queue is at least one, and the method further includes: The number of priority queues, the priority of each priority queue, and the data object corresponding to each priority are determined according to a preset delay requirement.

3. The adaptive data collection method according to claim 2, characterized in that: The OMC list information includes the server's IP address, port, data type, data storage file root path, login account, and login password. The step of traversing the OMC list information one by one to obtain target data file information includes: The OMC list information is traversed one by one according to a preset period, the data storage file root path of the target data is found, and the target data file information is obtained; wherein the target data file information includes the name and size of the file.

4. The adaptive data collection method according to claim 3, characterized in that: The step of packaging the target data file information into a task package according to the priority of the target data file information, and writing the task package into a corresponding priority queue includes: According to the priority of each target data file information, packing at least one target data file information of the same priority into a task package; Write task packets of different priorities into the corresponding priority queues respectively.

5. The adaptive data collection method according to claim 4, characterized in that: The priority queue includes a first priority queue and a second priority queue, and the priority of the first priority queue is higher than the priority of the second priority queue, then pulling a corresponding number of task packets from the priority queue according to the target number to parse the target data in the task packet includes: Pulling a corresponding number of task packets from the first priority queue according to the target number to parse the target data in the task packets; If the first priority queue is empty, a corresponding number of task packets are pulled from the second priority queue to parse the target data in the task packets.

6. The adaptive data collection method according to claim 5, characterized in that: The method further comprises: One or more threads are configured to run in parallel according to the number of OMCs and the number of target data.

7. An adaptive data acquisition device, characterized in that: include: A data detection module is used to traverse the OMC list information one by one to obtain the target data file information; A data packaging module, used for packaging the target data file information into a task package according to the priority of the target data file information, and writing the task package into a corresponding priority queue; A task correction module, used for adaptively correcting the target number of task packets pulled from the priority queue according to the processing time of data analysis; The data acquisition module is used to pull a corresponding number of task packets from the priority queue according to the target number, so as to parse the target data in the task packets.

8. A terminal device, characterized in that: It comprises a processor and a memory, wherein a computer program is stored in the memory and the computer program is configured to be executed by the processor, and when the processor executes the computer program, the adaptive data acquisition method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the device where the computer-readable storage medium is located executes the computer program, the adaptive data collection method according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program or a computer instruction, and when the computer program or the computer instruction is executed by a processor, the adaptive data collection method according to any one of claims 1 to 6 is implemented.

Images