Unified data acquisition method and system and data system
By scheduling data collection tasks and unified data format processing, the problems of data sources differences and conflicts in the new generation of cloud network operation systems are solved, unified data collection and management are realized, and data processing efficiency is improved.
Patent Information
- Application Number
- CN202410132613.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-01-31
- Publication Date
- 2025-08-01
AI Technical Summary
In the new generation of cloud network operation system, since various professional network management systems are horizontally connected, differences and conflicts in data sources are caused, affecting data quality and management efficiency.
By scheduling data acquisition tasks, performing data acquisition services, checking and processing of collected data in a unified data format, blocking differences in data sources, and realizing unified data collection and management.
It realizes unified data collection and management, blocks differences from different sources, improves the efficiency of data processing in the operation system, and avoids the problems of multiple repeated acquisitions and data inconsistencies.
Smart Images
Figure CN120407319A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of communication technologies, and particularly relates to a method and system for unified data collection and a data system. Background Art
[0002] In some related technologies, the data of a specific specialty is managed by a dedicated network management system, and the entire life cycle of the data from collection to application is carried out in the same set of dedicated systems.
[0003] In the new generation of cloud network operation systems, the network management systems of each specialty are horizontally integrated, and the collection, storage, and application of data are completed by different platforms. Data quality problems have become prominent, mainly manifested as differences and conflicts in the data of multiple specialties due to differences in manufacturers and equipment models of multiple data sources. Summary of the Invention
[0004] In an embodiment of the present disclosure, by scheduling the data collection service of the corresponding specialty of the data collection task to execute the data collection task, the initial data collected is verified and processed with a unified data format, and the data obtained after passing the verification and unified data format processing is used as the finally collected unified data, thereby shielding the differences in data sources, realizing unified data collection and management, and further providing a unified data service for external systems based on the unified data.
[0005] Some embodiments of the present disclosure provide a method for unified data collection, including: scheduling the data collection service of the corresponding specialty of the data collection task to execute the data collection task; verifying the first data collected by the data collection service; performing unified data format processing on the first data collected by the data collection service; and using the second data obtained after passing the verification and unified data format processing as the response data of the data collection task.
[0006] In some embodiments, the verifying the first data collected by the data collection service includes: when the first data includes data of a first information item obtained from a first source and a second source respectively, performing a consistency verification on the data of the first information item obtained from the first source and the data of the first information item obtained from the second source. If the data of the first information item obtained from the first source is consistent with the data of the first information item obtained from the second source, the consistency verification passes.
[0007] In some embodiments, performing unified data format processing on the first data collected by the data collection service includes: according to a unified data model, performing conversion processing on the first data collected by the data collection service, where the unified data model is formed based on the union of the data characteristics of all manufacturers.
[0008] In some embodiments, the unified data format processing of the first data collected by the data collection service includes: adding a source identifier to the first identifier data in the first data, and jointly using the source identifier and the first identifier data as the unique identifier for unified data format processing.
[0009] In some embodiments, the verification of the first data collected by the data collection service includes: verifying whether the first data collected by the data collection service is complete, including one or more of the entities, attributes, records, and fields required by the data collection task. If it is complete, the integrity verification passes.
[0010] In some embodiments, the verification of the first data collected by the data collection service includes: verifying whether the first data collected by the data collection service conforms to the preset specifications and preset ranges. If it conforms, the validity verification passes.
[0011] In some embodiments, the verification of the first data collected by the data collection service includes: comparing the current data volume of the first data collected by the data collection service with the historical data volume. If the difference between the current data volume and the historical data volume does not exceed the threshold, the volatility verification passes. If the difference between the current data volume and the historical data volume exceeds the threshold, a data volatility alarm is generated.
[0012] In some embodiments, it further includes: in the case where the verification fails, repeating the step of scheduling the data collection service of the corresponding specialty of the data collection task to execute the data collection task; verifying the third data repeatedly collected by the data collection service; and generating a data review alarm in the case where the verification of the third data fails.
[0013] In some embodiments, it further includes: monitoring the execution of the data collection task, and in the case where an execution exception is monitored, rescheduling the data collection service of the corresponding specialty of the data collection task to execute the data collection task.
[0014] In some embodiments, the specialties of the data collection service include one or more of a radio network, a bearer network, a core network, a transmission network, cloud resources, and network virtualization devices.
[0015] In some embodiments, the collected first data includes one or more of performance data, configuration data, alarm data, deep packet inspection data, terminal measurement report data, flow data, and network element control operations.
[0016] Some embodiments of the present disclosure propose a data unified collection system, including:
[0017] A task scheduling module, configured to schedule the data collection services corresponding to the specialties of the data collection tasks to execute the data collection tasks;
[0018] A data verification module, configured to verify the first data collected by the data collection service;
[0019] A unified processing module, configured to perform unified data format processing on the first data collected by the data collection service;
[0020] A service interface module, configured to use the second data obtained after passing verification and unified data format processing as the response data of the data collection task.
[0021] In some embodiments, it further includes: a collection monitoring module, configured to monitor the execution of the data collection task, and in the case of monitoring an abnormal execution, reschedule the data collection services corresponding to the specialties of the data collection task to execute the data collection task.
[0022] Some embodiments of the present disclosure propose a data unified collection system, including: a memory; and a processor coupled to the memory, the processor being configured to execute a data unified collection method based on instructions stored in the memory.
[0023] Some embodiments of the present disclosure propose a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the data unified collection method are implemented.
[0024] Some embodiments of the present disclosure propose a computer program product, including a computer program, and when the computer program is executed by a processor, the steps of the data unified collection method are implemented.
[0025] Some embodiments of the present disclosure propose a data system, including:
[0026] A data storage and sharing system, configured to query whether the required data is stored locally according to the data requirement. If the required data is stored locally, directly return the required data. If the required data is not stored locally, send the data requirement to the data unified collection system;
[0027] A data unified collection system, configured to judge whether the required data is periodically collected data according to the data requirement. If it is periodically collected data, return the required data stored locally to the data storage and sharing system. If it is not periodically collected data, establish a corresponding data collection task for the data requirement and execute the steps of the data unified collection method. Description of the Drawings
[0028] The accompanying drawings required for use in the embodiments or related technical descriptions will be briefly introduced below. The present disclosure can be more clearly understood according to the following detailed description with reference to the accompanying drawings.
[0029] Obviously, the accompanying drawings in the following description are only some embodiments of the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.
[0030] Figure 1 Schematic diagram showing the position of the data unified acquisition system according to some embodiments of the present disclosure in the new generation cloud network operation system.
[0031] Figure 2 Schematic diagram showing the interaction between the data unified acquisition system according to some embodiments of the present disclosure and other systems.
[0032] Figure 3 Schematic diagram showing the structure of the data unified acquisition system according to some embodiments of the present disclosure.
[0033] Figure 4 Schematic diagram showing the data unified acquisition method according to some embodiments of the present disclosure.
[0034] Figure 5 Schematic diagram showing an example of the unified data model according to some embodiments of the present disclosure.
[0035] Figure 6 Schematic diagram showing the structure of the data unified acquisition system according to some embodiments of the present disclosure. Detailed implementation manners
[0036] It should be noted that: unless otherwise specifically stated, the relative arrangements, numerical expressions, and numerical values of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.
[0037] Those skilled in the art can understand that terms such as "first", "second", etc. in the embodiments of the present disclosure are only used to distinguish different steps, devices, or modules, etc., and neither represent any specific technical meaning nor indicate an inevitable logical order between them.
[0038] It should also be understood that in the embodiments of the present disclosure, "a plurality" may refer to two or more, and "at least one" may refer to one, two, or more.
[0039] It should also be understood that for any component, data, or structure mentioned in the embodiments of the present disclosure, without clear definition or contrary indication in the context, it can generally be understood as one or more.
[0040] In addition, the term "and / or" in the present disclosure is merely an association relationship describing associated objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in the present disclosure generally represents an "or" relationship between the preceding and following associated objects.
[0041] It should also be understood that the descriptions of the various embodiments in the present disclosure emphasize the differences between the various embodiments, and their similarities can be referred to each other. For the sake of brevity, they will not be elaborated one by one.
[0042] Meanwhile, it should be understood that for the convenience of description, the dimensions of the various parts shown in the drawings are not drawn according to the actual proportional relationship.
[0043] The following description of at least one exemplary embodiment is actually merely illustrative and in no way restricts the present disclosure or its application or use.
[0044] Technologies, methods, and devices known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, the said technologies, methods, and devices should be regarded as part of the specification.
[0045] It should be noted that similar reference numerals and letters denote similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.
[0046] In addition, to avoid obscuring the present disclosure with unnecessary details, only the processing steps and / or device structures closely related to at least the solutions of the present disclosure are shown in the drawings, while other details less related to the present disclosure are omitted. It should also be noted that similar reference numerals and letters in the drawings indicate similar items, and thus once an item is defined in one drawing, it is not necessary to discuss it again for subsequent drawings.
[0047] Regarding the problems of differences and conflicts in various professional data after horizontally integrating network management systems for each specialty, some embodiments of the present disclosure propose a data unified acquisition system.
[0048] Figure 1 A schematic diagram showing the position of the data unified acquisition system according to some embodiments of the present disclosure in a new generation cloud network operation system.
[0049] As Figure 1As shown in the figure, the new-generation cloud network operation system includes: a cloud network unified control system 110, a capability open and sharing system 120, an alarm processing system 130, a performance processing system 140, a resource processing system 150, various application systems 160, etc. In the embodiments of the present disclosure, a data unified collection system 170 and a data storage and sharing system 180 are added to the new-generation cloud network operation system, and the two are collectively referred to as the data system 190.
[0050] The cloud network unified control system 110 is responsible for the unified control of cloud resources and network resources. The capability open and sharing system 120 opens network capabilities to third parties through the capability open and sharing interface, so that third parties can design customized network services according to their respective needs. The alarm processing system 130 is responsible for processing alarm data and providing fault location and demarcation. The performance processing system 140 is responsible for the data summary and analysis related to network performance. The resource processing system 150 is responsible for the unified management and cross-domain association of resource and network operation data. The data unified collection system 170 is responsible for the unified collection and control of various network elements and network management data. The specialties of the collected data include, for example, but are not limited to, wireless networks, bearer networks, core networks, transmission networks, cloud resources, network virtualization devices, etc. The collected data includes, for example, but is not limited to, performance data, configuration data, alarm data, deep packet inspection data, terminal measurement report data, flow data, network element control operations, etc. The data storage and sharing system 180 provides unified data storage, access, sharing, and management based on the unified data collected by the data unified collection system 170. Each application system 160 obtains the required data through the data storage and sharing system 180 and performs corresponding applications.
[0051] The data unified collection system 170 realizes the unified collection of various network elements and network management data, provides unified standard data for the upper-layer operation system, and provides real-time data collection services in the form of interfaces for the upper-layer operation system to call. The data unified collection system 170 unifies the data of multi-vendor devices and provides unified data for the upper-layer operation system, which can shield the differences in the interfaces of network elements of different vendors and models, and improve the efficiency of the operation system in processing data. According to the specialty, the unified management of the data collection of multiple specialties is carried out, avoiding repeated data collection and the problem of inconsistent data results from different sources.
[0052] Figure 2 The figure shows a schematic diagram of the interaction between the data unified collection system and other systems in some embodiments of the present disclosure. As Figure 2 shown, the interaction process between the data unified collection system 170 and other systems includes the following steps.
[0053] In step 210, other systems (such as a certain application system 160) send data requirements to the data storage and sharing system 180.
[0054] In step 220, the data storage and sharing system 180 queries whether the required data is stored locally according to the data requirement. If the required data is stored locally, the required data is directly returned. If the required data is not stored locally, the data requirement is sent to the data unified collection system 170 for data collection.
[0055] Among them, the data storage and sharing system 180 sends the data requirement to the data unified collection system 170 through the service interface of the data unified collection system 170, so that the data unified collection system 170 conducts data collection for the data requirement.
[0056] In step 230, the data unified collection system 170 determines whether the required data is periodically collected data according to the data requirement. If it is periodically collected data, the required data stored locally is returned to the data storage and sharing system 180. If it is not periodically collected data, but personalized collection requirement data, a data collection task corresponding to the data requirement is established, and the data unified collection method is executed to obtain the data corresponding to the data requirement. The data unified collection method will be specifically described later.
[0057] In step 240, the data unified collection system 170 sends the data corresponding to the data requirement collected to the data storage and sharing system 180.
[0058] In step 250, the data storage and sharing system 180 sends the data corresponding to the data requirement to other systems that issue the data requirement.
[0059] The data unified collection system 170 is the data base of the operation system and is mainly docked with the data storage and sharing system 180. For data that the data storage and sharing system 180 does not have or is not the latest, the collection interface of the data unified collection system 170 can be called for data collection. The data unified collection system 170 transmits the collected data to the data storage and sharing system 180 for storage, and the data storage and sharing system 180 can share the data with other system applications.
[0060] Figure 3 The structural schematic diagram of the data unified collection system showing some embodiments of the present disclosure is as Figure 3 shown. The data unified collection system 170 includes the following modules.
[0061] The service interface module 171 is connected to the data storage and sharing system and is configured to receive data requirements. It determines whether the required data is periodically collected data according to the data requirements. If it is periodically collected data, it returns the required data stored in the data storage module to the data storage and sharing system. If it is not periodically collected data, it establishes a corresponding data collection task for the data requirements to execute the data unified collection method and returns the data corresponding to the data requirements obtained through the data unified collection method.
[0062] The task scheduling module 172 is configured to manage each data collection task according to information such as the specialty and time of the data collection task, and schedule the data collection service corresponding to the specialty of the data collection task to execute the data collection task.
[0063] The collection monitoring module 173 is configured to implement monitoring functions for services, tasks, interfaces, data quality, etc., form statistical analyses, and notify the task scheduling module 172 of abnormal situations for corresponding processing, such as rescheduling the data collection service corresponding to the specialty of the data collection task to execute the data collection task. The collection monitoring content includes collection task monitoring, collection environment monitoring, collection quality monitoring, etc., provides corresponding monitoring pages, provides corresponding logs, and provides real-time alarms for abnormalities. For example, the collection monitoring module 173 monitors the execution of the data collection task. In the case of monitoring an execution abnormality, it reschedules the data collection service corresponding to the specialty of the data collection task to execute the data collection task. Collection task monitoring: Monitor the execution of the collection task, including information such as task name, scheduling time, start / end time, task execution status, execution log, task status last update time, task failure reason, etc. Collection environment monitoring: Complete monitoring of the load, heartbeat, etc. of the collection service. Collection quality monitoring: According to the formulated rules, flexibly detect and monitor the collection services of different data streams, and form a collection data quality monitoring result according to the detection results, including but not limited to the monitoring of the following service data quality: performance and configuration data, alarm data. Performance and configuration data include: monitoring the collected IP addresses, the number of collected files, and the file sizes, forming a collection result record with the IP address and data type as the original granularity. For the collection parsing record number change rate index, it should be possible to configure according to the corresponding threshold values and report the integrity of the collection parsing record number. Alarm data includes the long-term absence of alarm message reports for the alarm connection and the interruption of the alarm connection.
[0064] The data verification module 174 is configured to flexibly verify the collection of different data streams according to the formulated data verification rules, form a collection data quality result according to the verification results, and if the data verification is unqualified, re-collect or generate a collection abnormality alarm.
[0065] Among them, data verification includes data accuracy verification, data integrity verification, data validity verification, data volatility verification, etc. Data integrity verification: Verify whether there are any omissions and missing during the data transmission process through the data volume size and field measurement, including four aspects: entity integrity, attribute integrity, record integrity, and field value integrity. Data validity verification: Preset data standard specifications and reasonable ranges, and verify whether the data and information are standardized and whether the data is logical during the collection and transmission processes. Data accuracy verification: For data that can be obtained from multiple sources, such as data obtained from 2 fewer network elements or 2 fewer specialties, perform consistency verification on the data obtained from multiple sources to verify whether the data truly and accurately records the original data. Data volatility verification: Compare the currently collected data volume (e.g., the number of network elements) with the historical data volume to detect whether there are significant fluctuations.
[0066] In addition, in the case where the verification fails, first attempt to repeat the collection and verification. If the verification still fails, generate a data review warning and wait for manual verification and processing.
[0067] The unified processing module 175 is configured to implement a unified data format processing service for the collected original data of each specialty through actions such as parsing, calculating, extracting, and cleaning.
[0068] Among them, the unified data format processing includes identity unified data format processing and data unified data format processing. Identity unified data format processing: For the collected identity data, such as network elements, IP addresses, etc., add a source identifier, and combine the source identifier and the initial identity data as the unique identifier for unified data format processing. The unified data format processing identifier can also include improving feature data, such as adding the creator ID, manufacturer ID, etc. Data unified data format processing: Form a unified data model based on the union of the data characteristics of all manufacturers, and perform conversion processing on the collected data according to the unified data model. For example, the lengths of the network element names of each manufacturer are different. Take the maximum name length in the union of the network element names of each manufacturer, and convert the lengths of the network element names of each manufacturer to this maximum name length.
[0069] The data storage module 176 is configured to store the collected data after unified processing and distribute it according to the subscription requirements (data demands) of the requester for use by other systems.
[0070] The data collection service modules 177 for each specialty are respectively connected to the devices / network management of each specialty to implement the function of data collection service for each specialty. For example, the data collection service modules 177 for each specialty include a radio network data collection service module, a core network data collection service module, an IP data collection service module, an optical network data collection service module, a cloud computing data collection service module, etc. The radio network data collection service module is connected to the radio network devices / network management. The core network data collection service module is connected to the core network devices / network management. The IP data collection service module is connected to the IP devices / network management. The optical network data collection service module is connected to the optical network devices / network management. The cloud computing data collection service module is connected to the cloud computing devices / network management.
[0071] The data unified collection method executed by the data unified collection system will be described below.
[0072] Figure 4 A schematic diagram showing the data unified collection method of some embodiments of the present disclosure. As Figure 4 shown, the data unified collection method includes the following steps.
[0073] In step 410, the service interface module receives the data requirements of other systems (such as the data storage and sharing system), and determines whether the required data is periodically collected data according to the data requirements. In step 420, if it is periodically collected data, it queries the data storage module to see if the required data has been collected. If it has been collected, the required data is returned to the data storage and sharing system. In step 430, if it is not periodically collected data, a corresponding data collection task is established for the data requirements and assigned to the task scheduling module.
[0074] In step 440, the task scheduling module schedules the data collection service (module) of the corresponding specialty of the data collection task to execute the data collection task. For example, the wireless data collection task is assigned to the wireless data collection service module, and the wireless data collection service implemented by the wireless data collection service module executes the data collection task.
[0075] In step 450, the task scheduling module synchronizes the data collection task and its scheduling information to the collection monitoring module.
[0076] In step 460, the collection monitoring module monitors the execution of the data collection task. In the case of monitoring an execution exception, the data collection service of the corresponding specialty of the data collection task is rescheduled to execute the data collection task.
[0077] In step 470, the data acquisition service module that receives the data acquisition task obtains the corresponding original acquisition data (first data) of the data acquisition task through the interaction with the device / network management docked with it, and sends the first data to the data verification module.
[0078] In step 480, the data verification module verifies the acquired first data, and sends it to the unified processing module after the verification passes.
[0079] Among them, data verification includes data accuracy verification, data integrity verification, data validity verification, data volatility verification, etc.
[0080] Data accuracy verification includes, for example, data consistency verification. Data consistency verification includes: when the first data includes data of the first information item obtained from a certain source (the first source) and another source (the second source) respectively, performing consistency verification on the data of the first information item obtained from the first source and the data of the first information item obtained from the second source. If the data of the first information item obtained from the first source is consistent with the data of the first information item obtained from the second source, the consistency verification passes.
[0081] For example, perform consistency verification on the number of core network elements. For the data of the number of online core network elements, two data sources can be set. One is to obtain the number of online control plane network elements managed through the instructions of network element service management, and the other is to extract the number of online control plane network elements from the performance files of all control plane network elements. Compare the number of core network elements from these two data sources. If the number of online control plane network elements in the performance file is small, it means that there are missing performance files; if the number of currently queried online control plane network elements is small, it may be that the configured control plane name is incorrect or there are currently offline network elements. In the case of abnormal consistency verification, manual verification and processing can be performed.
[0082] Data integrity verification includes: verifying whether the first data collected by the data acquisition service is complete, including one or more of the entities, attributes, records, and fields required by the data acquisition task. If it is complete, the integrity verification passes.
[0083] Data validity verification includes: verifying whether the first data collected by the data acquisition service conforms to the preset specifications and preset ranges. If it conforms, the validity verification passes.
[0084] Data volatility verification includes: comparing the current data volume of the first data collected by the data acquisition service with the historical data volume. If the difference between the current data volume and the historical data volume does not exceed the threshold, the volatility verification passes. If the difference between the current data volume and the historical data volume exceeds the threshold, a data volatility alarm is generated.
[0085] In the case where the verification fails, repeat the steps of executing the data collection service for the corresponding specialty of the scheduling data collection task to execute the data collection task; verify the third data repeatedly collected by the data collection service; in the case where the verification of the third data fails, generate a data review warning.
[0086] In step 490, the unified processing module performs unified data format processing on the collected first data, and sends it to the data storage module after the unified data format processing.
[0087] Among them, the unified data format processing includes identification unified data format processing and data unified data format processing. The identification unified data format processing includes: adding a source identifier to the first identification data in the first data, and combining the source identifier and the first identification data as the unique identifier for the unified data format processing. Thus, the identification data from each source can have a unique identifier in the data unified collection system. The data unified data format processing includes: forming a unified data model according to the union of the data characteristics of all manufacturers, and performing conversion processing on the first data collected by the data collection service according to the unified data model. The unified data model includes the collected data items and the format requirements for each data item. The data collected from each source needs to be converted according to the requirements of the unified data model, so as to form data in a unified standard.
[0088] Figure 5 A schematic diagram showing an example of the unified data model of some embodiments of the present disclosure. As Figure 5 shown, the unified data model includes network element devices, entity specifications, entity specification attributes, rules, configuration object specifications, and configuration object specification attributes. The unified data model stipulates the data items that each part should include and the format requirements for each data item. For example, in the "entity specification" part of the unified data model, the data format requirement for the data item "name" is Varchar (variable character type), and the length requirement is 255. In addition, in the unified data model, CRUD represents Create (add), Read (read), Update (update), and Delete (delete), Number represents the numerical type, Timestamp represents the timestamp type, Bool represents the boolean type, JSON (JavaScript Object Notation, JS object notation) is a lightweight data exchange format, and Jsonb represents the JSON Binary (binary) type.
[0089] In step 4100, the data storage module stores the unified data (second data) obtained after passing the verification and unified data format processing.
[0090] In step 4110, the data storage module sends the second data to the service interface module.
[0091] In step 4120, the service interface module uses the second data as the response data of the data collection task and returns it to the system (such as the data storage and sharing system) that requests the data.
[0092] In the embodiments of the present disclosure, the data collection task is scheduled to the corresponding professional data collection service to execute the data collection task. The collected initial data is verified and processed with a unified data format. The data obtained after passing the verification and unified data format processing is used as the finally collected unified data, thereby shielding the differences in data sources, realizing unified data collection and management, and further providing a unified data service for external systems based on the unified data.
[0093] Figure 6 The structural schematic diagram of the data unified collection system according to some embodiments of the present disclosure is shown. As Figure 6 shown, the data unified collection system 600 of this embodiment includes: a memory 610 and a processor 620 coupled to the memory 610. The processor 620 is configured to execute the data unified collection method in any of the embodiments based on the instructions stored in the memory 610.
[0094] The data unified collection system 600 may further include an input / output interface 630, a network interface 640, a storage interface 650, etc. These interfaces 630, 640, 650 and the memory 610 and the processor 620 may be connected through a bus 660, for example.
[0095] Among them, the memory 610 may include a system memory, a fixed non-volatile storage medium, etc. The system memory stores an operating system, application programs, a boot loader, and other programs, for example.
[0096] Among them, the processor 620 may be implemented in a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor discrete hardware components.
[0097] Among them, the input / output interface 630 provides connection interfaces for input / output devices such as monitors, mice, keyboards, touchscreens, etc. The network interface 640 provides connection interfaces for various networking devices. The storage interface 650 provides connection interfaces for external storage devices such as SD cards and USB flash drives. The bus 660 can use any bus structure among a variety of bus structures. For example, the bus structure includes, but is not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, and Peripheral Component Interconnect (PCI) bus.
[0098] Embodiments of the present disclosure provide a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the data unified acquisition method are implemented. The storage medium includes, for example, a non-transitory computer-readable storage medium.
[0099] Embodiments of the present disclosure provide a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the data unified acquisition method are implemented.
[0100] Those skilled in the art should understand that the embodiments of the present disclosure can be provided as a method, a system, or a computer program product. Therefore, the present disclosure can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present disclosure can take the form of a computer program product implemented on one or more non-transitory computer-readable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer program code.
[0101] The present disclosure is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the specified functions in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.
[0102] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means that implement the functions specified in one or more of the flow Figure 1 acts or a plurality of acts and / or boxes Figure 1 or boxes or a plurality of boxes.
[0103] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions specified in one or more of the flow Figure 1 acts or a plurality of acts and / or boxes Figure 1 or boxes or a plurality of boxes.
[0104] The foregoing are only preferred embodiments of the present disclosure and are not intended to limit the present disclosure. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present disclosure shall be included within the protection scope of the present disclosure.
Claims
1. A data unified acquisition method, characterized in that, Including: Dispatch the data collection service corresponding to the professional of the data collection task to execute the data collection task; Verify the first data collected by the data collection service; Perform unified data format processing on the first data collected by the data collection service; Use the second data obtained after passing the verification and unified data format processing as the response data of the data collection task.
2. The method according to claim 1, characterized in that, The verification of the first data collected by the data collection service includes: In the case where the first data includes data of the first information item obtained from the first source and the second source respectively, perform consistency verification on the data of the first information item obtained from the first source and the data of the first information item obtained from the second source. If the data of the first information item obtained from the first source is consistent with the data of the first information item obtained from the second source, the consistency verification passes.
3. The method according to any one of claims 1-2, characterized in that The unified data format processing of the first data collected by the data collection service includes: According to the unified data model, perform conversion processing on the first data collected by the data collection service, and the unified data model is formed according to the union of the data characteristics of all manufacturers.
4. The method according to any one of claims 1 to 3, characterized in that The unified data format processing of the first data collected by the data collection service includes: Add a source identifier to the first identifier data in the first data, and combine the source identifier and the first identifier data as the unique identifier for unified data format processing.
5. The method according to any one of claims 1-4, characterized in that, The verification of the first data collected by the data collection service includes: Verify whether the first data collected by the data collection service is complete, including one or more of entities, attributes, records, and fields required by the data collection task. If it is complete, the integrity verification passes.
6. The method according to any one of claims 1-5, characterized in that, The verification of the first data collected by the data collection service includes: Verify whether the first data collected by the data collection service conforms to the preset specifications and preset ranges. If it conforms, the validity verification passes.
7. The method according to any one of claims 1 to 6, characterized in that The verification of the first data collected by the data collection service includes: Compare the current data volume of the first data collected by the data collection service with the historical data volume. If the gap between the current data volume and the historical data volume does not exceed the threshold, the volatility verification passes. If the gap between the current data volume and the historical data volume exceeds the threshold, generate a data volatility alarm.
8. The method according to any one of claims 1-7, characterized in that, Also including: In the case where the verification fails, repeat the step of dispatching the data collection service corresponding to the professional of the data collection task to execute the data collection task; Verify the third data repeatedly collected by the data collection service; In the case where the verification of the third data fails, generate a data review alarm.
9. The method according to any one of claims 1 to 8, characterized in that, Also including: Monitor the execution of the data collection task. In the case where an execution anomaly is monitored, reschedule the data collection service corresponding to the professional of the data collection task to execute the data collection task.
10. According to the method described in any one of claims 1-9, characterized in that The specialties of the data collection service include one or more of wireless network, bearer network, core network, transmission network, cloud resources, and network virtualization devices; The first data collected includes: one or more of performance data, configuration data, alarm data, deep packet inspection data, terminal measurement report data, flow data, and network element control operations.
11. A data unified acquisition system, characterized in that, include: A task scheduling module is configured to schedule a professional data collection service corresponding to a data collection task to execute the data collection task; a data verification module, configured to verify the first data collected by the data collection service; a unified processing module, configured to perform unified data format processing on the first data collected by the data collection service; The service interface module is configured to use the second data obtained after verification and unified data format processing as response data of the data collection task.
12. The system according to claim 11, wherein, Also includes: The collection monitoring module is configured to monitor the execution of the data collection task, and reschedule the professional data collection service corresponding to the data collection task to execute the data collection task when an execution anomaly is detected.
13. A data unified acquisition system, comprising: Memory; and a processor coupled to the memory, wherein the processor is configured to execute the method according to any one of claims 1 to 10 based on instructions stored in the memory.
14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.
15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.
16. A data system comprising: The data storage and sharing system is configured to query whether the required data is stored locally according to data requirements, and directly return the required data if the required data is stored locally; if the required data is not stored locally, the data requirement is sent to the unified data collection system; The unified data collection system is configured to determine whether the required data is periodically collected data based on data requirements. If it is periodically collected data, the required data stored locally is returned to the data storage and sharing system. If it is not periodically collected data, a corresponding data collection task is established for the data requirements, and the steps of the method described in any one of claims 1 to 10 are executed.