Method, device and computer storage medium for data acquisition
By finding the interface configuration information in the SDN controller and establishing a communication connection, data can be directly obtained and transformed, solving the problem of low data acquisition efficiency caused by inconsistent interface information of the SDN controller, and realizing low-code development and efficient data acquisition.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA MOBILE INFORMATION TECHNOLOGY CO LTD
- Filing Date
- 2023-05-22
- Publication Date
- 2026-06-05
Smart Images
Figure CN116471183B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, specifically to a method, apparatus, device, and computer storage medium for data acquisition. Background Technology
[0002] Currently, Software Defined Networking (SDN) is widely used as the main network technology for data center cloud resource pools.
[0003] Currently, the business data and other information required by upper-layer services need to be obtained by connecting to the northbound interfaces provided by various software-defined network controller vendors. However, each vendor's SDN has hundreds of interfaces, and the interface information provided by each vendor's SDN is different and the standards are inconsistent. If the required business data is to be obtained, it is necessary to develop for each different interface, which is time-consuming, costly, and results in low data acquisition efficiency. Summary of the Invention
[0004] The purpose of this application is to provide a method, apparatus, device, and computer storage medium for data acquisition, thereby solving the problem of low data acquisition efficiency.
[0005] The technical solution of this application is as follows:
[0006] Firstly, a data acquisition method is provided, which includes:
[0007] Upon receiving a data acquisition instruction, the system searches for the target acquisition task information corresponding to the acquisition task ID in the data acquisition instruction, based on the preset correspondence between acquisition task information and acquisition task ID. The target acquisition task information includes the interface model ID.
[0008] The interface configuration information is obtained based on the interface model ID, and the interface is assembled and called based on the interface configuration information;
[0009] When a communication connection is established with the SDN controller corresponding to the interface model ID by calling the interface, the original data of the SDN controller corresponding to the SDN interface model corresponding to the interface model ID is obtained.
[0010] Based on the preset mapping relationship between the SDN interface model and the business model, the original data is transformed into target business data, wherein the original data includes target fields, the target business data includes target business fields, and the mapping relationship includes the mapping relationship between the target fields and the target business fields;
[0011] The target business data is stored in a data warehouse.
[0012] Secondly, a data acquisition apparatus is provided, the apparatus comprising:
[0013] The data acquisition scheduling service module is used to obtain the data acquisition task ID and generate data acquisition instructions;
[0014] The data acquisition service module is used to receive data acquisition instructions sent by the acquisition scheduling service module through the data acquisition engine in the data acquisition service module, and to find the target acquisition task information corresponding to the acquisition task ID in the data acquisition instruction according to the preset correspondence between acquisition task information and acquisition task ID. The target acquisition task information includes the interface model ID.
[0015] The data acquisition service module is also used to obtain the raw data of the SDN controller through the SDN interface model corresponding to the interface model ID when the data acquisition engine establishes a communication connection with the corresponding SDN controller according to the interface model ID.
[0016] The data acquisition service module is further configured to convert the raw data into target business data through the data parsing engine in the data acquisition service module according to the preset mapping relationship between the SDN interface model and the business model. The raw data includes target fields, the target business data includes target business fields, and the mapping relationship includes the mapping relationship between the target fields and the target business fields.
[0017] The storage module is used to store the target business data in a data warehouse.
[0018] This application provides a method, apparatus, device, and computer storage medium for data acquisition. Upon receiving a data acquisition instruction, the method searches for the target acquisition task information corresponding to the acquisition task ID in the data acquisition instruction based on a preset correspondence between acquisition task information and acquisition task ID. Then, it obtains interface configuration information based on the interface model ID in the target acquisition task information and assembles a call interface based on the interface configuration information. When a communication connection is established between the call interface and the SDN controller corresponding to the interface model ID, the raw data of the SDN controller corresponding to the SDN interface model corresponding to the interface model ID can be obtained. In this way, by assembling the call interface based on the interface configuration information, the raw data of the SDN controller corresponding to the SDN interface model corresponding to the interface model ID can be obtained without developing an interface for each SDN controller, achieving low-code development, reducing acquisition costs, and saving R&D resources. Finally, based on the preset mapping relationship between the SDN interface model and the business model, the raw data can be directly converted into target business data and stored in a data warehouse without any other operations, further achieving low-code development and improving the efficiency of target business data acquisition.
[0019] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description
[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application, and do not constitute an undue limitation of this application.
[0021] Figure 1 This is one of the structural diagrams of the data acquisition system provided in the embodiments of this application;
[0022] Figure 2 This is a second schematic diagram of the structure of the data acquisition system provided in the embodiments of this application;
[0023] Figure 3 This is one of the flowcharts illustrating the data acquisition method provided in the embodiments of this application;
[0024] Figure 4 This is a schematic diagram of the structure of the data acquisition device provided in the embodiments of this application;
[0025] Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation
[0026] To enable those skilled in the art to better understand the technical solutions of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.
[0027] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples consistent with some aspects of this application as detailed in the appended claims.
[0028] As described in the background section, currently, the network resources and service data required by upper-layer services need to be obtained by connecting to interfaces provided by various SDN controller vendors. However, the interface information provided by each vendor is different and the standards are inconsistent. Therefore, it is usually necessary to collect and develop interfaces one by one. This results in high data collection costs and a serious drain on R&D resources. To address the aforementioned issues, this application provides a method, apparatus, device, and computer storage medium for data acquisition. Upon receiving a data acquisition instruction, the method locates the target acquisition task information corresponding to the acquisition task ID in the data acquisition instruction based on a preset correspondence between acquisition task information and acquisition task IDs. This eliminates the need for developing separate acquisition tasks for each task. Then, interface configuration information is obtained based on the interface model ID in the target acquisition task information, and a calling interface is assembled based on this configuration information. When a communication connection is established between the calling interface and the SDN controller corresponding to the interface model ID, the raw data of the SDN controller corresponding to the SDN interface model corresponding to the interface model ID can be obtained. By assembling the calling interface based on the interface configuration information, the raw data of the SDN controller corresponding to the SDN interface model corresponding to the interface model ID can be obtained without developing an interface for each SDN controller, achieving low-code development, reducing acquisition costs, and saving R&D resources. Finally, based on a preset mapping relationship between the SDN interface model and the business model, the raw data can be directly converted into target business data, and the target business data can be stored in a data warehouse without further operations, further achieving low-code development and improving the efficiency of target business data acquisition.
[0029] The data acquisition method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.
[0030] Before introducing the data acquisition method provided in the embodiments of this application, we will first introduce the data acquisition system that implements the data acquisition method in the embodiments of this application, such as... Figure 1 As shown, the data acquisition system 100 provided in this application embodiment includes a data acquisition scheduling service module 110, a data acquisition service module 120, and a storage module 130.
[0031] Among them, the data acquisition and scheduling service module 110 is used to obtain the data acquisition task ID and generate data acquisition instructions;
[0032] The data acquisition service module 120 is used to receive data acquisition instructions sent by the acquisition scheduling service module through the data acquisition engine in the data acquisition service module, and to find the target acquisition task information corresponding to the acquisition task ID in the data acquisition instruction according to the preset correspondence between acquisition task information and acquisition task ID. The target acquisition task information includes the interface model ID.
[0033] The data acquisition service module 120 is also used to obtain interface configuration information based on the interface model ID in the target acquisition task information, and assemble the call interface based on the interface configuration information. When a communication connection is established between the call interface and the SDN controller corresponding to the interface model ID, the raw data of the SDN controller is obtained through the data acquisition engine through the SDN interface model corresponding to the interface model ID.
[0034] The data acquisition service module 120 is also used to transform raw data into target business data through the data parsing engine in the data acquisition service module according to the preset mapping relationship between the SDN interface model and the business model.
[0035] Storage module 130 is used to store target business data in a data warehouse.
[0036] Among them, the data collection task ID can be a pre-set data collection task ID. Each data collection task has its own corresponding identifier, i.e., ID, which is used to distinguish each data collection task.
[0037] A data acquisition command is an instruction to collect data. The desired data can be collected based on this data acquisition command.
[0038] In some embodiments of this application, the data acquisition scheduling service module can obtain the data acquisition task ID and generate a data acquisition instruction based on the data acquisition task ID. The data acquisition task ID can be obtained by the user in the user layer configuration. The user can configure data acquisition scheduling task information in the user layer. After the data acquisition scheduling service module detects the completion of the data acquisition scheduling task information configuration in the user layer, it can store the data acquisition scheduling task information in the data acquisition task table in the data acquisition scheduling service module. Then, the data acquisition scheduling service module can retrieve the data acquisition task from the data acquisition task table, obtain the data acquisition task ID, and generate a data acquisition instruction based on the data acquisition task ID.
[0039] The information collected for task scheduling may include: task name, task ID, universal resource locator (URL) for the task interface, token permissions, and task configuration table ID.
[0040] The scheduling task ID here can be a scheduling thread pre-configured for each data collection task. When executing the data collection task, the scheduling task ID can be used to determine which scheduling thread will execute it.
[0041] The data collection task configuration table ID can be stored in the data collection task configuration table in the data collection scheduling service module after the data collection task information is configured at the user level. This data collection task configuration table has its corresponding ID, so that the data collection task can be retrieved from the data collection task configuration table corresponding to the data collection task configuration table ID in the future.
[0042] It should be noted that before sending a data collection instruction, the data collection and scheduling service module must verify the token permission. If the token permission verification is successful, the corresponding data collection instruction can be generated based on the data collection task ID.
[0043] It should be noted that the data acquisition scheduling service module can send data acquisition scheduling task information to the data acquisition service module and store it in the preloaded information hash table in the data acquisition service module.
[0044] The data collection task information can be configured by the user at the user level. Specific data collection task information may include: username, password, vendor SDN interface communication protocol, interface host, interface port number, interface request method, interface URI, and interface model ID, as shown in Table 1 below:
[0045] Table 1
[0046] Configuration Name Configuration type Notes username String password String Interface Protocol String Examples: http, https, REST Interface host String Interface port number String Interface input parameter message String API request method String Example get, post Interface URI String Return data type String Interface Model Name String
[0047] The aforementioned username, password, and vendor SDN interface communication protocol can be used to establish a communication connection between the interface model ID and the SDN controller.
[0048] Once the data acquisition service module detects that the acquisition task information configuration has been completed, it can store it in a preloaded information hash table.
[0049] The target acquisition task information can be the acquisition task information corresponding to the acquisition task ID.
[0050] It should be noted that each data collection task scheduling information contains corresponding data collection task information, and the correspondence between the data collection task information and the data collection task ID in the data collection task scheduling information needs to be preloaded into the preloaded information hash table.
[0051] In some embodiments of this application, the data acquisition service module includes a data acquisition engine. After receiving a data acquisition instruction from the acquisition scheduling service module, the data acquisition engine searches a pre-loaded information hash table for the target acquisition task corresponding to the acquisition task ID in the data acquisition instruction, based on a pre-set correspondence between acquisition task information and acquisition task ID. This target acquisition task includes an interface model ID. Then, the engine obtains interface configuration information based on the interface model ID in the target acquisition task and assembles a call interface based on the interface configuration information. When a communication connection is established between the call interface and the SDN controller corresponding to the interface model ID, the engine obtains the raw data from the SDN controller through the SDN interface model corresponding to the interface model ID. Then, the data parsing engine in the data acquisition service module searches the pre-loaded hash table for the mapping relationship between the business model and the interface model corresponding to the interface model ID. Finally, based on the preset mapping relationship between the SDN interface model and the business model, the engine converts the raw data into target business data.
[0052] It should be noted that the data acquisition service is an independent microservice. After receiving the data acquisition instruction, it uses a hash algorithm to find the corresponding acquisition task from the pre-loaded acquisition task hash table based on the acquisition task ID.
[0053] The aforementioned SDN interface model can be that each SDN controller has multiple corresponding interface models, which are used to obtain the raw data from the corresponding SDN controller.
[0054] The business model has multiple business model fields, each of which corresponds to a field in the interface model. The data in this business model is business data that corresponds to the original data in the SDN controller. This business data is data that describes the original data according to certain rules.
[0055] The raw data can be the initial data managed in the SDN controller.
[0056] The target business data can be the business data corresponding to the original data.
[0057] In some embodiments of this application, the original data may include a target field, which may be a field contained in the original data, such as at least one of the following: interface model name, interface model code, interface return field name, or interface return field code.
[0058] The target business data may include target business fields, which may include at least one of the following: business model name, business model code, business model field name, or business model field code corresponding to the target field.
[0059] The specific mapping relationship is as follows: the interface model name field in the interface model corresponds to the business model name field in the business model; the interface model code in the interface model corresponds to the business model code in the business model; the interface return field name in the interface model corresponds to the business model field name in the business model; and the interface return field code in the interface model corresponds to the business model field code in the business model.
[0060] When the data parsing engine transforms vendor SDN interface data into business data, the model mapping relationships are pre-loaded into a hash table. Based on the interface model code, the data parsing engine uses a hash algorithm to find the corresponding model mapping relationship information. In each model mapping relationship, the business field code and the interface field code correspond one-to-one. Data returned by the vendor SDN interface is stored in a hash table according to the interface model format. The data parsing engine iteratively reads data from the hash table. For each set of vendor SDN interface return data read, the data parsing engine uses a hash algorithm to retrieve the mapping relationship corresponding to the interface code from the hash table storing the mapping relationships. Then, based on the interface fields, it uses a hash algorithm to find the corresponding business field code, business field name, and business model ID.
[0061] In some embodiments of this application, users need to pre-configure the interface model and business model, as well as the mapping relationship between the interface model and the business model, at the user layer. After configuration, the data acquisition service listens for the configuration of the interface model and business model, and the mapping relationship between them, and stores it in the preloaded information list. Specific configuration information for the interface model is shown in Table 2 below, configuration information for the business model is shown in Table 3 below, and the mapping relationship between the interface model and the business model is shown in Table 4 below.
[0062] Table 2
[0063] Configuration Name Configuration type Notes Manufacturer Code String Globally unique Manufacturer Name String Interface model coding String Interface return field encoding String Unique under the same data model API returns field names String
[0064] It's important to note that the SDN vendor code and SDN vendor name are primarily used to distinguish data sources. The interface data model is loaded into a hash table by the data acquisition service module, providing it to the data acquisition engine to parse the data stream returned by the SDN vendor's interface. The SDN vendor code and SDN vendor name in the interface model are mainly used to distinguish the data source, belonging to a specific SDN vendor. The interface code in the interface model is used to mark and distinguish different interface model data. The interface return field code and interface return field name are created according to the SDN vendor's interface specification. There may be multiple sets of interface return field codes and interface return field names; it is not a single set.
[0065] Table 3
[0066] Configuration Name Configuration type Notes Business Model Name String Business model coding String Globally unique Business model field names String Business model field encoding String Unique under the same business model
[0067] It should be noted that the business model can be abstracted and defined by business experts based on the upper-layer business requirements in actual operation and maintenance scenarios. The business data model is mainly used to abstract and unify the field encoding and field names of various SDN vendor interfaces, and to provide users with standardized field encoding and field names.
[0068] Table 4
[0069]
[0070]
[0071] In some embodiments of this application, after obtaining the target business data, the target business data can be stored in a data warehouse. Specifically, the storage can be performed as follows: the data parsing engine stores the values corresponding to the interface data fields into the values corresponding to the business fields; after the business model data is transformed, the business model data is stored in a queue; and finally, the data parsing engine stores the queued business data in batches into the data warehouse.
[0072] In the embodiments of this application, the acquisition task ID is obtained based on the acquisition scheduling service module, and a data acquisition instruction is generated. The data acquisition engine in the data acquisition service module receives the data acquisition instruction, and according to the preset correspondence between acquisition task information and acquisition task ID, it searches for the target acquisition task information corresponding to the acquisition task ID in the data acquisition instruction. It obtains the interface configuration information based on the interface model ID in the target acquisition task information, and assembles the calling interface based on the interface configuration information. When a communication connection is established between the calling interface and the SDN controller corresponding to the interface model ID, the raw data of the SDN controller corresponding to the SDN interface model corresponding to the interface model ID is obtained. This adapts to different changes in acquisition data, eliminating the need for interface development for each different SDN controller, achieving low-code development, reducing acquisition costs, and saving R&D resources. Then, the data parsing engine in the data acquisition service module searches for the mapping relationship through the interface model ID. Based on the mapping relationship, the raw data can be directly converted into target business data. Then, the target business data is stored in the data warehouse based on the storage module, without any other operations, further achieving low-code development and improving the efficiency of target business data acquisition.
[0073] To better understand the data acquisition system described above, this application also provides another possible implementation of the data acquisition system, such as... Figure 2 As shown, the data acquisition system provided in the application embodiment includes a data acquisition and scheduling service module 110, a data acquisition service module 120, a user layer 140, and a configuration service module 150.
[0074] It should be noted that the same terms and definitions used in this application embodiment as those used in the above implementation will not be repeated here.
[0075] The functions of the data acquisition and scheduling service module 110, the data acquisition service module 120, the storage module 130, and the user layer 140 have been described in the above embodiments and will not be repeated here.
[0076] It should be noted that after the user layer 140 completes the configuration of the interface model, business model, and the mapping relationship between the interface model and the business model, the data acquisition service module, upon detecting the completion of its configuration, can load it into the pre-loaded information list, such as... Figure 2 The configuration information in the preloaded information list of the data acquisition service module.
[0077] It should be noted that after the user layer 140 completes the configuration of the interface model, business model, and the mapping relationship between the interface model and the business model, it can send this configuration to the configuration service module 150. The configuration service module 150 stores it in the data warehouse. After the mapping relationship between the interface model, business model, and the business model is configured, corresponding data tables will be created in the data warehouse, and the mapping relationship between the interface model, business model, and the business model will be stored in the corresponding data tables. This facilitates offline retrieval of the mapping relationship between the interface model, business model, and the business model, eliminating the need for real-time data retrieval and improving data acquisition efficiency.
[0078] The data configuration service module receives configuration information forms submitted by the user layer (the form includes configuration information such as interface models, business models, and the mapping relationships between interface models and business models), and stores the configuration form data in the data warehouse. Then, it notifies the data acquisition service module of data update messages via a message queue. Upon receiving the configuration data update message, the data acquisition service module updates the pre-loaded configuration information. The update is based on the configuration model ID stored in the message queue. The data acquisition service retrieves the model data from the hash table based on the model ID, uses a hash algorithm to match the model data, and finally updates the data.
[0079] It should be noted that the raw data from various vendors' SDN controllers can be stored in the data layer. This data layer enables the management of interfaces between SDN controllers from different vendors, message caching, and data storage capabilities, including functional modules such as interface services, message queues, and data warehouses. The interface service enables interfaces with multiple vendors' SDN controllers, manages access accounts, passwords, and connection addresses, and collects raw data from the SDN controller interfaces; the message queue enables message caching and asynchronous processing; and the data warehouse enables continuous data collection and storage of collection logs.
[0080] The functions of the data acquisition service module 120 are described in detail below:
[0081] Figure 3 This is a flowchart illustrating a data acquisition method provided in an embodiment of this application. The execution entity of this data acquisition method can be the aforementioned... Figure 1 and Figure 2 The data acquisition service module 120 in the middle.
[0082] It should be noted that the same terms and definitions used in this application embodiment as those used in the above implementation will not be repeated here.
[0083] like Figure 3 As shown, the data acquisition method provided in this application embodiment may include steps 310-320.
[0084] Step 310: Upon receiving a data acquisition instruction, based on the preset correspondence between acquisition task information and acquisition task ID, find the target acquisition task information corresponding to the acquisition task ID in the data acquisition instruction. The target acquisition task information includes the interface model ID.
[0085] Step 320: Obtain the interface configuration information based on the interface model ID, and assemble the call interface based on the interface configuration information.
[0086] The interface configuration information can be the username, password, vendor SDN interface communication protocol, interface host, interface port number, interface request method, and interface URI from the data collection task information. Based on this interface configuration information, a calling interface can be formed, which is used to represent the target interface request information and request data to be connected.
[0087] Step 330: When a communication connection is established between the interface and the SDN controller corresponding to the interface model ID, obtain the raw data of the SDN controller corresponding to the SDN interface model corresponding to the interface model ID.
[0088] In some embodiments of this application, each vendor's SDN controller has multiple corresponding interface models, and each vendor's SDN controller has multiple corresponding calling interfaces. When the data acquisition service establishes a communication connection with the SDN controller corresponding to the interface model ID through the calling interface, the raw data of the SDN controller corresponding to the SDN interface model corresponding to the interface model ID can be obtained.
[0089] Step 340: Based on the preset mapping relationship between the SDN interface model and the business model, transform the raw data into target business data.
[0090] The original data includes the target field, the target business data includes the target business field, and the mapping relationship includes the mapping relationship between the target field and the target business field.
[0091] Step 350: Store the target business data in the data warehouse.
[0092] It should be noted that in this application embodiment, storing the target business data in the data warehouse is done by the data acquisition service module. In the above embodiment, it is done by a storage module that is different from the data acquisition service module. That is, for this application embodiment, the storage module in the above embodiment can be set in the data acquisition service module, or it can be set separately as in the above embodiment. There is no limitation here.
[0093] In this embodiment, upon receiving a data acquisition instruction, the system searches for the target acquisition task information corresponding to the acquisition task ID in the data acquisition instruction based on a preset correspondence between acquisition task information and acquisition task ID. This eliminates the need to develop separate acquisition tasks for each task. Then, the system obtains interface configuration information based on the interface model ID in the target acquisition task information and assembles a call interface based on this configuration. When a communication connection is established between the call interface and the SDN controller corresponding to the interface model ID, the system can obtain the raw data of the SDN controller corresponding to the SDN interface model corresponding to the interface model ID. By assembling the call interface based on the interface configuration information, the system can obtain the raw data of the SDN controller corresponding to the SDN interface model corresponding to the interface model ID. This eliminates the need to develop interfaces for each SDN controller, achieving low-code development. Finally, based on a preset mapping relationship between the SDN interface model and the business model, the raw data can be directly converted into target business data, and the target business data can be stored in a data warehouse without further operations, further achieving low-code development and improving the efficiency of target business data acquisition.
[0094] In some embodiments of this application, in order to improve the efficiency of data acquisition, the above method may further include:
[0095] Receive data acquisition instructions periodically.
[0096] In some embodiments of this application, the data acquisition scheduling service module can periodically obtain data acquisition tasks from the data acquisition task table, then periodically generate data acquisition instructions, and send the data acquisition instructions to the data acquisition service module. Therefore, the data acquisition service module will periodically receive the data acquisition instructions sent by the data acquisition scheduling service module.
[0097] In the embodiments of this application, data can be collected periodically by receiving data collection instructions at regular intervals, thus ensuring the efficiency of data collection.
[0098] In some embodiments of this application, the data acquisition instruction may further include a scheduling task ID. To further improve the efficiency of data acquisition, step 310 may specifically include:
[0099] Based on the pre-defined correspondence between the data acquisition task information and the data acquisition task ID and the scheduling task ID, find the target data acquisition task information corresponding to the scheduling task ID and the data acquisition task ID.
[0100] In some embodiments of this application, the scheduling task ID can be a scheduling thread pre-configured for each collection task information. When executing the collection task, the scheduling task ID can be used to determine which scheduling thread will execute the task.
[0101] In the embodiments of this application, the target acquisition task information corresponding to the scheduling task ID and acquisition task ID is found according to the preset correspondence between acquisition task information and acquisition task ID and scheduling task ID. This enables multi-threaded execution of acquisition tasks and improves the efficiency of data acquisition.
[0102] In some embodiments of this application, the target field includes at least one of the following: interface model name, interface model code, interface return field name, or interface return field code;
[0103] The target business field includes at least one of the following: business model name, business model code, business model field name, or business model field code.
[0104] The mapping relationship between the target field and the target business field is as described in the above example, and will not be repeated here.
[0105] In some embodiments of this application, in order to accurately obtain the raw data to be acquired, the above method may further include the following steps before step 110:
[0106] The system receives first configuration information input by the user on the configuration page. The first configuration information includes the correspondence between the data collection task information and the data collection task ID.
[0107] The first configuration information can be configuration information entered by the user on the configuration page, and may include configuration information showing the correspondence between collection task information and collection task ID. The configuration page here can be the page displayed in the user layer of the above embodiment, on which the user can enter the information to be configured. The configuration service can obtain the corresponding configuration information based on the user's input, and send it to the data warehouse for storage, and also to the data collection service module.
[0108] In the embodiments of this application, by receiving the first configuration information entered by the user on the configuration page, the first configuration information can be precisely configured according to the user's needs, so as to obtain the original data required by the user based on the first configuration information, thereby improving the accuracy of obtaining the original data.
[0109] In some embodiments of this application, to further improve the accuracy of data acquisition, the above method may further include the following steps before step 330:
[0110] Receive the second configuration information entered by the user on the configuration page. The second configuration information includes the SDN interface model, the business model, and the mapping relationship between the SDN interface model and the business model.
[0111] Alternatively, the system can receive third configuration information input by the user on the configuration page, which includes at least one of the following: change information of the SDN interface model, change information of the business model, and change information of the mapping relationship between the SDN interface model and the business model; and update at least one of the following: the SDN interface model, the business model, and the mapping relationship between the SDN interface model and the business model based on the third configuration information.
[0112] The second configuration information can be configuration information entered by the user in the configuration interface for configuring the SDN interface model, the business model, and the mapping relationship between the SDN interface model and the business model.
[0113] The third configuration information may be at least one of the following: change information for configuring the SDN interface model, change information for the business model, and change information for the mapping relationship between the SDN interface model and the business model, which are entered by the user in the configuration interface.
[0114] In some embodiments of this application, the configuration of the SDN interface model, service model, and the mapping relationship between the SDN interface model and the service model at the user layer can be found in the above embodiments, and will not be repeated here.
[0115] In some embodiments of this application, if the previously configured SDN interface model, service model, and mapping relationship between the SDN interface model and the service model need to be changed, the user can also input change information for at least one of the SDN interface model, service model, and mapping relationship between the SDN interface model and the service model at the user level. After the data acquisition service module detects the change information for at least one of the SDN interface model, service model, and mapping relationship between the SDN interface model and the service model, it can update at least one of the SDN interface model, service model, and mapping relationship between the SDN interface model and the service model accordingly.
[0116] In the embodiments of this application, by receiving at least one of the following information input by the user on the configuration page: change information of the SDN interface model, change information of the business model, and change information of the mapping relationship between the SDN interface model and the business model, and updating at least one of the following information according to the third configuration information: when the data fields, interface model fields, and business model fields of the original data change, the data fields, interface model fields, and business model fields of the original data can be automatically updated without secondary development and modification of the engineering code, thus realizing the dynamic configuration of the data fields, interface model fields, and business model fields of the original data.
[0117] The technical solution of this application, through the development of general modules such as a model configuration module, a data acquisition engine, and a data parsing engine, solves the problem of numerous interfaces from various vendors, differing interface information, and the need for developers to develop each interface individually when interfacing with multi-vendor SDN interfaces. This saves significant development time and costs, and improves the efficiency of vendor SDN data acquisition. Simultaneously, the data acquisition engine loads the latest acquisition task configuration and interface model configuration in real time. That is, after modifications to model data and configuration data, the data acquisition engine automatically detects them, eliminating the need for customized development and reducing program iterations. The data parsing engine loads the latest model data and model mapping relationship data in real time. Thus, after modifications to model data and configuration data, the data parsing engine automatically detects them, eliminating the need for customized development and reducing program iterations. Furthermore, by inputting configuration data for the interface model, business model, acquisition scheduling task information, and data acquisition information into the configuration interface, the system supports dynamic modification and flexible configuration of the above configuration data, and the program does not need to be modified after modification. Based on the multi-vendor SDN data acquisition engine and data parsing engine loading model data and configuration data, data acquisition from multi-vendor data interfaces is achieved. The data interface integration process has evolved from the original one-to-one development to configurable data collection, enabling low-code development of vendor SDN data collection. The implementation process is simple and has minimal dependence on business systems.
[0118] To better understand this solution, the following flowchart provides another data acquisition method in this application embodiment.
[0119] 1. User configuration includes: configuring the interface model and service model corresponding to each vendor's SDN controller, as well as the mapping relationship between the interface model and the service model, and configuring the data scheduling task and the data acquisition task.
[0120] 2. After data configuration is complete, the configuration information is stored in the data warehouse, and a corresponding data storage table is created in the data warehouse; the data warehouse is the data layer in the above embodiment. Simultaneously, the data acquisition scheduling task and data acquisition service load the configuration data and generate data scheduling instructions and data acquisition instructions.
[0121] 3. After receiving the timed scheduling command, the data acquisition service module starts the data acquisition engine.
[0122] 4. The data acquisition engine requests raw data from the vendor's SDN controller based on data acquisition commands.
[0123] 5. After receiving the data acquisition call request from the data acquisition engine, the vendor's SDN controller returns the corresponding raw data. After receiving the returned raw data, the data acquisition service starts the data parsing engine.
[0124] 6. The data parsing engine uses a hash algorithm to obtain the mapping relationship between the interface model and the business model, and uses the mapping relationship to parse and obtain the target business data corresponding to the original data.
[0125] 7. After the data parsing is completed, the obtained target business data is stored in the data warehouse for upper-layer applications to read, and the entire data collection process ends.
[0126] It should be noted that the data acquisition method provided in this application embodiment can be executed by a data acquisition device or a control module in the data acquisition device for executing the data acquisition method.
[0127] Based on the same inventive concept as the data acquisition method described above, this application also provides a data acquisition apparatus. The following is in conjunction with... Figure 4 The data acquisition apparatus provided in the embodiments of this application will be described in detail.
[0128] Figure 4 This is a schematic diagram illustrating the structure of a data acquisition apparatus according to an exemplary embodiment, which is applied to the above-described... Figure 1 and Figure 2 The data acquisition service module in the system.
[0129] like Figure 4 As shown, the data acquisition device 400 may include:
[0130] The lookup module 410 is used to, upon receiving a data acquisition instruction, look up the target acquisition task information corresponding to the acquisition task ID in the data acquisition instruction according to the preset correspondence between acquisition task information and acquisition task ID. The target acquisition task information includes the interface model ID.
[0131] Assembly module 420 is used to obtain interface configuration information according to the interface model ID, and assemble the calling interface based on the interface configuration information;
[0132] The acquisition module 430 is used to acquire the original data of the SDN controller corresponding to the SDN interface model corresponding to the interface model ID when a communication connection is established with the SDN controller corresponding to the interface model ID by calling the interface;
[0133] The conversion module 440 is used to convert the original data into target business data according to the preset mapping relationship between the SDN interface model and the business model. The original data includes target fields, the target business data includes target business fields, and the mapping relationship includes the mapping relationship between the target fields and the target business fields.
[0134] Storage module 450 is used to store the target business data in a data warehouse.
[0135] In the embodiments of this application, upon receiving a data acquisition instruction, the target acquisition task information corresponding to the acquisition task ID in the data acquisition instruction is searched according to the preset correspondence between acquisition task information and acquisition task ID. This eliminates the need to develop different acquisition tasks one by one. Then, the interface configuration information is obtained based on the interface model ID in the target acquisition task information, and the calling interface is assembled based on the interface configuration information. When a communication connection is established between the calling interface and the SDN controller corresponding to the interface model ID, the raw data of the SDN controller corresponding to the SDN interface model corresponding to the interface model ID can be obtained. In this way, by assembling the calling interface based on the interface configuration information, the raw data of the SDN controller corresponding to the SDN interface model corresponding to the interface model ID can be obtained without developing an interface for each SDN controller, thus achieving low-code development. Finally, according to the preset mapping relationship between the SDN interface model and the business model, the raw data can be directly converted into target business data, and the target business data can be stored in the data warehouse without any other operations, further realizing low-code development and improving the efficiency of target business data acquisition.
[0136] In some embodiments of this application, the above-described apparatus may further include:
[0137] The receiving module is used to receive data acquisition commands at regular intervals.
[0138] In some embodiments of this application, the data acquisition instruction further includes a scheduling task ID, and the lookup module 410 can specifically be used for:
[0139] Based on the correspondence between the preset acquisition task information and the acquisition task ID and scheduling task ID, the target acquisition task information corresponding to the scheduling task ID and acquisition task ID is found.
[0140] In some embodiments of this application, the target field includes at least one of the following: interface model name, interface model code, interface return field name, or interface return field code; the target business field includes at least one of the following: business model name, business model code, business model field name, or business model field code corresponding to the target field.
[0141] In some embodiments of this application, the above-described apparatus may further include:
[0142] The input module is used to receive the first configuration information entered by the user on the configuration page, the first configuration information including the correspondence between the collection task information and the collection task ID;
[0143] The input module is also used to receive second configuration information input by the user on the configuration page. The second configuration information includes an SDN interface model, a service model, and a mapping relationship between the SDN interface model and the service model.
[0144] Alternatively, it can be used to receive third configuration information input by the user on the configuration page, the third configuration information including at least one of the following: change information of the SDN interface model, change information of the service model, and change information of the mapping relationship between the SDN interface model and the service model; and update at least one of the following: the SDN interface model, the service model, and the mapping relationship between the SDN interface model and the service model according to the third configuration information.
[0145] The data acquisition apparatus provided in this application embodiment can be used to execute the data acquisition methods provided in the above method embodiments. The implementation principle and technical effect are similar, and for the sake of brevity, they will not be described again here.
[0146] Based on the same inventive concept, embodiments of this application also provide an electronic device.
[0147] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 5 As shown, the electronic device may include a processor 501 and a memory 502 storing computer programs or instructions.
[0148] Specifically, the processor 501 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of the present invention.
[0149] Memory 502 may include mass storage for data or instructions. For example, and not limitingly, memory 502 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 502 may include removable or non-removable (or fixed) media. Where appropriate, memory 502 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 502 is non-volatile solid-state memory. Memory may include read-only memory (ROM), random-access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, a memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described in the data acquisition methods provided in the above embodiments.
[0150] The processor 501 implements any of the data acquisition methods described in the above embodiments by reading and executing computer program instructions stored in the memory 502.
[0151] In one example, the electronic device may also include a communication interface 503 and a bus 510. Wherein, as... Figure 5 As shown, the processor 501, memory 502, and communication interface 503 are connected through bus 510 and complete communication with each other.
[0152] The communication interface 503 is mainly used to realize communication between various modules, devices, units and / or devices in the embodiments of the present invention.
[0153] Bus 510 includes hardware, software, or both, that couples components of an electronic device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 510 may include one or more buses. Although specific buses are described and illustrated in embodiments of the invention, the invention contemplates any suitable bus or interconnect.
[0154] The electronic device can execute the data acquisition method in the embodiments of the present invention, thereby realizing the data acquisition method described in any of the above embodiments.
[0155] Furthermore, in conjunction with the data acquisition methods described in the above embodiments, this invention can be implemented using a readable storage medium. This readable storage medium stores program instructions, which, when executed by a processor, implement any of the data acquisition methods described in the above embodiments.
[0156] In addition, in conjunction with the data acquisition methods in the above embodiments, the present invention can provide a computer program product, wherein when the instructions in the computer program product are executed by the processor of an electronic device, the electronic device is able to execute any of the data acquisition methods in the above embodiments.
[0157] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.
[0158] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the required tasks. The programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.
[0159] It should also be noted that the exemplary embodiments mentioned in this invention describe methods or systems based on a series of steps or apparatus. However, this invention is not limited to the order of the steps described above; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.
[0160] The aspects of this application have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.
[0161] The above description is merely a specific embodiment of the present invention. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the protection scope of the present invention.
Claims
1. A method for data acquisition, characterized in that, include: Upon receiving a data acquisition instruction, the system searches for the target acquisition task information corresponding to the acquisition task ID in the data acquisition instruction, based on the preset correspondence between acquisition task information and acquisition task ID. The target acquisition task information includes the interface model ID. The interface configuration information is obtained based on the interface model ID, and the interface is assembled and called based on the interface configuration information; When a communication connection is established between the SDN controller corresponding to the interface model ID and the interface call interface, the original data of the SDN controller corresponding to the SDN interface model corresponding to the interface model ID is obtained. Based on the preset mapping relationship between the SDN interface model and the business model corresponding to the interface model ID, the original data is transformed into target business data. The original data includes target fields, the target business data includes target business fields, and the mapping relationship includes a mapping relationship between the target fields, the target business fields, and the business model ID. The target business data is stored in a data warehouse.
2. The method according to claim 1, characterized in that, Before searching for the target acquisition task information corresponding to the acquisition task ID in the data acquisition instruction based on the preset correspondence between acquisition task information and acquisition task ID, the method further includes: Receive data acquisition instructions periodically.
3. The method according to claim 1 or 2, characterized in that, The data acquisition instruction also includes a scheduling task ID. The step of searching for the target acquisition task information corresponding to the acquisition task ID in the data acquisition instruction, based on a preset correspondence between acquisition task information and acquisition task IDs, includes: Based on the preset correspondence between the data acquisition task information and the data acquisition task ID and the scheduling task ID, the target data acquisition task information corresponding to the scheduling task ID and the data acquisition task ID is found.
4. The method according to claim 1, characterized in that, The target field includes at least one of the following: interface model name, interface model code, interface returned field name, or interface returned field code; The target business field includes at least one of the following: business model name, business model code, business model field name, or business model field code.
5. The method according to claim 1, characterized in that, Before searching for the target acquisition task information corresponding to the acquisition task ID in the data acquisition instruction based on the preset correspondence between acquisition task information and acquisition task ID, the method further includes: The system receives first configuration information input by the user on the configuration page. The first configuration information includes the correspondence between the data collection task information and the data collection task ID.
6. The method according to claim 1, characterized in that, Before obtaining the raw data of the SDN controller corresponding to the SDN interface model corresponding to the interface model ID, the method further includes: Receive the second configuration information entered by the user on the configuration page. The second configuration information includes the SDN interface model, the business model, and the mapping relationship between the SDN interface model and the business model. Alternatively, the system receives third configuration information input by the user on the configuration page, the third configuration information including at least one of the following: change information of the SDN interface model, change information of the service model, and change information of the mapping relationship between the SDN interface model and the service model; and updates at least one of the following based on the third configuration information: the SDN interface model, the service model, and the mapping relationship between the SDN interface model and the service model.
7. A data acquisition device, characterized in that, include: The data acquisition scheduling service module is used to obtain the data acquisition task ID and generate data acquisition instructions; The data acquisition service module is used to receive data acquisition instructions sent by the acquisition scheduling service module through the data acquisition engine in the data acquisition service module, and to find the target acquisition task information corresponding to the acquisition task ID in the data acquisition instruction according to the preset correspondence between acquisition task information and acquisition task ID. The target acquisition task information includes the interface model ID. The data acquisition service module is also used to obtain the raw data of the SDN controller through the SDN interface model corresponding to the interface model ID when the data acquisition engine establishes a communication connection with the corresponding SDN controller according to the interface model ID. The data acquisition service module is further configured to convert the raw data into target business data by using the data parsing engine in the data acquisition service module according to the preset mapping relationship between the SDN interface model and the business model corresponding to the interface model ID. The raw data includes target fields, the target business data includes target business fields, and the mapping relationship includes a mapping relationship between the target fields, the target business fields, and the business model ID. The storage module is used to store the target business data in a data warehouse.
8. The apparatus according to claim 7, characterized in that, Also includes: The input module is used to receive the first configuration information entered by the user on the configuration page. The first configuration information includes the correspondence between the collection task information and the collection task ID. The input module is also used to receive second configuration information input by the user on the configuration page. The second configuration information includes an SDN interface model, a service model, and a mapping relationship between the SDN interface model and the service model. Alternatively, it can be used to receive third configuration information input by the user on the configuration page, the third configuration information including at least one of the following: change information of the SDN interface model, change information of the service model, and change information of the mapping relationship between the SDN interface model and the service model; and update at least one of the following: the SDN interface model, the service model, and the mapping relationship between the SDN interface model and the service model according to the third configuration information.
9. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the data acquisition method as described in any one of claims 1-6.
10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the data acquisition method as described in any one of claims 1-6.