Data acquisition methods, devices, electronic equipment and storage media
By parsing filtering information in the NETCONF protocol and performing preprocessing filtering on the data model and container nodes, the problem of low data acquisition efficiency is solved, achieving efficient data acquisition and merging processing and improving user experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING GW TECH
- Filing Date
- 2022-12-14
- Publication Date
- 2026-05-26
Smart Images
Figure CN116186001B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to a data acquisition method, apparatus, electronic device, and storage medium. Background Technology
[0002] In current operator transmission networks, defining management data models (data modules) using the YANG language and managing network elements using the NETCONF protocol has become the mainstream network management method. When managing network elements, the controller needs to collect network element data through the get / get-config interface and subtree filter defined by the NETCONF protocol.
[0003] As network elements need to support increasingly more data models and larger data volumes, NETCONF requires the collection of specific data for subtree filtering.
[0004] The data acquisition method in related technologies involves collecting all data models in full, filtering them one by one, and merging them one by one for output. This method includes processing a large number of invalid data models, which is not only inefficient but also often causes timeouts due to excessive data volume, thus degrading the user experience. Summary of the Invention
[0005] In view of this, this application provides a data acquisition method, apparatus, electronic device, and storage medium that can improve data acquisition efficiency, reduce data processing volume, and enhance user experience.
[0006] To solve the above-mentioned technical problems, the technical solution of this application is implemented as follows:
[0007] In one embodiment, a data acquisition method is provided, the method comprising:
[0008] Obtain filtering information;
[0009] The filtering information is parsed to obtain data model preprocessing information, container node preprocessing information, and data processing information;
[0010] Match data models in the data model set based on the data model preprocessing information;
[0011] In response to the matched data model, container nodes are matched in the data model based on the container node preprocessing information;
[0012] In response to the matched container nodes, retrieve the data corresponding to each matched container node;
[0013] Based on the data processing information, the data corresponding to the obtained container nodes is processed to obtain valid data for each container node;
[0014] For each valid data obtained from the container node, a response data packet is generated corresponding to each matched data model;
[0015] The response data messages corresponding to the matched data models are merged.
[0016] Wherein, when the data model preprocessing information includes a namespace string, the step of matching the data model in the data model set based on the data model preprocessing information includes:
[0017] The namespace string in the data model preprocessing information is matched with the namespace string of the data model in the data model set.
[0018] The step of matching container nodes in the data model based on the container node preprocessing information includes:
[0019] Obtain the outermost container node from the preprocessed container node information;
[0020] Based on the outermost container node in the obtained container node information, match the container node in the corresponding data model.
[0021] Wherein, after obtaining the data corresponding to the matched container nodes respectively, and before processing the obtained data based on the data processing information, the method further includes:
[0022] The data corresponding to the obtained container nodes will be used to generate XML data according to the structure of the corresponding data model.
[0023] The step of processing the data corresponding to the acquired container nodes based on the data processing information to obtain valid data for each container node includes:
[0024] The data of the acquired container nodes is recursively filtered based on the data processing information to obtain the valid data of the container nodes.
[0025] In another embodiment, a data acquisition apparatus is provided, the apparatus comprising:
[0026] The first acquisition unit is configured to acquire filtered information.
[0027] The parsing unit is configured to parse the filtering information to obtain data model preprocessing information, container node preprocessing information, and data processing information.
[0028] The first matching unit is configured to perform data model matching in the data model set based on the data model preprocessing information;
[0029] The second matching unit is configured to perform a response to the matched data model, matching container nodes in the data model based on the container node preprocessing information;
[0030] The second acquisition unit is configured to perform actions in response to the matched container nodes, acquiring the data corresponding to each matched container node.
[0031] The third acquisition unit is configured to process the acquired data corresponding to the container node based on the data processing information, and acquire valid data for each container node.
[0032] The generation unit is configured to generate a response data packet corresponding to each matched data model for the valid data of the acquired container node;
[0033] The merging unit is configured to perform merging processing on the response data packets corresponding to the matched data models.
[0034] in,
[0035] The first matching unit is configured to, when performing data model matching in the data model set based on the data model preprocessing information, use the namespace string in the data model preprocessing information to match the namespace string of the data model in the data model set; wherein, the data model preprocessing information includes the namespace string.
[0036] in,
[0037] The second matching unit is configured to, when performing container node matching in the data model based on the container node preprocessing information, obtain the outermost container node in the container node preprocessing information; and match container nodes in the corresponding data model based on the outermost container node in the obtained container node information.
[0038] In another embodiment, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the data acquisition method described above.
[0039] In another embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps of the data acquisition method.
[0040] As can be seen from the above technical solutions, the embodiments described above obtain data model preprocessing information, container node preprocessing information, and data processing information by parsing filtering information; filter the data model based on the data model preprocessing information, and filter the container nodes of the filtered data model based on the node preprocessing information; process the data corresponding to all filtered container nodes based on the data processing information to obtain valid data, generate response data messages, and merge them to obtain the data to be obtained. The technical solution provided in this application uses two-step preprocessing (data model and container nodes) to filter out most invalid data, and then obtains the full data of the corresponding container nodes for further filtering to finally obtain the required data. This data acquisition method can improve data acquisition efficiency, reduce data processing volume, and improve user experience. Attached Figure Description
[0041] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0042] Figure 1 This is a schematic diagram of a data acquisition process in an embodiment of this application;
[0043] Figure 2 This is a schematic diagram of another data acquisition process in an embodiment of this application;
[0044] Figure 3 This is a schematic diagram of the data acquisition device structure in an embodiment of this application;
[0045] Figure 4 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation
[0046] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0047] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0048] The technical solution of the present invention will be described in detail below with reference to specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.
[0049] In related technologies, NETCONF requires collecting specific data for subtree filtering, and then processing that data. The current collection method involves collecting all data models, filtering them one by one, and then merging the outputs. This method includes processing a large number of invalid data models. Reading all data for each model and merging all filtered outputs are both very time-consuming steps. This is not only inefficient but also frequently causes timeouts due to excessive data volume, degrading the user experience.
[0050] To address the aforementioned issues, this application provides a data acquisition method that involves parsing filtering information to obtain data model preprocessing information, container node preprocessing information, and data processing information. The method filters the data model based on the data model preprocessing information and filters the container nodes of the filtered data model based on the node preprocessing information. The data corresponding to all filtered container nodes is then processed based on the data processing information to obtain valid data, and a response data message is generated and merged to obtain the desired data. The technical solution provided in this application uses two-step preprocessing (data model and container nodes) to filter out most invalid data, and then obtains the full data of the corresponding container nodes for further filtering to finally obtain the required data. This data acquisition method improves data acquisition efficiency, reduces data processing volume, and enhances user experience.
[0051] The data acquisition process in the embodiments of this application will be described in detail below with reference to the accompanying drawings.
[0052] See Figure 1 , Figure 1 This is a schematic diagram of a data acquisition process according to an embodiment of this application. The specific steps are as follows:
[0053] Step 101: Obtain filter information.
[0054] Step 102: Parse the filtering information to obtain data model preprocessing information, container node preprocessing information, and data processing information.
[0055] There is a correspondence between the container node preprocessing information and the model preprocessing information, that is, the data model corresponding to the container node in the container node preprocessing information is indicated.
[0056] Step 103: Match data models in the data model set based on the data model preprocessing information.
[0057] This step involves filtering the data models, removing those from which data is not needed.
[0058] Step 104: In response to the matched data model, match container nodes in the data model based on the container node preprocessing information.
[0059] This step involves filtering the container nodes of the filtered data model, removing container nodes that do not contain data.
[0060] Step 105: In response to the matched container nodes, obtain the data corresponding to each matched container node.
[0061] Step 106: Process the data corresponding to the obtained container nodes based on the data processing information to obtain the valid data of each container node.
[0062] This step is similar to the implementation in related technologies. It involves obtaining valid data from the acquired data. In related technologies, valid data is obtained based on the full amount of acquired data, that is, the data of all container nodes of all data models. However, in this step, valid data is obtained based on the filtered data models and container node data.
[0063] Step 107: Generate a response data packet corresponding to each matched data model for the valid data of the obtained container nodes.
[0064] Step 108: Merge the response data packets corresponding to the matched data models.
[0065] The specific merging process here conforms to the merge flow defined by the standard NETCONF protocol, and will not be described in detail here.
[0066] This application embodiment obtains data model preprocessing information, container node preprocessing information, and data processing information by parsing filtering information; it filters the data model based on the data model preprocessing information, and filters the container nodes of the filtered data model based on the node preprocessing information; it processes the data corresponding to all filtered container nodes based on the data processing information to obtain valid data, generates response data messages, and merges them to obtain the desired data. The technical solution provided in this application embodiment filters out most invalid data through two-step preprocessing (data model and container nodes), then obtains the full data of the corresponding container nodes and performs relevant filtering processing again to finally obtain the required data. This data acquisition method can improve data acquisition efficiency, reduce data processing volume, and improve user experience.
[0067] In some examples,
[0068] In step 103, when the data model preprocessing information includes namespace strings, the data model is matched in the data model set based on the data model preprocessing information, including:
[0069] The namespace strings in the data model preprocessing information are matched with the namespace strings of the data models in the data model collection.
[0070] In step 104, matching container nodes in the data model based on container node preprocessing information includes:
[0071] Retrieve the outermost container node from the preprocessed information of container nodes;
[0072] Based on the outermost container node in the obtained container node information, match the container node in the corresponding data model.
[0073] See Figure 2 , Figure 2 This is a schematic diagram of another data acquisition process in an embodiment of this application. The specific steps are as follows:
[0074] Step 201: Obtain filter information.
[0075] In this embodiment, when addressing the issues of low efficiency and data return timeout when the netconf protocol management device in the communication device obtains specified data through subtree filtering, the obtained filtering information is subtree filter information.
[0076] Step 202: Parse the filtering information to obtain data model preprocessing information, container node preprocessing information, and data processing information.
[0077] There is a correspondence between the container node preprocessing information and the model preprocessing information, that is, the data model corresponding to the container node in the container node preprocessing information is indicated.
[0078] The data model preprocessing information includes a namespace string, which is used to identify a data model;
[0079] The container node preprocessing information includes container node identifiers, meaning that the data corresponding to these container node identifiers is the data that needs to be obtained.
[0080] Data processing information refers to the effective acquisition and use of specific data from container nodes based on the acquired data model.
[0081] By first analyzing the filtered information, it was determined which specific data models' container nodes' data needed to be obtained.
[0082] Step 203: Match data models in the data model set based on the data model preprocessing information.
[0083] This step involves filtering the data models, removing those from which data is not needed.
[0084] Step 204: Obtain the outermost container node from the container node preprocessing information.
[0085] In this embodiment of the application, since the data is stored in a tree structure, when it is necessary to obtain the data of the corresponding container node, it is only necessary to obtain the data corresponding to the outermost container node, and then obtain the data of the corresponding intermediate node and leaf node. Therefore, when determining which container nodes to obtain the data, only the outermost container node needs to be determined, so that it is not necessary to repeatedly obtain the data of container nodes other than the outermost container node.
[0086] The outermost container node, i.e., the path from this container node to the root node of the tree corresponding to the data model, no longer passes through other container nodes in the container node preprocessing information.
[0087] Step 205: Match container nodes in the corresponding data model based on the outermost container node in the obtained container node information.
[0088] This step involves filtering the container nodes of the filtered data model, removing container nodes that do not contain necessary data.
[0089] Step 206: In response to the matched container nodes, obtain the data corresponding to the matched container nodes respectively.
[0090] The matched container nodes are identified, and the specific identification method is not limited. For example, it can be:
[0091] Assuming that data model i defines 8 container nodes, each bit of the hexadecimal number 0xFF is used to identify them. If the container node is needed, the bit is set to 1; otherwise, it is set to 0. This generates the identifier of the container node that needs to be obtained.
[0092] After acquiring the data in this step, the data corresponding to the container nodes is used to generate XML data according to the structure of the corresponding data model. This facilitates data processing and better conforms to the implementation of relevant protocols.
[0093] Step 207: Process the data corresponding to the obtained container nodes based on the data processing information to obtain the valid data of each container node.
[0094] This step is similar to the implementation in related technologies. It involves obtaining valid data from the acquired data. In related technologies, valid data is obtained based on the full amount of acquired data, that is, the data of all container nodes of all data models. However, in this step, valid data is obtained based on the filtered data models and container node data.
[0095] This step recursively filters the data of all acquired container nodes, deleting unnecessary nodes. For each level of container nodes, leaf nodes that are identical to those in the data processing information are retained, while those that are different have their corresponding data deleted.
[0096] The information of the container leaf nodes exists in the container node preprocessing information, and may or may not exist in the data processing information. If it exists, you can use any container node identifier for filtering.
[0097] Step 208: Generate a response data packet corresponding to each matched data model for the valid data of the obtained container nodes.
[0098] The process of generating the reply-data message can be carried out in accordance with the generation methods in the relevant technologies.
[0099] Step 209: Merge the response data packets corresponding to the matched data models.
[0100] The specific merging process here conforms to the merge flow defined by the standard NETCONF protocol, and will not be described in detail here.
[0101] Suppose the current data model set includes 100 data models, i.e. 100 trees. The filtering information indicates that the data to be obtained are the two trees identified as 50 and 60, the data corresponding to container nodes 1 and 2 of the tree identified as 50, and the data corresponding to container nodes 4 and 5 of the tree identified as 60.
[0102] The implementation in related technologies involves directly acquiring all the data corresponding to the 100 trees in the data model set, then obtaining the valid data based on the filtering information, generating response data packets for the data models identified as 50 and 60 based on the obtained valid data, and merging the response data packets of the two data models.
[0103] The technical solution provided by the embodiments of this application is implemented as follows:
[0104] First, identify the two trees identified as 50 and 60, and filter out the data from the other 98 trees. Then, determine which of container node 1 and container node 2 is the outer node. If both are outer nodes, keep both. If there is an inner node, do not match that container node. For example, if container node 2 is an inner node of container node 1, then only match container node 1. Container node 4 and container node 5 are nodes on two different branches of the same tree, so both need to be matched.
[0105] Therefore, it is only necessary to obtain the data of container node 1 of data model 50, and the data corresponding to container node 4 and container node 5 of data model 60.
[0106] After acquisition, the data needs to be processed again to obtain the final valid data. The subsequent implementation is the same as the existing implementation.
[0107] In this embodiment, the data models that do not need to be processed are first filtered out, and then the container nodes that do not need to be processed are filtered out. This greatly reduces the amount of data obtained and processed, significantly improves processing efficiency, and shortens processing time.
[0108] Based on the same inventive concept, this application also provides a data acquisition device. See also Figure 3 , Figure 3 This is a schematic diagram of the data acquisition device in an embodiment of this application. The data acquisition device includes:
[0109] The first acquisition unit 301 is configured to acquire filter information.
[0110] The parsing unit 302 is configured to parse the filtering information to obtain data model preprocessing information, container node preprocessing information, and data processing information;
[0111] The first matching unit 303 is configured to perform data model matching in the data model set based on the data model preprocessing information;
[0112] The second matching unit 304 is configured to perform a response to the matched data model, matching container nodes in the data model based on the container node preprocessing information;
[0113] The second acquisition unit 305 is configured to perform actions in response to the matched container nodes, acquiring the data corresponding to the matched container nodes respectively;
[0114] The third acquisition unit 306 is configured to process the acquired data corresponding to the container node based on the data processing information, and acquire valid data for each container node.
[0115] The generation unit 307 is configured to generate a response data packet corresponding to each matched data model for the valid data of the acquired container node;
[0116] Merging unit 308 is configured to perform merging processing on the response data packets corresponding to the matched data models.
[0117] In another embodiment,
[0118] The first matching unit 303 is configured to, when performing data model matching in the data model set based on the data model preprocessing information, use the namespace string in the data model preprocessing information to match the namespace string of the data model in the data model set; wherein, the data model preprocessing information includes the namespace string.
[0119] In another embodiment,
[0120] The second matching unit 304 is configured to, when performing container node matching in the data model based on the container node preprocessing information, obtain the outermost container node in the container node preprocessing information; and match container nodes in the corresponding data model based on the outermost container node in the obtained container node information.
[0121] The units in the above embodiments can be integrated into one unit or deployed separately; they can be merged into one unit or further divided into multiple sub-units.
[0122] In another embodiment, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of a data acquisition method.
[0123] In another embodiment, a computer-readable storage medium is also provided, having stored thereon computer instructions that, when executed by a processor, can implement the steps in the data acquisition method.
[0124] Figure 4 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention. Figure 4As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute the following methods:
[0125] Obtain filtering information;
[0126] Parse the filtering information to obtain data model preprocessing information, container node preprocessing information, and data processing information;
[0127] Match data models in the data model set based on data model preprocessing information;
[0128] In response to the matched data model, container nodes are matched in the data model based on the preprocessed information of the container nodes;
[0129] In response to the matched container nodes, retrieve the data corresponding to each matched container node;
[0130] Based on the data processing information, the data corresponding to the obtained container nodes is processed to obtain the valid data of each container node;
[0131] For each valid data obtained from the container nodes, a response data packet is generated corresponding to each matched data model;
[0132] The response data messages corresponding to the matched data models are merged.
[0133] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0134] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0135] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0136] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A data acquisition method, characterized by, The method includes: Obtain filtering information; The filtering information is parsed to obtain data model preprocessing information, container node preprocessing information, and data processing information; Match data models in the data model set based on the data model preprocessing information; In response to the matched data model, container nodes are matched in the data model based on the container node preprocessing information; In response to the matched container nodes, retrieve the data corresponding to each matched container node; Based on the data processing information, the data corresponding to the obtained container nodes is processed to obtain valid data for each container node; For each valid data obtained from the container node, a response data packet is generated corresponding to each matched data model; The response data packets corresponding to the matched data models are merged. Wherein, after obtaining the data corresponding to the matched container nodes respectively, and before processing the obtained data based on the data processing information, the method further includes: The data corresponding to the obtained container nodes will be used to generate XML data according to the structure of the corresponding data model.
2. The method according to claim 1, characterized in that, When the data model preprocessing information includes a namespace string, the step of matching the data model in the data model set based on the data model preprocessing information includes: The namespace string in the data model preprocessing information is matched with the namespace string of the data model in the data model set.
3. The method according to claim 1, characterized in that, The matching of container nodes in the data model based on the container node preprocessing information includes: Obtain the outermost container node from the preprocessed container node information; Based on the outermost container node in the obtained container node information, match the container node in the corresponding data model.
4. The method according to any one of claims 1-3, characterized in that, The step of processing the data corresponding to the acquired container nodes based on the data processing information to obtain valid data for each container node includes: The data of the acquired container nodes is recursively filtered based on the data processing information to obtain the valid data of the container nodes.
5. A data acquisition device, characterized in that, The device includes: The first acquisition unit is configured to acquire filtered information. The parsing unit is configured to parse the filtering information to obtain data model preprocessing information, container node preprocessing information, and data processing information. The first matching unit is configured to perform data model matching in the data model set based on the data model preprocessing information; The second matching unit is configured to perform a response to the matched data model, matching container nodes in the data model based on the container node preprocessing information; The second acquisition unit is configured to perform actions in response to the matched container nodes, acquiring the data corresponding to each matched container node. The third acquisition unit is configured to process the acquired data corresponding to the container node based on the data processing information, and acquire valid data for each container node. The generation unit is configured to generate a response data packet corresponding to each matched data model for the valid data of the acquired container node; The merging unit is configured to perform merging processing on the response data packets corresponding to the matched data models; The third acquisition unit is further configured to generate XML data from the acquired data corresponding to the container node according to the structure of the corresponding data model before processing the acquired data corresponding to the container node based on the data processing information.
6. The apparatus according to claim 5, characterized in that, The first matching unit is configured to, when performing data model matching in the data model set based on the data model preprocessing information, use the namespace string in the data model preprocessing information to match the namespace string of the data model in the data model set; wherein, the data model preprocessing information includes the namespace string.
7. The apparatus according to claim 5, characterized in that, The second matching unit is configured to, when performing container node matching in the data model based on the container node preprocessing information, obtain the outermost container node in the container node preprocessing information; and match container nodes in the corresponding data model based on the outermost container node in the obtained container node information.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1-4.
9. A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method as claimed in any one of claims 1-4.