Data processing methods, apparatus, electronic devices and storage media

Abstract

This application relates to the field of computer technology and discloses a data processing method, apparatus, electronic device, and storage medium. The method includes: responding to a data acquisition task, determining a group of acquisition objects corresponding to the data acquisition task; for each group of acquisition objects, grouping the objects in the group based on the number of objects in the group to obtain at least one first group; for each group of acquisition objects, adjusting each first group based on parameter configuration information corresponding to the object data to obtain at least one second group; determining a concurrent thread corresponding to each second group, and acquiring object data of each object in each group of acquisition objects in the first data acquisition cycle based on the concurrent thread corresponding to each second group. Using this application embodiment can improve data processing efficiency during the data acquisition process and has high applicability.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the continuous development of computer technology, large-scale data management scenarios require the collection of object data from each object in a data collection group for further processing. For example, a device management system needs to collect data from each device in a device cluster to achieve unified management of the cluster through data analysis.

[0003] Existing methods for collecting object data often involve a fixed collection approach based on a uniform deployment of a certain number of objects. However, when the number of objects in a collection group is large, the extended collection time leads to reduced data timeliness. Conversely, when the number of objects in a collection group is small, this fixed collection method consumes more resources. For example, the Operation and Maintenance Center (OMC) manages various network equipment (NEs) in an operator's network. The North Manage Agent (NMA) obtains data from each NE through the OMC. As the amount of data from each NE increases, the data collection and reporting time lengthens, failing to meet timeliness requirements. When the number of NEs is small, the NMA consumes significant resources; when the number of NEs is large, the NMA cannot complete data collection in a timely manner.

[0004] Therefore, improving data processing efficiency and reducing resource consumption in the above scenarios has become an urgent problem to be solved. Summary of the Invention

[0005] This application provides a data processing method to address the problems of low data processing efficiency and high resource consumption when collecting data from multiple objects in the prior art.

[0006] Accordingly, embodiments of this application also provide a data processing device, an electronic device, and a storage medium to ensure the implementation and application of the above methods.

[0007] To address the aforementioned problems, this application discloses a data processing method, which includes:

[0008] In response to the data acquisition task, determine the group of acquisition objects corresponding to the above data acquisition task;

[0009] For each of the above-mentioned collection object groups, the objects in the collection object group are grouped based on the number of objects in the collection object group to obtain at least one first group.

[0010] For each of the above-mentioned collection object groups, based on the parameter configuration information corresponding to the object data, the above-mentioned first groups corresponding to the collection object group are adjusted to obtain at least one second group;

[0011] Determine the concurrent threads corresponding to each of the above-mentioned second groups, and based on the concurrent threads corresponding to each of the above-mentioned second groups, collect the object data of each of the above-mentioned objects in each of the above-mentioned collection object groups during the first data collection cycle.

[0012] This application also discloses a data processing apparatus, the apparatus comprising:

[0013] The object determination module is used to determine the group of objects to be collected corresponding to the data collection task in response to the data collection task.

[0014] The initial adjustment module is used to group each of the above-mentioned objects in each of the above-mentioned object groups based on the number of objects in the object group, so as to obtain at least one first group.

[0015] The grouping adjustment module is used to adjust each of the first groups corresponding to each of the above-mentioned collection object groups based on the parameter configuration information corresponding to the object data, so as to obtain at least one second group.

[0016] The data processing module is used to determine the concurrent threads corresponding to each of the above-mentioned second groups, and based on the concurrent threads corresponding to each of the above-mentioned second groups, to collect object data of each of the above-mentioned objects in each of the above-mentioned collection object groups during the first data collection cycle.

[0017] This application also discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the data processing method provided in this application.

[0018] This application also discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements one or more of the methods described above in this application.

[0019] In this embodiment, concurrent threads can significantly improve the efficiency of object data collection, avoiding the risk of object data backlog and latency effects. Furthermore, the parameter configuration information of the object data allows for further adjustment of the first group within each object collection group, making the object grouping corresponding to each object more reasonable, thereby improving data processing efficiency and demonstrating high applicability. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating a data processing method provided in an embodiment of this application;

[0022] Figure 2 This is another schematic flowchart of the data processing method provided in the embodiments of this application;

[0023] Figure 3 This is a schematic diagram of a structure for collecting network data provided in an embodiment of this application;

[0024] Figure 4 This is a schematic diagram of a structure for adjusting network element groups provided in an embodiment of this application;

[0025] Figure 5 This is a schematic diagram of the structure for adjusting network element groups and updating parameter configuration information provided in an embodiment of this application;

[0026] Figure 6 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this application;

[0027] Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0028] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0029] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the word “comprising” as used in the specification of this application means the presence of features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, the term “connected” or “coupled” as used herein can include wireless connections or wireless coupling.

[0030] In the various embodiments of this application, it should be understood that the sequence numbers of the following processes do not imply a specific order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The terms "first," "second," etc., used in the specification and claims are used to distinguish similar objects and are not used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, including all or any unit and all combinations of one or more associated listed items, and the character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0031] In the embodiments provided in this application, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information.

[0032] Furthermore, those skilled in the art will understand that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0033] The data processing method provided in this application can be executed by any electronic device, such as a terminal device or a server. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal can be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein. The data processing method, apparatus, electronic device, and storage medium provided in this application aim to solve at least one of the technical problems existing in the prior art.

[0034] As an example, the data processing method provided in the embodiments of this application can be executed by NMA.

[0035] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0036] See Figure 1 , Figure 1 This is a flowchart illustrating a data processing method provided in an embodiment of this application. For example... Figure 1 As shown, the data processing method provided in this application embodiment may include the following steps:

[0037] Step S11: In response to the data acquisition task, determine the group of acquisition objects corresponding to the data acquisition task.

[0038] In some feasible implementations, each data acquisition task may correspond to at least one group of acquisition objects, and each group of acquisition objects includes at least one object. That is, each data acquisition task is used to acquire object data of each object in at least one group of acquisition objects.

[0039] In this embodiment, the "collection object group" represents the scope of the devices (objects) whose data are being collected, such as all objects in a certain area, all objects of a certain type, etc. The granularity of the scope division can be determined based on the actual application scenario requirements. For example, the collection object group can be used to represent a cell, and each object in the collection object group can be a network element in the cell. As another example, the collection object can be used to represent a base station, and each object in the collection object group can be a network element corresponding to the base station. Yet another example is that the collection object group can be used to represent a region, and each object in the collection object group can be a server cluster in that region, etc.

[0040] In some feasible implementations, for each data acquisition task, the data acquisition task can be used to acquire object data of each object in at least one acquisition object group. For example, based on the data acquisition task, network data of network elements in multiple cells can be acquired. For example, based on the data acquisition task, network data of network elements in multiple base stations can be acquired. Based on this, in response to the data acquisition task, at least one acquisition object group corresponding to the data acquisition task can be determined, and then object data of each object in the acquisition object group can be acquired.

[0041] Step S12: For each collection object group, group the objects in the collection object group according to the number of objects in the collection object group to obtain at least one first group.

[0042] In some feasible implementations, for each group of objects corresponding to a data acquisition task, before collecting the object data of each object in the group, the number of objects in the group can be determined, and then the objects in the group can be grouped based on the number of objects.

[0043] Specifically, a grouping threshold can be determined. If the number of objects in a given object group exceeds the grouping threshold, then the objects in that object group can be grouped according to the grouping threshold. That is, objects at each grouping threshold are designated as one group, and the remaining objects are designated as another group, resulting in multiple groups. If the number of objects in a given object group is less than the grouping threshold, then all objects in that object group are designated as one group. For ease of description, the initial grouping of objects in the given object group will be referred to as the first group.

[0044] For example, if there are 2112 network elements in a cell and the grouping threshold is 500, then every 500 network elements can be assigned to a first group, resulting in 4 first groups. The remaining 112 network elements can then be assigned to a first group, resulting in 5 first groups.

[0045] The grouping thresholds for different data collection object groups can be the same or different, depending on the specific application scenario requirements, and are not restricted here. For example, if the data collection object group represents base stations, the grouping threshold for base stations can be 200, meaning that every 200 network elements in a base station can be considered as a first group. If the data collection object group represents cells, the grouping threshold for cells can be 10, meaning that every 10 network elements in a cell can be considered as a first group.

[0046] Optionally, the object type of each object in the collection object group can be determined, and then the objects in the collection object group can be grouped based on the object type. For example, when grouping the objects in the collection object group, objects of one object type can be used as a first group. Alternatively, the objects can be classified according to their object type first, and for each type of object, the objects of that type can be grouped according to the number of objects in that type and the corresponding grouping threshold, thus obtaining at least one first group. The grouping thresholds corresponding to different object types can be the same or different, and can be determined based on the actual application scenario requirements, without any restrictions here.

[0047] The aforementioned object types can be categorized by device type, such as servers, computers, and mobile phones; by network applicability, including but not limited to 4G, 5G, and 4G / 5G co-mode types; or by device function, such as switches and routers. The specific categorization method can be determined based on actual application scenario requirements and is not limited here.

[0048] Step S13: For each group of objects to be collected, based on the parameter configuration information corresponding to the object data, adjust each of the first groups corresponding to the group of objects to be collected to obtain at least one second group.

[0049] In some feasible implementations, after grouping the objects in each collection object group based on the number of objects and the corresponding grouping threshold, in order to avoid resource waste caused by a small number of objects in each first group, or to avoid low data processing efficiency caused by the number of objects in each first group, the first groups corresponding to the collection object group can be further adjusted based on the parameter configuration information corresponding to the object data to obtain at least one second group.

[0050] Specifically, based on the parameter configuration information corresponding to the object data, the first object weight of the collection object group in the first data collection cycle can be determined. Then, based on the first object weight of the collection object group, each first group corresponding to the collection object group can be adjusted to obtain at least one second group.

[0051] It should be noted that, in this embodiment, the collection of object data for each object in each object group is based on a preset data collection period, that is, the object data for each object in each object group is collected periodically. The aforementioned first data collection period is the first period of the aforementioned preset data collection period.

[0052] The first object weight corresponding to the collection object group can be used to describe the expected total collection time for collecting object data of each object in the collection object group. The larger the first object weight, the longer the expected total collection time for collecting object data of each object in the collection object group; the smaller the first object weight, the shorter the expected total collection time for collecting object data of each object in the collection object group.

[0053] Alternatively, the first object weight corresponding to the group of objects to be collected can be used to indicate the expected amount of object data for each object in the group. The larger the first object weight, the larger the expected amount of object data for each object in the group; the smaller the first object weight, the smaller the expected amount of object data for each object in the group. A larger expected amount of data indicates a longer expected collection time, and a smaller expected amount of data indicates a shorter expected collection time.

[0054] In some feasible implementations, for each group of objects to be collected, when determining the first object weight of the group of objects to be collected in the first data collection cycle based on the above parameter configuration information, the number of parameters corresponding to each parameter type in the object data of each object in the group of objects to be collected, and the type weight corresponding to each parameter type, can be determined first based on the parameter configuration information.

[0055] The parameter configuration information corresponding to the object data can be used to describe the parameters included in the object data of each object in each collection object group, the parameter type of each parameter, the number of parameters of each parameter type, and the type weight of each parameter type.

[0056] The parameter types mentioned above include, but are not limited to, structure types, list types, and numeric types. The specific classification of parameter types can be determined based on the actual application scenario requirements and is not limited here. For example, parameters used to represent IP addresses can be considered as one parameter type, and parameters used to represent fault types can be considered as another parameter type. When obtaining network data of each network element in a cell, based on the above parameter configuration information, it can be determined that the network data of each network element includes structure type parameters, list type parameters, and numeric type parameters, or it can be determined that the network data of each network element includes IP type parameters, channel state type parameters, etc.

[0057] The aforementioned object data may include performance data, fault data, and resource data, such as network element performance data, network element fault data, and network element resource data. The specific data can be determined based on the actual application scenario requirements and is not limited here.

[0058] For each parameter type, the corresponding type weight indicates the estimated acquisition time for a parameter of that type. A larger type weight indicates a longer estimated acquisition time for a parameter of that type, and a smaller type weight indicates a shorter estimated acquisition time for a parameter of that type.

[0059] Alternatively, the type weight corresponding to the group of collected objects can be used to describe the expected data volume of a parameter of that parameter type. The larger the type weight, the larger the expected data volume of a parameter of that parameter type; the smaller the type weight, the smaller the expected data volume of a parameter of that parameter type. In this case, a larger expected data volume indicates a longer expected collection time, and a smaller expected data volume indicates a shorter expected collection time.

[0060] Furthermore, based on the parameter configuration information, the number of parameters corresponding to each parameter type in each object data, as well as the type weight of each parameter type, can be determined. For each parameter type in the object data, the total weight corresponding to that parameter type can be obtained based on the number of parameters and the type weight. For example, for any parameter type, the total weight corresponding to that parameter type in any object data is K. i a i Where i represents the parameter type, a i K represents the type weight corresponding to parameter type i. i This represents the number of parameters of type i in the object data. For this object data, the total weight corresponding to any parameter type can be used to describe the estimated total acquisition time for collecting all parameters of that parameter type in the object data. Alternatively, it can be used to describe the estimated data volume for all parameters of that parameter type in the object data.

[0061] Furthermore, the total weights corresponding to each type of parameter in the object data are summed to obtain the total weight of the object data. For this object data, the total weight can be used to indicate the expected collection time, or the expected data volume. For each object group, the total weights corresponding to the object data of each object in the group are summed to obtain the first object weight for that group in the first data collection cycle.

[0062] In some feasible implementations, for each group of objects to be collected, when grouping the corresponding first groups based on the first object weight of the object group, the correspondence between the first object weight and each preset group number can be determined. Different values ​​of the first object weight correspond to different preset group numbers. Then, the preset group number corresponding to the first object weight is used to adjust each first group of the object group to obtain at least one second group.

[0063] In some feasible implementations, for each group of objects to be collected, when grouping the corresponding first groups based on the first object weight of the object group, the average weight corresponding to the first object weight of each object group can be determined first. Further, the weight ratio between the first object weight of the object group and the corresponding average weight is determined, and then the first groups corresponding to the object group are adjusted based on this weight ratio to obtain at least one second group.

[0064] The average weight of the first object weight in each collection object group can be used to describe the average estimated collection time for collecting object data corresponding to a collection object group in the first data collection cycle. Alternatively, it can be used to describe the average estimated data volume of object data corresponding to a collection object group.

[0065] Based on this, for each group of objects to be collected, the weight ratio corresponding to the weight of the first object in that group can be compared with a first threshold (the first threshold is greater than or equal to 1). If the weight ratio is greater than the first threshold, it indicates that the expected collection time for collecting the object data corresponding to that group is greater than the average expected collection time for collecting the object data corresponding to a group, or that the expected data volume of the object data corresponding to that group is greater than the average expected data volume of the object data corresponding to a group. Therefore, the first groups corresponding to that group can be adjusted so that the number of second groups obtained after adjustment is greater than the number of first groups, that is, the number of objects in each first group corresponding to that group can be reduced, thereby increasing the number of groups.

[0066] Based on this, for each group of objects to be collected, if the weight ratio is less than the first threshold, it indicates that the expected collection time for collecting the object data corresponding to that group is less than the average expected collection time for collecting the object data corresponding to a group of objects, or that the expected data volume for the object data corresponding to that group is less than the average expected data volume for the object data corresponding to a group of objects. Therefore, the first groups corresponding to that group of objects can be adjusted so that the number of groups in the second group after adjustment is less than the number of groups in the first group, that is, the number of objects in each first group corresponding to that group of objects can be increased, thereby reducing the number of groups.

[0067] Optionally, the correspondence between the weight ratio and the number of preset groups in the grouping list can be determined, that is, different values ​​of the weight ratio correspond to different preset groups. Then, based on the preset group number corresponding to the weight ratio, each first group in the collection object group is adjusted to obtain at least one second group.

[0068] Optionally, adjustments can be made directly to each of the corresponding first groups based on the weight ratio. For example, if a weight ratio greater than 1 is determined, this ratio is rounded to the nearest integer, and the resulting value is used to determine the number of groups to be added. If the weight ratio is less than 1, a weight ratio less than 1 is determined, and then, based on the correspondence between the weight ratio less than 1 and the preset number of groups, the number of groups to be reduced is determined. Then, based on the number of groups to be added or reduced, each of the first groups corresponding to the data collection object group is adjusted to obtain at least one second group.

[0069] Based on the above method, the first group corresponding to each collection object group can be adjusted again, so that the expected collection time or expected data volume of the object data corresponding to each second group in each collection object group is similar.

[0070] Step S14: Determine the concurrent threads corresponding to each second group, and collect object data of each object in each collection object group during the first data collection cycle based on the concurrent threads corresponding to each second group.

[0071] In some feasible implementations, the above approach can make the expected collection time or expected data volume of the object data corresponding to each second group in each object collection group similar. However, since acquiring the object data of each object through a separate thread still leads to low overall data acquisition efficiency, a concurrent thread is generated for each second group, and then the object data of each object in the corresponding second group is collected based on each concurrent thread in the first data collection cycle.

[0072] Based on this, in this embodiment, concurrent threads can greatly improve the efficiency of object data collection, avoiding the risk of object data backlog and delay effects. Furthermore, by adjusting the first group within each object collection group, the collection time of the object data corresponding to each object is made similar, thus better adapting to the timeliness requirements of periodic data collection.

[0073] In some feasible implementations, after the first data acquisition cycle ends, in other data acquisition cycles, object data of each object in each second group can be obtained based on each concurrent thread corresponding to each second group of each acquisition object group.

[0074] Optionally, after the end of any data collection cycle, the data processing method provided in this application embodiment may also refer to... Figure 2 .like Figure 2 As shown, Figure 2 This is another schematic flowchart of the data processing method provided in this application embodiment. The data processing method provided in this application embodiment may further include the following steps:

[0075] Step S21: Determine the total collection time of object data for each object group in any data collection cycle.

[0076] In some feasible implementations, for each group of objects to be collected, when collecting object data of each object in the group during each data collection cycle, the total collection time of all object data corresponding to the group can be calculated.

[0077] Step S22: For each group of objects to be collected, based on the total collection time of the object data corresponding to the group of objects in any data collection cycle, determine the second object weight of the group of objects in the next data collection cycle. Based on the second object weight of the group of objects, adjust the grouping of each object in the group of objects in any data collection cycle to obtain at least one third group.

[0078] In some feasible implementations, for each group of objects to be collected, after collecting all object data corresponding to the group of objects in any data collection cycle, the second object weight of the group of objects in the next data collection cycle can be determined based on the total collection time of the object data corresponding to the group of objects in any data collection cycle.

[0079] For example, the total collection time of object data for a given object group in any data collection cycle can be determined as the second object weight for that object group in the next data collection cycle. The second object weight for that object group can be used to describe the actual total collection time of object data collection for each object in that object group in any data collection cycle, and thus determine it as the expected total collection time of object data collection for each object in that object group in the next data collection cycle. A larger second object weight indicates a longer expected total collection time of object data collection for each object in that object group in the next data collection cycle; conversely, a smaller second object weight indicates a shorter expected total collection time of object data collection for each object in that object group in the next data collection cycle.

[0080] Alternatively, the second object weight corresponding to the object group can be used to describe the actual amount of object data for each object in the object group collected in any data collection cycle, and then used to determine the expected amount of object data for each object in the object group collected in the next data collection cycle. A larger second object weight indicates a larger expected amount of object data for each object in the object group collected in the next data collection cycle; a smaller second object weight indicates a smaller expected amount of object data for each object in the object group collected in the next data collection cycle. A larger expected amount of data indicates a longer expected total collection time, and a smaller expected amount of data indicates a shorter expected total collection time.

[0081] For example, the second object weight of the object group in the second data collection period can be determined based on the total collection time of the object data corresponding to the first data collection period.

[0082] Furthermore, after determining the second object weights for each data collection object group in the next data collection cycle, for each data collection object group, the grouping of each object in that group in any data collection cycle can be adjusted based on the second object weights of that group to obtain a new grouping for the next data collection cycle (hereinafter referred to as the third grouping for ease of description). For example, based on the second object weights of that data collection object group in the second data collection cycle, the second grouping of each object in that group in the first data collection cycle can be adjusted to obtain at least one third grouping for the second data collection cycle.

[0083] Specifically, for each group of data collection objects, based on the second object weight of that group, the grouping of each object in that group for any data collection cycle is adjusted to obtain at least one third grouping for the next data collection cycle. For details on this implementation, please refer to [link to relevant documentation]. Figure 1 The implementation method shown in step S13, which adjusts the first group corresponding to the first object weight of the collected object group to obtain at least one second group, will not be described in detail here.

[0084] Step S23: Determine the concurrent threads corresponding to each third group, and based on the concurrent threads corresponding to each third group, collect the object data of each object in each collection object group in the next data collection cycle.

[0085] In some feasible implementations, after obtaining the third group corresponding to each collection object group for the next data collection cycle, the concurrent thread corresponding to each third group can be determined, and then the object data of each object can be obtained based on the concurrent thread of each third group.

[0086] Based on this, in the embodiments of this application, for each group of objects to be collected, the grouping of each object can be adjusted in real time in any data collection cycle after the first data collection cycle, based on the total collection time of the object data in the previous data collection cycle. This makes the total collection time of the object data corresponding to each group of objects in each data collection cycle closer, thereby further improving data collection efficiency, enhancing the uniformity of periodically collected object data, and making it highly applicable.

[0087] In some feasible implementations, after the end of each data collection cycle, the object data corresponding to the collected object group can be reported.

[0088] Specifically, the object data corresponding to each collected object can be summarized, and the summarized data can be further processed according to a preset format. Then, a data file can be generated based on the format-processed summarized data, and the data file can be reported.

[0089] In some feasible implementations, for each parameter type, the type weight is used to indicate the expected acquisition time for a parameter of that type, or the expected data volume for a parameter of that type. Therefore, after any data acquisition cycle, the type weights corresponding to each parameter type can be further adjusted for application in new data acquisition tasks.

[0090] Specifically, for each group of objects to be collected, the third object weight of that group in any given data collection period is determined based on the total collection time of the object data corresponding to that group in any given data collection period. For example, the total collection time of the object data corresponding to that group in any given data collection period can be used as the third object weight of that group in any given data collection period.

[0091] The third object weight corresponding to the collection object group can be used to describe the actual total collection time for collecting object data of each object in the collection object group in any data collection cycle. The larger the third object weight, the longer the actual total collection time for collecting object data of each object in the collection object group in any data collection cycle; the smaller the third object weight, the shorter the actual total collection time for collecting object data of each object in the collection object group in any data collection cycle.

[0092] Alternatively, the third object weight corresponding to the group of objects being collected can be used to describe the actual amount of object data for each object in the group of objects collected in any given data collection cycle. A larger third object weight indicates a larger actual amount of object data for each object in the group of objects collected in any given data collection cycle; conversely, a smaller third object weight indicates a smaller actual amount of object data for each object in the group of objects collected in the next data collection cycle. In this context, a larger actual data amount indicates a longer total actual collection time, and a smaller actual data amount indicates a shorter total actual collection time.

[0093] For example, the weight of the third object in the first data collection period can be determined based on the total collection time of the object data corresponding to the first data collection period of the object group.

[0094] Furthermore, since the first object weight of each collection object group is determined based on the number of parameters corresponding to each parameter type in the object data of each object in the corresponding collection object group, and the type weight of each parameter type, a new type weight corresponding to each parameter type can be determined based on the number of parameters corresponding to each parameter type in the object data of each object in each collection object group, and the corresponding third object weight.

[0095] After obtaining the new type weights corresponding to each parameter type, the parameter configuration information is updated based on the new type weights corresponding to each parameter type. That is, the original type weights corresponding to each parameter type in the parameter configuration information are updated so that when a new data acquisition task is executed, the first group corresponding to each collection object group in the first data acquisition cycle is adjusted based on the new type weights of each parameter type, so as to obtain object data based on the adjusted group.

[0096] Based on this, in this embodiment of the application, by updating the type weight in the parameter configuration information, the object weight determined based on the type weight can better meet the actual data collection scenario requirements, thereby making the data collection duration corresponding to the group obtained after grouping adjustment based on the object weight more similar, thereby further improving data collection efficiency, enhancing the uniformity of periodically collected object data, and having high applicability.

[0097] See below. Figure 3 The data processing method provided in the embodiments of this application will be further described. See [link to relevant documentation]. Figure 3 , Figure 3 This is a schematic diagram of a structure for collecting network data provided in an embodiment of this application. Wherein, Figure 3 The network element mentioned is the object in the embodiments of this application, and Figure 3The data processing method can be implemented through the data acquisition unit and dynamic network element grouping logic in NMA.

[0098] Specifically, in response to the data acquisition task, the data acquisition unit transmits the dynamic network element grouping logic into the object information. The dynamic network element grouping logic determines the network elements in each acquisition object group corresponding to the data acquisition task based on the object information. Then, the dynamic network element grouping logic groups each object in each acquisition object group based on the number of objects in each acquisition object group, to obtain at least one first group.

[0099] Furthermore, for each acquisition object group, the dynamic network element grouping logic adjusts each of the first groups corresponding to the acquisition object group based on the parameter configuration information corresponding to the object data to obtain at least one second group, and returns the network element group to the data acquisition unit.

[0100] Furthermore, the data acquisition unit generates a concurrent thread for each group of network elements based on the received packet information (i.e., Figure 3 The data acquisition unit uses concurrent threads to collect network data for each network element. Simultaneously, it generates a central control thread to monitor the operation of each concurrent thread, aggregates the collected network data, processes it according to a preset format, and finally generates a data file which is then uploaded to a File Transfer Protocol (FTP) server.

[0101] After acquiring network data in each data acquisition cycle, the data acquisition unit determines the total acquisition duration of the network data corresponding to each acquisition object group and feeds it back to the dynamic network element grouping logic. Based on the received total acquisition duration, the dynamic network element grouping logic adjusts the network element groups corresponding to the previous data acquisition cycle and returns the new grouping information to the data acquisition unit. The data acquisition unit can then acquire network data based on the new grouping information, thereby using the new network element groups and corresponding concurrent threads.

[0102] For details on the process of dynamically adjusting the initial network element grouping logic, please refer to [link to relevant documentation]. Figure 4 . Figure 4 This is a schematic diagram of a structure for adjusting network element groups provided in an embodiment of this application. For example... Figure 4 As shown, for each collection object group, after determining the number of network elements it includes, the dynamic network element grouping logic groups the network elements in the collection object group according to the network data, and then determines the object weight of the collection object group based on the parameter configuration information, so as to adjust the previous network element grouping according to the object weight to obtain a new network element grouping.

[0103] For details on the dynamic network element grouping logic adjustment process of adjusting network element groups and updating parameter configuration information during the continuous data acquisition cycle, please refer to [link to relevant documentation]. Figure 5 . Figure 5 This is a schematic diagram illustrating the structure for adjusting network element groups and updating parameter configuration information provided in an embodiment of this application. For example... Figure 5 As shown, the dynamic network element grouping logic determines the number of parameters of each parameter type and the type weight of each parameter type based on the parameter configuration information, and then determines the object weight of each collection object group. The dynamic network element grouping logic determines the average weight corresponding to the object weight of each collection object group, and then determines the weight ratio of the object weight to the average weight of each collection object group. The network element grouping corresponding to the corresponding collection object group is adjusted according to the weight ratio.

[0104] Furthermore, before the start of the next data collection cycle, the dynamic network element grouping logic determines the total collection duration of network data corresponding to each collection object group, determines the new object weight of the corresponding collection object group based on the total collection duration, and then adjusts the network element grouping of the corresponding collection object group in the previous data collection cycle based on the new object weight.

[0105] On the other hand, after determining the new object weights of each collection object group in any data collection cycle, the dynamic network element grouping logic can determine the new type weights of each type parameter based on the new object weights, and then update the parameter configuration information based on the new type weights.

[0106] Based on this, a dynamic concurrency mechanism is incorporated into the northbound data processing based on NMA, ensuring a dynamic balance in the generation of northbound tasks. Furthermore, during task initialization, the number of concurrent tasks for each collection group is determined by object weights, allowing for real-time updates to the concurrency level based on the actual running conditions. Additionally, by inferring type weights, a reference can be provided for subsequent deployments and version updates.

[0107] Based on the same principles as the methods provided in the embodiments of this application, the embodiments of this application also provide a data processing apparatus, such as... Figure 6 As shown, Figure 6 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this application. Figure 6 The provided data processing apparatus includes:

[0108] The object determination module 61 is used to determine the group of objects to be collected corresponding to the data acquisition task in response to the data acquisition task.

[0109] The initial grouping module 62 is used to group each of the above-mentioned objects in the collection object group based on the number of objects in the collection object group, so as to obtain at least one first group.

[0110] The grouping adjustment module 63 is used to adjust each of the first groups corresponding to each of the above-mentioned collection object groups based on the parameter configuration information corresponding to the object data, so as to obtain at least one second group.

[0111] The data processing module 64 is used to determine the concurrent threads corresponding to each of the above-mentioned second groups, and based on the concurrent threads corresponding to each of the above-mentioned second groups, to collect object data of each of the above-mentioned objects in each of the above-mentioned collection object groups during the first data collection cycle.

[0112] In some feasible implementations, for each of the aforementioned groups of objects to be collected, the grouping adjustment module 63 is used to:

[0113] Based on the above parameter configuration information, the first object weight of the collection object group in the first data collection cycle is determined;

[0114] Based on the first object weight of the collection object group, the first groups corresponding to the collection object group are adjusted to obtain at least one second group.

[0115] In some feasible implementations, for each of the aforementioned groups of objects to be collected, the grouping adjustment module 63 is used to:

[0116] Based on the above parameter configuration information, determine the number of parameters corresponding to each parameter type in the object data of each of the above objects in the collection object group, as well as the type weight of each of the above parameter types;

[0117] Based on the number of parameters corresponding to each parameter type in the object data of each of the above objects in the collection object group, and the type weight of each parameter type, the first object weight of the collection object group in the first data collection cycle is determined.

[0118] In some feasible implementations, for each of the aforementioned groups of objects to be collected, the grouping adjustment module 63 is used to:

[0119] Determine the average weight corresponding to the first object weight of each of the above-mentioned object groups;

[0120] Determine the weight ratio between the first object weight of the collection object group and the aforementioned average weight, and adjust each of the aforementioned first groups corresponding to the collection object group based on the aforementioned weight ratio to obtain at least one second group.

[0121] In some feasible implementations, for each of the above-mentioned collection object groups, if the first weight ratio corresponding to the collection object group is greater than the first threshold, then the number of groups in the second group corresponding to the collection object group is greater than the number of groups in the first group corresponding to the collection object group; if the first weight ratio corresponding to the collection object group is less than the first threshold, then the number of groups in the second group corresponding to the collection object group is less than the number of groups in the first group corresponding to the collection object group.

[0122] In some feasible implementations, the grouping adjustment module 63 described above is further used for:

[0123] Determine the total collection duration of object data for each of the above-mentioned object groups in any data collection cycle;

[0124] For each of the above-mentioned object groups, based on the total collection time of the object data corresponding to the above-mentioned object group in any of the above-mentioned data collection cycles, the second object weight of the object group in the next data collection cycle is determined. Based on the second object weight of the object group, the grouping of each of the above-mentioned objects in the object group in any of the above-mentioned data collection cycles is adjusted to obtain at least one third group.

[0125] The aforementioned data processing module 64 is also used for:

[0126] Determine the concurrent threads corresponding to each of the aforementioned third groups, and based on the concurrent threads corresponding to each of the aforementioned third groups, collect the object data of each of the aforementioned objects in each of the aforementioned object groups in the next data collection cycle.

[0127] In some feasible implementations, the grouping adjustment module 63 described above is further used for:

[0128] For each of the above-mentioned groups of objects, the third object weight of the object in any of the above-mentioned data collection cycles is determined based on the total collection time of the object data corresponding to the group of objects in any of the above-mentioned data collection cycles.

[0129] Based on the number of parameters corresponding to each parameter type in the object data of each of the above-mentioned objects in each of the above-mentioned object groups, and the corresponding third object weight, a new type weight corresponding to each of the above-mentioned parameter types is determined.

[0130] The parameter configuration information is updated based on the new type weights to adjust the grouping of objects based on the updated parameter configuration information when executing new data acquisition tasks.

[0131] In some feasible implementations, the data processing module 64 described above is further configured to:

[0132] After each data collection cycle, the object data corresponding to each of the above-mentioned collection object groups will be reported.

[0133] The data processing device provided in this application embodiment can achieve... Figures 1 to 2 The various processes implemented in the method embodiments are not described in detail here to avoid repetition.

[0134] Based on the same principles as the methods shown in the embodiments of this application, embodiments of this application also provide an electronic device, which may include, but is not limited to: a processor and a memory; the memory for storing computer programs; and the processor for executing the data processing method shown in any optional embodiment of this application by calling the computer program.

[0135] In an alternative embodiment, an electronic device, such as Figure 7 As shown, Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Figure 7 The illustrated electronic device 7, which collects data from a group of objects, can be a server, including a processor 7001 and a memory 7003. The processor 7001 and the memory 7003 are connected, for example, via a bus 7002. Optionally, the electronic device 7 may further include a transceiver 7004. It should be noted that in practical applications, the transceiver 7004 is not limited to one unit, and the structure of this electronic device 7's data collection group does not constitute a limitation on the embodiments of this application.

[0136] Processor 7001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 7001 may also be a combination that implements computing functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0137] Bus 7002 may include a pathway for transmitting information between the aforementioned components. Bus 7002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 7002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0138] The memory 7003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0139] The memory 7003 stores application code that executes the scheme of this application, and its execution is controlled by the processor 7001. The processor 7001 executes the application code stored in the memory 7003 to implement the content shown in the foregoing method embodiments.

[0140] Among them, electronic devices include, but are not limited to: mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (such as in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 7 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0141] The server provided in this application can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud mobile communication, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0142] This application provides a computer-readable storage medium storing a computer program that, when run on a computer, enables the computer to execute the corresponding content in the aforementioned method embodiments.

[0143] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0144] It should be noted that the computer-readable medium described above in this application can be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0145] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0146] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0147] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the data processing methods provided in the various optional implementations described above.

[0148] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0149] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0150] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the module itself; for example, the object determination module can also be described as "determining the group of acquisition objects corresponding to the data acquisition task".

[0151] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A data processing method, characterized by, The method comprises: in response to a data collection task, determining a collection object group corresponding to the data collection task; for each of the collection object groups, grouping each of the objects in the collection object group based on the number of objects in the collection object group, to obtain at least one first group; for each of the collection object groups, determining a first object weight of the collection object group in a first data collection period based on parameter configuration information corresponding to the object data, determining an average weight corresponding to the first object weight of each of the collection object groups, determining a weight ratio of the first object weight of the collection object group to the average weight, and adjusting each of the first groups corresponding to the collection object group based on the weight ratio, to obtain at least one second group; determining concurrent threads corresponding to each of the second groups, and collecting object data of each of the objects in each of the collection object groups in the first data collection period based on the concurrent threads corresponding to each of the second groups.

2. The method of claim 1, wherein, For each of the collection object groups, the determination of the first object weight of the collection object group in the first data collection period based on the parameter configuration information comprises: determining, based on the parameter configuration information, the number of parameters corresponding to each parameter type in the object data of each of the objects in the collection object group, and the type weight of each of the parameter types; determining the first object weight of the collection object group in the first data collection period based on the number of parameters corresponding to each of the parameter types in the object data of each of the objects in the collection object group, and the type weight of each of the parameter types.

3. The method of claim 1, wherein, For each of the collection object groups, if the first weight ratio corresponding to the collection object group is greater than a first threshold, the number of second groups corresponding to the collection object group is greater than the number of first groups corresponding to the collection object group; if the first weight ratio corresponding to the collection object group is less than the first threshold, the number of second groups corresponding to the collection object group is less than the number of first groups corresponding to the collection object group.

4. The method of claim 1, wherein, The method further comprises: determining the total collection time of the object data corresponding to each of the collection object groups in any data collection period; for each of the collection object groups, determining a second object weight of the collection object group in a next data collection period of the any data collection period based on the total collection time of the object data corresponding to the collection object group in the any data collection period, adjusting the grouping of each of the objects in the collection object group corresponding to the any data collection period based on the second object weight of the collection object group, to obtain at least one third group; determining concurrent threads corresponding to each of the third groups, and collecting object data of each of the objects in each of the collection object groups in the next data collection period based on the concurrent threads corresponding to each of the third groups.

5. The method of claim 4, wherein, The method further comprises: for each of the collection object groups, determining a third object weight of the collection object in the any data collection period based on the total collection time of the object data corresponding to the collection object group in the any data collection period; determine a new type weight corresponding to each parameter type based on a parameter quantity of a parameter corresponding to each parameter type in object data of each object in each of the collection object groups and a corresponding third object weight; update the parameter configuration information based on the new type weight to adjust the grouping of objects based on the updated parameter configuration information when performing a new data collection task.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: report object data corresponding to each of the collection object groups after each data collection period ends.

7. A data processing apparatus, characterized by, The apparatus includes: an object determination module configured to determine a collection object group corresponding to a data collection task in response to the data collection task; an initial adjustment module configured to group each of the objects in each of the collection object groups based on an object quantity of the objects in the collection object group to obtain at least one first group; a group adjustment module configured to determine a first object weight of each of the collection object groups based on parameter configuration information of object data of the objects in the collection object group for each of the collection object groups, determine an average weight corresponding to the first object weights of each of the collection object groups, determine a weight proportion of the first object weight of the collection object group to the average weight, and adjust each of the first groups corresponding to the collection object group based on the weight proportion to obtain at least one second group; a data processing module configured to determine a concurrent thread corresponding to each of the second groups and collect object data of each of the objects in each of the collection object groups in a first data collection period based on the concurrent thread corresponding to each of the second groups.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the method of any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program implements the method of any one of claims 1 to 6 when executed by the processor.

Images