Monitoring parameter effect evaluation method and device, communication equipment and storage medium

Abstract

The invention relates to a monitoring parameter effect evaluation method and device, communication equipment, a storage medium and a computer program product. The method comprises the following steps: determining a monitoring demand and a monitoring effect threshold value of a target network according to a communication service of the target network; monitoring parameter effect analysis indexes related to the monitoring requirements are selected from the monitoring parameter index set, corresponding weight coefficients are distributed to the monitoring parameter effect analysis indexes, and a monitoring parameter effect evaluation formula is generated; and taking the network state data corresponding to the current monitoring parameter set and the index data related to the monitoring parameter set as the input of a monitoring parameter effect evaluation formula, calculating the score of the current monitoring parameter set, and judging whether the monitoring effect of the monitoring parameter set reaches a monitoring effect threshold value or not. By adopting the method, the monitoring effect of the current monitoring parameter set can be objectively and comprehensively evaluated.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, communication device, storage medium, and computer program product for evaluating the effectiveness of monitoring parameters. Background Technology

[0002] As network technologies and architectures continue to evolve and upgrade, their scale grows exponentially, leading to a significant increase in the dimensions and volume of data required for network operation, monitoring, and data collection. To improve the efficiency of network monitoring and operation, the network parameter set can be dimensionality-reduced to obtain a minimal set of monitoring parameters, thereby reducing the amount of monitoring and data collection required. However, currently, there is a lack of methods to evaluate the dimensionality-reduced parameter set to ensure its objectivity and accuracy.

[0003] Traditional parameter set evaluation methods typically rely on experts to determine evaluation dimensions and metrics based on experience. Different categories of parameters are then manually combined by administrators based on expert experience to evaluate specific communication services. In multimodal network scenarios, because each service has its unique characteristics, administrators must manually combine different parameters to monitor each service individually.

[0004] However, different communication services have different monitoring requirements, and these requirements may change dynamically. Traditional parameter set evaluation methods require manually combining different parameters to monitor services one by one, which is inefficient and highly subjective, and cannot guarantee the objectivity and comprehensiveness of the parameter set evaluation results. Summary of the Invention

[0005] This application provides a method, apparatus, communication device, storage medium, and computer program product for evaluating the effectiveness of monitoring parameters, which can bring the beneficial effect of objectively and comprehensively evaluating the monitoring effect of the current set of monitoring parameters.

[0006] A method for evaluating the effectiveness of monitoring parameters, the method comprising:

[0007] Based on the communication services of the target network, determine the monitoring requirements and monitoring effect thresholds of the target network;

[0008] Select monitoring parameter effect analysis indicators related to the monitoring requirements from the set of monitoring parameter indicators, assign weight coefficients to the monitoring parameter effect analysis indicators, and generate a monitoring parameter effect evaluation formula.

[0009] Using the network status data corresponding to the current monitoring parameter set and the indicator data related to the monitoring parameter set as inputs to the monitoring parameter effect evaluation formula, the score of the current monitoring parameter set is calculated to determine whether the monitoring effect of the monitoring parameter set has reached the monitoring effect threshold.

[0010] In one embodiment, the method further includes:

[0011] Construct the monitoring parameter indicator set; the monitoring parameter indicator set includes: indicator data related to the monitoring parameter set; the related indicator data includes monitoring complexity and monitoring coverage.

[0012] In one embodiment, the method further includes:

[0013] Key indicators are identified and extracted from historical data using data mining or machine learning methods, and these key indicators are then added to the set of monitoring parameter indicators.

[0014] In one embodiment, determining the monitoring requirements and monitoring effect thresholds of the target network based on the communication services of the target network includes:

[0015] Based on the communication services described by the multimodal communication service network elements and the topology model, determine the current monitoring requirements and monitoring effect thresholds for the target network.

[0016] In one embodiment, the step of selecting monitoring parameter effect analysis indicators related to the monitoring requirement from the set of monitoring parameter indicators and assigning weight coefficients to the monitoring parameter effect analysis indicators includes:

[0017] The decision analysis method is used to select monitoring parameter effect analysis indicators that meet the monitoring requirements and objectives from the set of monitoring parameter indicators, and weight coefficients are assigned to the monitoring parameter effect analysis indicators.

[0018] In one embodiment, using network status data corresponding to the current monitoring parameter set and indicator data related to the monitoring parameter set as inputs to the monitoring parameter effect evaluation formula includes:

[0019] The network status data corresponding to the current monitoring parameter set provided by the multimodal communication service network element and the topology model, as well as the indicator data related to the current monitoring parameter set, will be used as the input to the monitoring parameter effect evaluation formula.

[0020] In one embodiment, calculating the score of the current monitoring parameter set and determining whether the monitoring effect of the monitoring parameter set reaches the monitoring effect threshold includes:

[0021] Calculate the score of the current monitoring parameter set to obtain the evaluation result of this round of monitoring parameter set;

[0022] The evaluation results of the current monitoring parameter set and the monitoring effect threshold are passed to the minimum monitoring parameter generation model so that the minimum monitoring parameter generation model can optimize the selection of the monitoring parameter set and determine the minimum monitoring parameter set of the target network through iteration.

[0023] In one embodiment, the method further includes:

[0024] The weight allocation for different communication services can be trained using machine learning methods to obtain a fitting scheme for the weight allocation of indicators that adapts to the monitoring needs of different communication services.

[0025] In one embodiment, the method further includes:

[0026] Based on the monitoring needs and objectives of communication services in different target networks, the indicators in the monitoring parameter indicator set are weighted using an indicator allocation weight model. The consistency of the current weight allocation is then checked to obtain the check results.

[0027] The weight allocation is adjusted based on the test results to generate evaluation formulas for the monitoring parameters corresponding to different communication services.

[0028] In one embodiment, the monitoring parameter index set includes a target layer, a criterion layer, and a scheme layer, and constructing the monitoring parameter index set includes:

[0029] Using the hierarchical analysis of indicators, a hierarchical relationship is established among the indicators in the monitoring parameter indicator set; the target layer in the hierarchical relationship is the score of the monitoring parameter set, the scheme layer is the monitoring parameter set, and the criterion layer contains multi-level indicators, including at least a primary indicator and a secondary indicator corresponding to each primary indicator.

[0030] In one embodiment, the step of assigning weights to the indicators in the monitoring parameter indicator set using an indicator allocation weight model includes:

[0031] For monitoring needs and objectives of different communication services, a subjective weighting method is used to assign weight coefficients to the primary indicators; the score of the monitoring parameter set is obtained by weighted calculation of the primary indicators, and the primary indicators are obtained by weighted calculation of the secondary indicators;

[0032] The objective weighting method is used to assign weight coefficients to the secondary indicators.

[0033] In one embodiment, after adjusting the weight allocation based on the test results and generating evaluation formulas for the monitoring parameters corresponding to different communication services, the method further includes:

[0034] The monitoring parameter set is evaluated based on the aforementioned monitoring parameter effect evaluation formula to obtain the evaluation results;

[0035] Based on the evaluation results and the corresponding feedback information, weight allocations are assigned to different communication services, and machine learning methods are used to train a weight allocation model that adapts to different communication services.

[0036] In one embodiment, the multimodal communication service network element and topology model includes a twin system event simulation module and a monitoring parameter analysis module. Before calculating the score of the current monitoring parameter set and determining whether the monitoring effect of the monitoring parameter set has reached the monitoring effect threshold, using network status data corresponding to the current monitoring parameter set and indicator data related to the monitoring parameter set as inputs to the monitoring parameter effect evaluation formula, the method further includes:

[0037] The twin system event simulation module simulates and generates events according to the proposed scenario, and transmits the events to the network environment simulation system so that the network environment simulation system can generate events.

[0038] The event information is transmitted to the monitoring parameter analysis module so that the monitoring parameter analysis module can analyze and compare the event.

[0039] In one embodiment, after optimizing the selection of the monitoring parameter set by the minimum monitoring parameter generation model and iteratively determining the minimum monitoring parameter set of the target network, the method further includes:

[0040] The monitoring parameter relationship analysis module optimizes the monitoring parameter relationships in the monitoring parameter set based on the evaluation results provided by the monitoring parameter effect analysis module, and then passes the optimization results to the monitoring parameter relationship pattern repository. The monitoring parameter relationship analysis module, the monitoring parameter relationship pattern repository, and the monitoring parameter selection and aggregation module optimize and produce the minimum monitoring parameter set for the current target network based on the evaluation results of the monitoring parameter set.

[0041] A monitoring parameter effect analysis device, the device comprising:

[0042] The determination module is used to determine the monitoring requirements and monitoring effect thresholds of the target network based on the communication services of the target network.

[0043] The indicator weight allocation module is used to select monitoring parameter effect analysis indicators related to the monitoring requirements from the monitoring parameter indicator set, assign weight coefficients to the monitoring parameter effect analysis indicators, and generate monitoring parameter effect evaluation formulas.

[0044] The effect evaluation module is used to calculate the score of the current monitoring parameter set by taking the network status data corresponding to the current monitoring parameter set and the indicator data related to the monitoring parameter set as inputs to the monitoring parameter effect evaluation formula, and to determine whether the monitoring effect of the monitoring parameter set has reached the monitoring effect threshold.

[0045] A communication device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0046] Based on the communication services of the target network, determine the monitoring requirements and monitoring effect thresholds of the target network;

[0047] Select monitoring parameter effect analysis indicators related to the monitoring requirements from the set of monitoring parameter indicators, assign weight coefficients to the monitoring parameter effect analysis indicators, and generate a monitoring parameter effect evaluation formula.

[0048] Using the network status data corresponding to the current monitoring parameter set and the indicator data related to the monitoring parameter set as inputs to the monitoring parameter effect evaluation formula, the score of the current monitoring parameter set is calculated to determine whether the monitoring effect of the monitoring parameter set has reached the monitoring effect threshold.

[0049] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0050] Based on the communication services of the target network, determine the monitoring requirements and monitoring effect thresholds of the target network;

[0051] Select monitoring parameter effect analysis indicators related to the monitoring requirements from the set of monitoring parameter indicators, assign weight coefficients to the monitoring parameter effect analysis indicators, and generate a monitoring parameter effect evaluation formula;

[0052] Using the network status data corresponding to the current monitoring parameter set and the indicator data related to the monitoring parameter set as inputs to the monitoring parameter effect evaluation formula, the score of the current monitoring parameter set is calculated to determine whether the monitoring effect of the monitoring parameter set has reached the monitoring effect threshold.

[0053] A computer program product includes a computer program, characterized in that, when executed by a processor, the computer program implements the monitoring parameter effect evaluation method provided in the embodiments of this application, the method being:

[0054] Based on the communication services of the target network, determine the monitoring requirements and monitoring effect thresholds of the target network;

[0055] Select monitoring parameter effect analysis indicators related to the monitoring requirements from the set of monitoring parameter indicators, assign weight coefficients to the monitoring parameter effect analysis indicators, and generate a monitoring parameter effect evaluation formula.

[0056] Using the network status data corresponding to the current monitoring parameter set and the indicator data related to the monitoring parameter set as inputs to the monitoring parameter effect evaluation formula, the score of the current monitoring parameter set is calculated to determine whether the monitoring effect of the monitoring parameter set has reached the monitoring effect threshold.

[0057] The aforementioned monitoring parameter effectiveness evaluation method, apparatus, communication equipment, storage medium, and computer program product determine the monitoring requirements and monitoring effectiveness thresholds of the target network based on the communication services of the target network; select monitoring parameter effectiveness analysis indicators related to the monitoring requirements from a set of monitoring parameter indicators, assign weight coefficients to these indicators, and generate a monitoring parameter effectiveness evaluation formula; use the network status data corresponding to the current monitoring parameter set and the indicator data related to the monitoring parameter set as inputs to the monitoring parameter effectiveness evaluation formula, calculate the score of the current monitoring parameter set, and determine whether the monitoring effectiveness of the monitoring parameter set reaches the monitoring effectiveness threshold. This method pre-constructs a set of monitoring parameter indicators and pre-trains the allocation of indicator weights. Then, given the determined monitoring requirements and monitoring effectiveness thresholds of the target network's communication services, it selects monitoring parameter effectiveness analysis indicators related to the monitoring requirements from the set of monitoring parameter indicators, assigns corresponding weight coefficients to these indicators, generates a monitoring parameter effectiveness evaluation formula, and uses this formula to evaluate the monitoring effectiveness of the current monitoring parameter set. This method is adapted to the current target network and objectively and comprehensively evaluates the monitoring effectiveness of the current monitoring parameter set. Attached Figure Description

[0058] Figure 1 This is a system architecture diagram of a method for evaluating the effectiveness of monitoring parameters in one embodiment;

[0059] Figure 2 This is an internal structure diagram of the monitoring parameter effect analysis module in one embodiment;

[0060] Figure 3 This is a flowchart illustrating a method for evaluating the effectiveness of monitoring parameters in one embodiment;

[0061] Figure 4 This is a flowchart illustrating the steps for determining the minimum set of monitoring parameters in one embodiment;

[0062] Figure 5 This is a flowchart illustrating the steps involved in generating a formula for evaluating the effectiveness of monitoring parameters for different communication services in one embodiment.

[0063] Figure 6 This is a schematic diagram of the internal structure of a set of monitoring parameter indicators in one embodiment;

[0064] Figure 7 This is a flowchart illustrating the indicator weight allocation method in one embodiment;

[0065] Figure 8 This is a flowchart illustrating the process of establishing an indicator weight allocation model in one embodiment;

[0066] Figure 9 This is a flowchart illustrating the steps of applying the monitoring parameter effectiveness evaluation method to a digital twin system in one embodiment.

[0067] Figure 10 This is a flowchart illustrating an example of the establishment process for a monitoring parameter effect analysis system in one embodiment;

[0068] Figure 11 This is a structural block diagram of a monitoring parameter effect evaluation device in one embodiment;

[0069] Figure 12 This is an internal structural diagram of a communication device in one embodiment. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0071] Figure 1 This application provides a method for evaluating the effectiveness of monitoring parameters, which is applied to a monitoring parameter effectiveness evaluation system. Figure 1 As shown, this monitoring parameter effectiveness evaluation system includes a monitoring parameter effectiveness analysis module, a monitoring parameter relationship pattern repository, a monitoring parameter relationship analysis module, a monitoring parameter selection and aggregation module, and a multimodal communication service network element and topology model. Among them, such as... Figure 2 As shown, the monitoring parameter effectiveness analysis module includes a monitoring parameter indicator set, an indicator weight allocation module, and a monitoring parameter effectiveness evaluation formula. The monitoring parameter indicator set provides the possible dimensions of the parameters used to evaluate the monitoring parameter set, reflecting its monitoring effectiveness. The indicator weight allocation module integrates an indicator weight allocation model, which is used to assign weight coefficients to the indicators used to evaluate the monitoring parameter effectiveness. The monitoring parameter effectiveness evaluation formula is used to assess the monitoring effectiveness of the monitoring parameter set.

[0072] Currently, to improve the efficiency of network monitoring and operation and maintenance, the amount of monitoring and data collection can be reduced by monitoring a minimum set of monitoring parameters and collecting a dataset of these minimum parameters. However, different categories of monitoring parameters are typically manually combined by administrators based on expert experience to evaluate specific communication services. In multimodal network scenarios, because each service has its unique characteristics, administrators must manually combine different parameters to monitor each service individually. Today, diverse communication services are emerging at an extremely rapid pace, and generating accurate parameter sets for each service in a short time far exceeds the capabilities of operators. Different communication services have different monitoring requirements, and these requirements may change dynamically. For example, during peak network periods, special periods, and emergencies, rapid response to network faults and anomalies is required, demanding the highest sensitivity and accuracy of the monitoring parameter set; routine operation and maintenance scenarios require lower costs and more effective monitoring; data collection and monitoring for data analysis or model training require more comprehensive, granular, and accurate parameters. Furthermore, different modalities and different services have different requirements. Therefore, the minimum set of monitoring parameters for a target network is not unique; rather, there is a corresponding minimum set of monitoring parameters for different communication services and their specific objectives and monitoring needs. In order to find the minimum set of monitoring parameters that meets the monitoring requirements of the target network, a method for evaluating the quality of monitoring parameters is needed.

[0073] In traditional technologies, the quality assessment of the minimum monitoring parameters affects the generation and selection of the minimum monitoring parameter set. Traditional assessment methods are usually simple weighted summations of one or more dimensions. The selection of assessment dimensions and parameters is based on expert and experience-based choices, which are highly subjective and limited for massive parameter sets. Therefore, an objective and adaptive parameter selection and weighting mechanism is needed to effectively and accurately assess the monitoring parameter set under different application scenarios.

[0074] Based on the aforementioned traditional techniques, this application provides a method for evaluating the effectiveness of monitoring parameters. By pre-constructing a set of monitoring parameter indicators and pre-training the allocation of indicator weights, and then, given the determined monitoring requirements and monitoring effect thresholds for the communication services of the target network, the method selects monitoring parameter effect analysis indicators related to the monitoring requirements from the set of monitoring parameter indicators, assigns corresponding weight coefficients to these indicators, generates a monitoring parameter effect evaluation formula, and uses this formula to evaluate the monitoring effect of the current monitoring parameter set. This method is adapted to the current target network and objectively and comprehensively evaluates the monitoring effect of the current monitoring parameter set.

[0075] It should be noted that the beneficial effects or technical problems solved by the embodiments of this application are not limited to this one, but may also be other implicit or related problems. For details, please refer to the description of the embodiments below.

[0076] Before introducing specific embodiments of the present invention, the technical terms involved in the present invention will be explained:

[0077] Minimum monitoring parameter set: The minimum set of monitoring parameters that can meet the monitoring needs of the current scenario;

[0078] Minimum Monitoring Parameter Set Generation Model (MMPSGM): This model uses the MPEAM, MPRAM, MPSAM, and MPRPR modules to compute the minimum monitoring parameter set;

[0079] The Monitoring Parameters Effect Analysis Module (MPEAM) is responsible for collecting network status data from the network, calculating the monitoring effect of the new set of monitoring parameters applied in the network, and determining whether to trigger the minimum monitoring parameter generation process based on the monitoring quality.

[0080] The Monitoring Parameters Relationship Analysis Module (MPRAM) is responsible for mining the inherent relationships among monitoring parameters, such as correlation, dependency, and association, generating monitoring parameter relationship patterns, and saving them to MPRPR.

[0081] The Monitoring Parameters Selection and Aggregation Module (MPSAM) is responsible for selecting and aggregating monitor parameters to form a minimal set based on the parameter relationship pattern provided by MPRPR.

[0082] The Monitoring Parameter Relationship Pattern Repository (MPRPR) is responsible for storing the monitoring parameter relationship patterns calculated by MPRAM and generating relationship graphs of the monitoring parameters. The parameters can be broadly categorized into three types: overall high-order common parameter patterns for multi-degree multimodal communication services, high-order common parameter patterns, and low-order single parameter patterns for dedicated services. These three types of parameter patterns can comprehensively cover almost all relationships in the monitoring data.

[0083] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0084] In one embodiment, such as Figure 3 As shown, a method for evaluating the effectiveness of monitoring parameters is provided. Taking the application of this method to computer equipment as an example, the method includes the following steps:

[0085] Step 302: Determine the monitoring requirements and monitoring effect thresholds of the target network based on the communication services of the target network.

[0086] In implementation, the computer equipment integrates a monitoring parameter effect evaluation system. The monitoring parameter effect analysis module (MPRAM) in the monitoring parameter effect evaluation system determines the monitoring requirements and monitoring effect thresholds of the target network based on the communication services of the target network provided by the multimodal communication service network element and topology model (NETMCS).

[0087] Step 304: Select monitoring parameter effect analysis indicators related to monitoring needs from the monitoring parameter indicator set, assign weight coefficients to the monitoring parameter effect analysis indicators, and generate monitoring parameter effect evaluation formulas.

[0088] In implementation, a set of monitoring parameter indicators and an indicator weight allocation model are pre-established in the monitoring parameter effect analysis module. Therefore, for the communication service of the current target network, the monitoring parameter effect analysis module selects monitoring parameter effect analysis indicators that meet the monitoring parameter requirements and objectives of the communication service from the pre-established set of monitoring parameter indicators, and assigns corresponding weight coefficients to the indicators through the pre-trained indicator weight allocation model, thereby generating a monitoring parameter effect evaluation formula for evaluating the monitoring parameter set.

[0089] Step 306: Using the network status data corresponding to the current monitoring parameter set and the indicator data related to the monitoring parameter set as inputs to the monitoring parameter effect evaluation formula, calculate the score of the current monitoring parameter set and determine whether the monitoring effect of the monitoring parameter set has reached the monitoring effect threshold.

[0090] In implementation, when the monitoring parameter effect analysis module evaluates the monitoring parameter set, it can select the monitoring parameter set based on the Network Elements and Topology Model of Multimodal Communication Services (NETMCS), using the monitoring parameter set and its corresponding network status data as input to the monitoring parameter effect evaluation formula. Then, the monitoring parameter effect analysis module calculates the comprehensive score of the current monitoring parameter set and, based on the relationship between this comprehensive score and a threshold, determines whether the monitoring effect of the monitoring parameter set reaches the monitoring effect threshold. If the monitoring effect of the monitoring parameter set reaches the monitoring effect threshold, it indicates that the monitoring parameter set has passed the preliminary evaluation. This monitoring parameter set is a necessary preparation for generating the minimum monitoring parameter set.

[0091] Optionally, the monitoring parameter set obtained from the initial evaluation can be passed to the minimum monitoring parameter generation model to further optimize the monitoring parameter set and ultimately determine the minimum monitoring parameter set for the target network.

[0092] In the above-mentioned monitoring parameter effect evaluation method, a set of monitoring parameter indicators and a pre-trained weight allocation of indicators are constructed in advance. Then, given the monitoring requirements and monitoring effect thresholds of the target network's communication services, monitoring parameter effect analysis indicators related to the monitoring requirements are selected from the set of monitoring parameter indicators, and corresponding weight coefficients are assigned to these indicators to generate a monitoring parameter effect evaluation formula. This formula is used to evaluate the monitoring effect of the current monitoring parameter set, adapting to the current target network and objectively and comprehensively evaluating the monitoring effect of the current monitoring parameter set.

[0093] In one embodiment, when evaluating a set of monitoring parameters, this application provides a method for establishing a set of monitoring parameters, which further includes:

[0094] Build a set of monitoring parameters and metrics.

[0095] The monitoring parameter indicator set includes: indicator data related to the monitoring parameter set, including monitoring complexity and monitoring coverage.

[0096] In implementation, the Monitoring Parameter Effectiveness Analysis Module (MPEAM) in the monitoring parameter effectiveness evaluation system utilizes the Analytic Hierarchy Process (AHP) to establish a set of monitoring parameter indicators. This constructed set of indicators contains multiple indicator layers, such as: target layer, criterion layer, and scheme layer. Furthermore, the set includes evaluation indicators of types such as monitoring complexity, monitoring coverage, and the gap in monitoring results between the previous monitoring parameter set and the current monitoring parameter set. Simultaneously, commonly used industry evaluation indicators and indicators mined using data mining methods can also be incorporated into the monitoring parameter indicator set.

[0097] In one embodiment, this application provides a method for establishing a set of monitoring parameter indicators. The set of monitoring parameter indicators obtained by this method can be further optimized. Therefore, the method further includes:

[0098] Key indicators are identified and extracted from historical data using data mining or machine learning methods, and then added to the set of monitoring parameter indicators.

[0099] In implementation, the monitoring parameter effect analysis module uses data mining or machine learning methods to identify and mine key indicators from historical data, and then adds the identified key indicators to the monitoring parameter indicator set, resulting in a supplemented monitoring parameter indicator set that enriches the types of indicators in the monitoring parameter indicator set.

[0100] Optionally, both the primary and secondary indicators in the monitoring parameter indicator set can be dynamically adjusted, and the calculation methods and formulas for the newly added secondary indicators are the results of fitting, which can be adjusted and corrected according to actual changes.

[0101] In this embodiment, a set of monitoring parameter indicators and an indicator weighting model are established in advance. This enables the evaluation of the monitoring parameter effect of the current communication service's monitoring parameter set selected from the set of monitoring parameter indicators during the monitoring parameter effect evaluation process, and the allocation of appropriate weight coefficients to each evaluation indicator. This achieves an objective and comprehensive evaluation of the current communication service's monitoring parameter set.

[0102] In one embodiment, the specific processing procedure of step 302 includes:

[0103] Based on the communication services described by the multimodal communication service network elements and the topology model, determine the current monitoring requirements and monitoring effect thresholds for the target network.

[0104] In one embodiment, the specific processing procedure of step 304 includes:

[0105] The decision analysis method is used to select monitoring parameter effect analysis indicators that meet the monitoring needs and objectives from the set of monitoring parameter indicators, and weight coefficients are assigned to the monitoring parameter effect analysis indicators.

[0106] In implementation, the Monitoring Parameter Effectiveness Analysis Module (MPEAM) in the Monitoring Parameter Effectiveness Evaluation System uses decision analysis to select monitoring parameter effectiveness analysis indicators that meet the monitoring needs and objectives from the set of monitoring parameter indicators, and assigns corresponding weight coefficients to the indicators.

[0107] In this embodiment, by using decision analysis to screen the indicators for analyzing the monitoring effect of monitoring parameters, the evaluation process can be made more scientific and systematic, ensuring the objectivity and reliability of the evaluation results. At the same time, reasonable indicator selection and weight allocation can also help to better evaluate the monitoring effect of the monitoring parameter set, ensuring the objectivity and accuracy of the monitoring effect evaluation.

[0108] In one embodiment, the specific processing steps in step 306, which use the network status data corresponding to the current monitoring parameter set and the indicator data related to the monitoring parameter set as inputs to the monitoring parameter effect evaluation formula, include:

[0109] The network status data corresponding to the current monitoring parameter set provided by the multimodal communication service network element and the topology model, as well as the indicator data related to the current monitoring parameter set, will be used as the input to the monitoring parameter effect evaluation formula.

[0110] In implementation, in the monitoring parameter effect evaluation system, the multimodal communication service network element and topology model selects a set of monitoring parameters from the monitoring parameter relationship pattern repository based on the current target network. This set of monitoring parameters is used to monitor the communication services of the current target network, and is also the set of monitoring parameters to be evaluated.

[0111] The multimodal communication service network element and topology model includes a monitoring parameter acquisition module, a monitoring parameter analysis module, and a twin system event simulation module. Specifically, the minimum monitoring parameter generation model selects a parameter set based on the monitoring requirements of the target network and passes it to the monitoring parameter acquisition module for collecting monitoring parameter data. That is, the monitoring parameter acquisition module mainly obtains the dataset of collected parameters and related data such as parameter acquisition time and cost; the monitoring parameter analysis module mainly obtains feedback on the application of the current parameter set to this scenario, such as fault response speed, fault location accuracy, and fault event judgment accuracy.

[0112] Then, the multimodal communication service network element and topology model provide the network status data corresponding to the selected monitoring parameter set to the monitoring parameter effect analysis module. The monitoring parameter effect analysis module (MPEAM) then uses the network status data provided by the multimodal communication service network element and topology model, as well as the indicator data related to the current monitoring parameter set, as input to the monitoring parameter effect evaluation formula.

[0113] In this embodiment, the network status data and indicator data related to the monitoring parameter set corresponding to the communication service of the target network are automatically selected by the multimodal communication service network element and topology model. These are used as inputs to the monitoring parameter effect evaluation formula, thereby enabling the monitoring parameter effect analysis module to support the evaluation of the monitoring parameter set of communication services of different modal target networks.

[0114] In one embodiment, such as Figure 4 As shown, the specific processing procedure in step 306 includes steps 401 to 402, wherein:

[0115] Step 401: Calculate the score of the current monitoring parameter set to obtain the evaluation result of the monitoring parameter set in this round.

[0116] In implementation, the monitoring parameter effectiveness analysis module selects monitoring parameter effectiveness analysis indicators from the monitoring parameter indicator set and assigns corresponding weight coefficients to generate a monitoring parameter effectiveness evaluation formula. This formula is then adapted to the evaluation scenario of the current communication service's monitoring parameter set. The comprehensive score of the current monitoring parameter set is calculated using this evaluation formula, yielding the evaluation result for this round of monitoring parameter set evaluation. A higher score indicates better quality of each monitoring parameter in the monitoring parameter set and better suitability for the current scenario.

[0117] Step 402: The evaluation results of the current monitoring parameter set and the monitoring effect threshold are passed to the minimum monitoring parameter generation model so that the minimum monitoring parameter generation model can optimize the selection of the monitoring parameter set and determine the minimum monitoring parameter set of the target network through iteration.

[0118] In implementation, the monitoring parameter effect analysis module passes the evaluation results and thresholds of the current monitoring parameter set to the minimum monitoring parameter generation model, so that the minimum monitoring parameter generation model can optimize the selection of the monitoring parameter set, adjust the monitoring parameter index set and index weight allocation model, and determine the minimum monitoring parameter set of the target network in the monitoring parameter set of each round of evaluation through iteration.

[0119] In this embodiment, the monitoring parameter effect evaluation formula adapted to the current target network's communication service is generated by the monitoring parameter effect analysis module. The comprehensive score of the monitoring parameter set in each round is calculated, which objectively and comprehensively evaluates the monitoring effect of the current monitoring parameter set. Based on the minimum monitoring parameter generation model, the monitoring parameter set in each round is optimized to determine the minimum monitoring parameter set of the target network, thereby improving the monitoring accuracy of the minimum monitoring parameter set.

[0120] In one embodiment, each method further includes:

[0121] The weight allocation for different communication services can be trained using machine learning methods to obtain a fitting scheme for the weight allocation of indicators that adapts to the monitoring needs of different communication services.

[0122] In one embodiment, when evaluating a set of monitoring parameters, this application selects suitable evaluation indicators from the set of monitoring parameter indicators and assigns weight coefficients to the evaluation indicators. Therefore, this application provides a method for establishing an indicator weighting model, such as... Figure 5 As shown, the method also includes:

[0123] Step 501: Based on the monitoring needs and objectives of communication services of different target networks, the indicators in the monitoring parameter indicator set are weighted using the indicator allocation weight model, and the consistency of the current weight allocation is checked to obtain the check results.

[0124] In implementation, the monitoring parameter effect analysis module assigns weights to the indicators in the monitoring parameter indicator set according to the monitoring needs and objectives of communication services of different target networks through the indicator allocation weight model, and performs a consistency check on the current weight allocation to determine whether the evaluation model corresponding to the current scenario meets expectations, thereby obtaining the check results.

[0125] Optionally, the indicator allocation weight model includes subjective weighting and objective weighting methods to allocate weights to the indicators in the monitoring parameter indicator set.

[0126] Step 502: Adjust the weight allocation based on the test results to generate evaluation formulas for the monitoring parameters corresponding to different communication services.

[0127] In implementation, the monitoring parameter effectiveness analysis module determines whether further adjustments to the weight allocation are needed based on the consistency check results of the weight allocation for each communication service within different communication services. Specifically, if the weight allocation scheme for the current communication service passes the consistency check, the module generates a monitoring parameter effectiveness evaluation formula for the current communication service based on the monitoring effectiveness evaluation indicators for the current network and the weight allocation schemes corresponding to each indicator. If the weight allocation scheme for the current communication service fails the consistency check, the monitoring parameter effectiveness analysis module readjusts the weight allocation based on the failed check result until the weight allocation scheme for the communication services of the current target network is obtained, generating the corresponding monitoring parameter effectiveness evaluation formula. Thus, the monitoring parameter effectiveness analysis module can generate monitoring parameter effectiveness evaluation formulas for various communication services.

[0128] In this embodiment, a set of monitoring parameter indicators and an indicator weighting model are established in advance. This enables the evaluation of the monitoring parameter effect of the current communication service's monitoring parameter set selected from the set of monitoring parameter indicators during the monitoring parameter effect evaluation process, and the allocation of appropriate weight coefficients to each evaluation indicator. This achieves an objective and comprehensive evaluation of the current communication service's monitoring parameter set.

[0129] In one embodiment, the monitoring parameter indicator set includes a target layer, a criterion layer, and a scheme layer. The specific process for constructing the monitoring parameter indicator set includes:

[0130] Using the hierarchical analysis of indicators, we establish the hierarchical relationship among the indicators in the monitoring parameter indicator set.

[0131] In this hierarchical relationship, the target layer is the score of the monitoring parameter set, the scheme layer is the monitoring parameter set, and the criterion layer contains multi-level indicators, including at least a first-level indicator and a second-level indicator corresponding to each first-level indicator.

[0132] In implementation, the monitoring parameter effect analysis module utilizes the hierarchical analysis method to establish hierarchical relationships among the indicators in the monitoring parameter indicator set. For example... Figure 6 As shown, the monitoring parameter index set includes a target layer, a criterion layer, and a scheme layer. The target layer is a comprehensive evaluation of the entire monitoring parameter set, and the resulting score is the overall score of the monitoring parameter set. The scheme layer contains the monitoring parameter set corresponding to the communication services of the multimodal target network, while the criterion layer contains multi-level evaluation indicators used to analyze and evaluate the monitoring effect of the monitoring parameter set. Figure 6Taking the criterion layer as an example, which includes two levels of indicators, these two levels are primary indicators and the corresponding secondary indicators (columns) for each primary indicator. Primary indicators are abstract and general, while secondary indicators are specific and detailed versions of the primary indicators. The calculation formulas for secondary indicators are obtained through definition or fitting and require standardization. For example, the primary indicators of the criterion layer include: cost, speed, complexity, accuracy, etc. Secondary indicators corresponding to cost (primary indicator) include: cost of acquisition equipment, resource usage cost, etc. Secondary indicators corresponding to speed (primary indicator) include: acquisition completion time, parameter analysis speed, etc. Secondary indicators corresponding to complexity (primary indicator) include: data coverage, data granularity, etc. Secondary indicators corresponding to accuracy (primary indicator) include: fault detection accuracy, root cause analysis accuracy, etc.

[0133] Specifically, after determining the hierarchical relationship of the indicators included in the monitoring parameter indicator set, a calculation formula for each secondary indicator is defined, ensuring that the output of each secondary indicator is standardized. Furthermore, the range of each secondary indicator is determined based on historical experience data. Since each primary indicator corresponds to multiple secondary indicators, the primary indicators are also standardized.

[0134] Optionally, this application embodiment does not limit the hierarchical structure of the monitoring parameter indicator set, nor does it limit the type and number of indicators contained in each hierarchical structure. The monitoring parameter indicator set can be adaptively adjusted based on the different communication service scenarios and monitoring needs to be evaluated.

[0135] In this embodiment, the hierarchical relationship of each indicator in the monitoring parameter indicator set is established by using the indicator hierarchy analysis method. Based on the hierarchical relationship between each indicator, the weight distribution relationship between each indicator is clarified. Thus, based on the determined indicators and their corresponding weight coefficients, a comprehensive score of the monitoring parameter set can be achieved.

[0136] In one embodiment, such as Figure 7 As shown, the specific processing steps of step 501 include:

[0137] Step 701: For the monitoring needs and objectives of different communication services, assign weight coefficients to the primary indicators using the subjective weighting method.

[0138] The scores for the monitoring parameter set are calculated by weighting the primary indicators, and the primary indicators are calculated by weighting the secondary indicators.

[0139] In implementation, the monitoring parameter effectiveness analysis module assigns weight coefficients to primary indicators using a subjective weighting method to meet the monitoring needs and objectives of different communication services. Specifically, different communication services and scenarios correspond to different objectives and needs. For example, in peak communication service scenarios, the objective is sensitivity to faults and anomalies; time, accuracy, and sensitivity are all important scoring indicators. Therefore, these primary indicators are given higher weights. In scenarios involving the routine operation and maintenance of communication services, monitoring requires more comprehensive, granular, and accurate parameter collection; therefore, the primary indicators reflecting the monitoring effectiveness are given higher weights.

[0140] In addition, the target layer included in the monitoring parameter indicator set is used to comprehensively evaluate the monitoring parameter set and obtain the score of the monitoring parameter set. Therefore, based on the hierarchical relationship of each indicator in the monitoring parameter indicator set, the score of the monitoring parameter set output by the target layer is obtained by weighted calculation of the first-level indicators of the criterion layer, and the first-level indicators are obtained by weighted calculation of the second-level indicators.

[0141] Step 702: Assign weight coefficients to the secondary indicators using the objective weighting method.

[0142] In implementation, the monitoring parameter effect analysis module uses an objective weighting method to assign weight coefficients to secondary indicators. Specifically, weights are assigned to indicators based on the monitoring needs of different target networks. Therefore, an objective weighting method is used to assign weights to indicators under different scenarios. Objective weighting methods include CRITIC (Criteria Importance Through Intercriteria Correlation), entropy weighting, and TOPSIS (Technique for Order Preference by Similarity to an Ideal Solution), and multiple methods can be used in combination, such as using entropy weighting and TOPSIS together.

[0143] This paper presents a process for assigning weight coefficients to the secondary indicators corresponding to a primary indicator using the entropy weighting method (specifically, an objective weighting method). The primary indicator is an accuracy indicator. The process for assigning weight coefficients to the secondary indicators is as follows:

[0144] The secondary indicators under this primary indicator include three sub-indicators: fault response result (speed), fault location result (accuracy, distance), and fault root cause analysis result (accuracy, distance). This involves dedimensionalizing each indicator. Assume there are m secondary indicators: Each secondary indicator has n data samples: Standardization of each secondary indicator yields Positive indicators: Negative indicators: Calculate the ratio: Calculate the information entropy of each secondary indicator: Weights are calculated using information entropy: This allows us to obtain the weights of the secondary indicators relative to the primary indicators. Similarly, the weights of the primary indicators can be adjusted using both objective and subjective weighting methods when calculating the overall score.

[0145] In this embodiment, by monitoring the communication service needs and purposes of different target networks, the indicators in the monitoring parameter indicator set are weighted so that the most suitable indicator can be selected under different communication services. This ensures full coverage of all communication services involved in the monitoring parameter effect analysis system, while also ensuring the objectivity and accuracy of the evaluation.

[0146] In one embodiment, such as Figure 8 As shown, prior to step 502, the method further includes:

[0147] Step 801: Evaluate each set of monitoring parameters based on the monitoring parameter effect evaluation formula to obtain the evaluation results.

[0148] In implementation, for different communication services, the monitoring parameter effect analysis module evaluates each monitoring parameter set based on the corresponding monitoring parameter effect evaluation formula, and obtains the evaluation results of the monitoring parameter sets under different communication services.

[0149] Step 802: Based on the evaluation results and the corresponding feedback information, assign weights to different communication services and train the model using machine learning to obtain an indicator weight allocation model that adapts to different communication services.

[0150] In implementation, the monitoring parameter effect analysis module uses machine learning to train the weight allocation corresponding to different communication services based on the evaluation results of each monitoring parameter set and the feedback information of each evaluation result. That is, it readjusts the monitoring parameter indicator set and indicator weight allocation until the obtained indicator weight allocation scheme meets the preset conditions, thereby obtaining an indicator weight allocation model adapted to different communication services.

[0151] Optionally, by assigning weights to evaluation metrics of discrete monitoring parameter sets corresponding to different communication services, and training the model using machine learning methods, discrete metric weight allocation models for different scenarios and corresponding weight allocations can be obtained. Once sufficient scenario and corresponding weight allocation data are obtained, this data can be used to train a fitting function for scenario objectives and weight allocation, resulting in a continuous metric weight allocation model for scenarios and weight allocations.

[0152] In this embodiment, by monitoring the communication service needs and purposes of different target networks, the indicators in the monitoring parameter indicator set are weighted so that the most suitable indicator can be selected under different communication services. This ensures full coverage of all communication services involved in the monitoring parameter effect analysis system, while also ensuring the objectivity and accuracy of the evaluation.

[0153] In one embodiment, the multimodal communication service network element and topology model includes a twin system event simulation module and a monitoring parameter analysis module, such as... Figure 9 As shown, an automated evaluation process for a monitoring parameter effectiveness evaluation method is applied to a digital twin system. Before step 502, the method further includes:

[0154] Step 901: The twin system event simulation module simulates and generates events according to the proposed scenario, and transmits the events to the network environment simulation system so that the network environment simulation system generates events.

[0155] In implementation, based on the proposed scenario as input, suitable indicators and weights are selected from the set of monitoring parameters to serve as the model for evaluating parameter quality. The monitoring parameter generation module selects the parameter set according to the scenario and passes it to the monitoring parameter acquisition module to collect monitoring parameter data. Through the twin system event simulation module, events are simulated and generated according to the proposed scenario, and the events are passed to the network environment simulation system to enable the network environment simulation system to generate events.

[0156] Step 902: The event information is transmitted to the monitoring parameter analysis module so that the monitoring parameter analysis module can analyze and compare the event.

[0157] In practice, the twin system event simulation module also transmits the event information of the generated events to the monitoring parameter analysis module, so that the monitoring parameter analysis module can analyze and compare the events.

[0158] In this embodiment, a monitoring parameter effectiveness evaluation method is presented for application in an automated evaluation process within a digital twin system, thereby enabling the application of the monitoring parameter effectiveness evaluation method.

[0159] In one embodiment, after step 402, the method further includes:

[0160] Based on the evaluation results provided by the monitoring parameter effect analysis module, the monitoring parameter relationship analysis module optimizes the relationship between monitoring parameters in the monitoring parameter set and passes it to the monitoring parameter relationship pattern repository.

[0161] Among them, the monitoring parameter relationship analysis module, the monitoring parameter relationship pattern repository, and the monitoring parameter selection and aggregation module optimize and produce the minimum monitoring parameter set for the current target network based on the evaluation results of the monitoring parameter set.

[0162] In implementation, the monitoring parameter effect analysis module transmits the evaluation results of the monitoring parameter set to the monitoring parameter relationship analysis module. Based on these evaluation results, the monitoring parameter relationship analysis module optimizes the monitoring parameter relationships and transmits them to the monitoring parameter relationship pattern repository. This repository stores the monitoring parameter relationship patterns calculated by MPRAM and generates a relationship graph of the monitoring parameters.

[0163] In this embodiment, the monitoring parameter relationship analysis module optimizes the relationship between monitoring parameters based on the evaluation results provided by the monitoring parameter effect analysis, thereby clarifying the correlation between each monitoring parameter and ensuring a comprehensive and accurate evaluation of the monitoring parameter set.

[0164] In one embodiment, such as Figure 10 As shown, an example of the establishment process for a monitoring parameter effect analysis system is given. The example process includes:

[0165] Step 1001: Collection and analysis of basic monitoring parameter data;

[0166] Step 1002: Construct a set of monitoring parameter indicators based on the hierarchical analysis of indicators;

[0167] Step 1003: Select the communication service of the target network and set the monitoring requirements and objectives for the communication service;

[0168] Step 1004: Adjust the weights of the primary indicators corresponding to the communication services of the target network using the subjective weighting method;

[0169] Step 1005: Adjust the weights of the secondary indicators corresponding to the communication services of the target network using the objective weighting method;

[0170] Step 1006: Score the monitoring effect of the monitoring parameter data and determine the consistency of weight allocation;

[0171] Step 1007: Determine whether the indicator weight allocation model is optimal; if yes, proceed to step 1003; if no, proceed to step 1004.

[0172] Step 1008: Record the weight allocation of communication services of different target networks in different scenarios;

[0173] Step 1009: Machine learning methods fit weight allocation functions for different scenarios;

[0174] Step 1010: Generate monitoring parameter effect analysis models adapted to different scenarios;

[0175] Step 1011: Determine whether the monitoring parameter effect analysis model needs to be updated; if it needs to be updated, proceed to step 1001.

[0176] It should be understood that, although Figures 3 to 5 , Figures 9 to 10 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figures 3 to 5 , Figures 9 to 10 At least some of the steps in the process may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.

[0177] In one embodiment, such as Figure 11 As shown, a monitoring parameter effect analysis device is provided, including: a determination module 1101, an indicator weight allocation module 1102, and an effect evaluation module 1103, wherein:

[0178] The determination module 1101 is used to determine the monitoring requirements and monitoring effect thresholds of the target network based on the communication services of the target network.

[0179] The indicator weight allocation module 1102 is used to select monitoring parameter effect analysis indicators related to monitoring needs from the monitoring parameter indicator set, assign weight coefficients to the monitoring parameter effect analysis indicators, and generate monitoring parameter effect evaluation formulas.

[0180] The effect evaluation module 1103 is used to calculate the score of the current monitoring parameter set by taking the network status data corresponding to the current monitoring parameter set and the indicator data related to the monitoring parameter set as inputs to the monitoring parameter effect evaluation formula, and to determine whether the monitoring effect of the monitoring parameter set has reached the monitoring effect threshold.

[0181] In one embodiment, the device further includes:

[0182] The building module is used to build a set of monitoring parameter metrics. The set of monitoring parameter metrics includes: metric data related to the monitoring parameter set; the related metric data includes monitoring complexity and monitoring coverage.

[0183] In one embodiment, the device further includes:

[0184] The supplementary module is used to identify and mine key indicators from historical data through data mining or machine learning methods, and to supplement the set of monitoring parameter indicators with these key indicators.

[0185] In one embodiment, the device further includes:

[0186] Based on the communication services described by the multimodal communication service network elements and the topology model, determine the current monitoring requirements and monitoring effect thresholds for the target network.

[0187] In one embodiment, the indicator weight allocation module 1102 is specifically used to select monitoring parameter effect analysis indicators that meet the monitoring needs and objectives from the monitoring parameter indicator set using decision analysis method, and to assign weight coefficients to the monitoring parameter effect analysis indicators.

[0188] In one embodiment, the effect evaluation module 1103 is specifically used to take the network status data corresponding to the current monitoring parameter set provided by the multimodal communication service network element and the topology model, as well as the indicator data related to the current monitoring parameter set, as input to the monitoring parameter effect evaluation formula.

[0189] In one embodiment, the effect evaluation module 1103 is specifically used to calculate the score of the current monitoring parameter set and obtain the evaluation result of the monitoring parameter set in this round.

[0190] The evaluation results of the current monitoring parameter set and the monitoring effect threshold are passed to the minimum monitoring parameter generation model so that the minimum monitoring parameter generation model can optimize the selection of the monitoring parameter set and determine the minimum monitoring parameter set of the target network through iteration.

[0191] In one embodiment, the device further includes:

[0192] The first training module is used to assign weights to different communication services. It can be trained using machine learning methods to obtain a fitting scheme for the weight allocation of indicators that adapts to the monitoring needs of different communication services.

[0193] In one embodiment, the device further includes:

[0194] The weight allocation verification module is used to allocate weights to the indicators in the monitoring parameter indicator set according to the monitoring needs and objectives of communication services of different target networks, and to perform consistency verification on the current weight allocation to obtain the verification results.

[0195] The generation module is used to adjust the weight allocation based on the test results and generate evaluation formulas for the monitoring parameters corresponding to different communication services.

[0196] In one embodiment, the construction module is specifically used to establish a hierarchical relationship between the indicators in the monitoring parameter indicator set using the indicator hierarchy analysis method; the target layer in the hierarchical relationship is the score of the monitoring parameter set, the scheme layer is the monitoring parameter set, and the criterion layer contains multi-level indicators, including at least a first-level indicator and a second-level indicator corresponding to each first-level indicator.

[0197] In one embodiment, the weight allocation verification module is specifically used to assign weight coefficients to primary indicators using a subjective weighting method for monitoring needs and objectives of different communication services; the score of the monitoring parameter set is obtained by weighted calculation of primary indicators, and the primary indicators are obtained by weighted calculation of secondary indicators.

[0198] The objective weighting method is used to assign weight coefficients to the secondary indicators.

[0199] In one embodiment, the device further includes:

[0200] The evaluation module is used to evaluate each set of monitoring parameters based on the evaluation formula for the effectiveness of monitoring parameters, and obtain the evaluation results.

[0201] The second training module is used to train a weight allocation model for different communication services based on the evaluation results and the corresponding feedback information, using machine learning methods.

[0202] In one embodiment, the multimodal communication service network element and topology model includes a twin system event simulation module and a monitoring parameter analysis module, and the device further includes:

[0203] The twin simulation module is used to simulate and generate events based on a given scenario through the twin system event simulation module, and then transmit the events to the network environment simulation system so that the network environment simulation system can generate events.

[0204] The transmission module is used to transmit event information to the monitoring parameter analysis module, so that the monitoring parameter analysis module can analyze and compare the event.

[0205] In one embodiment, the device further includes:

[0206] The optimization and delivery module is used to optimize the relationships between monitoring parameters in the monitoring parameter set based on the evaluation results provided by the monitoring parameter effect analysis module, and then deliver it to the monitoring parameter relationship pattern repository. The monitoring parameter relationship analysis module, the monitoring parameter relationship pattern repository, and the monitoring parameter selection and aggregation module optimize and produce the minimum monitoring parameter set for the current target network based on the evaluation results of the monitoring parameter set.

[0207] Specific limitations regarding the monitoring parameter effect analysis device can be found in the limitations of the monitoring parameter effect analysis method described above, and will not be repeated here. Each module in the aforementioned monitoring parameter effect analysis device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independently of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0208] In one embodiment, a communication device is provided, such as Figure 12 As shown, it includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0209] Based on the communication services of the target network, determine the monitoring requirements and monitoring effect thresholds of the target network;

[0210] Select monitoring parameter effect analysis indicators that are relevant to the monitoring needs from the set of monitoring parameter indicators, assign weight coefficients to the monitoring parameter effect analysis indicators, and generate monitoring parameter effect evaluation formulas.

[0211] Using the network status data and related indicator data of the current monitoring parameter set as inputs to the monitoring parameter effect evaluation formula, the score of the current monitoring parameter set is calculated to determine whether the monitoring effect of the monitoring parameter set has reached the monitoring effect threshold.

[0212] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0213] Construct a set of monitoring parameters and metrics; the set of monitoring parameters and metrics includes: metric data related to the monitoring parameter set; the related metric data includes monitoring complexity and monitoring coverage.

[0214] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0215] Key indicators are identified and extracted from historical data using data mining or machine learning methods, and then added to the set of monitoring parameter indicators.

[0216] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0217] Based on the communication services described by the multimodal communication service network elements and the topology model, determine the current monitoring requirements and monitoring effect thresholds for the target network.

[0218] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0219] The decision analysis method is used to select monitoring parameter effect analysis indicators that meet the monitoring needs and objectives from the set of monitoring parameter indicators, and weight coefficients are assigned to the monitoring parameter effect analysis indicators.

[0220] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0221] The network status data corresponding to the current monitoring parameter set provided by the multimodal communication service network element and the topology model, as well as the indicator data related to the current monitoring parameter set, will be used as inputs to the monitoring parameter effect evaluation formula.

[0222] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0223] Calculate the score of the current monitoring parameter set to obtain the evaluation result of this round of monitoring parameter set;

[0224] The evaluation results of the current monitoring parameter set and the monitoring effect threshold are passed to the minimum monitoring parameter generation model so that the minimum monitoring parameter generation model can optimize the selection of the monitoring parameter set and determine the minimum monitoring parameter set of the target network through iteration.

[0225] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0226] The weight allocation for different communication services can be trained using machine learning methods to obtain a fitting scheme for the weight allocation of indicators that adapts to the monitoring needs of different communication services.

[0227] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0228] Based on the monitoring needs and objectives of communication services in different target networks, the indicators in the monitoring parameter indicator set are weighted using an indicator allocation weight model. The consistency of the current weight allocation is then checked to obtain the check results.

[0229] Adjust the weight allocation based on the test results to generate evaluation formulas for the monitoring parameters corresponding to different communication services.

[0230] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0231] Using the hierarchical analysis of indicators, a hierarchical relationship is established among the indicators in the monitoring parameter indicator set. The target layer in the hierarchical relationship is the score of the monitoring parameter set, the scheme layer is the monitoring parameter set, and the criterion layer contains multi-level indicators, including at least the first-level indicators and the second-level indicators corresponding to each first-level indicator.

[0232] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0233] For monitoring needs and objectives of different communication services, a subjective weighting method is used to assign weight coefficients to the primary indicators; the scores of the monitoring parameter set are obtained by weighted calculation of the primary indicators, and the primary indicators are obtained by weighted calculation of the secondary indicators.

[0234] The objective weighting method is used to assign weight coefficients to the secondary indicators.

[0235] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0236] The evaluation results are obtained by evaluating each set of monitoring parameters based on the evaluation formula for the effectiveness of monitoring parameters.

[0237] Based on the evaluation results and the corresponding feedback information, the weight allocation for different communication services is trained using machine learning methods to obtain an indicator weight allocation model adapted to different communication services.

[0238] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0239] The twin system event simulation module simulates and generates events based on the proposed scenario, and transmits the events to the network environment simulation system so that the network environment simulation system can generate events.

[0240] The event information is transmitted to the monitoring parameter analysis module so that the monitoring parameter analysis module can analyze and compare the event.

[0241] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0242] Based on the evaluation results provided by the monitoring parameter effect analysis module, the monitoring parameter relationship analysis module optimizes the relationship between monitoring parameters in the monitoring parameter set and passes it to the monitoring parameter relationship pattern repository. The monitoring parameter relationship analysis module, the monitoring parameter relationship pattern repository, and the monitoring parameter selection and aggregation module optimize and produce the minimum monitoring parameter set for the current target network based on the evaluation results of the monitoring parameter set.

[0243] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0244] Based on the communication services of the target network, determine the monitoring requirements and monitoring effect thresholds of the target network;

[0245] Select monitoring parameter effect analysis indicators that are relevant to the monitoring needs from the set of monitoring parameter indicators, assign weight coefficients to the monitoring parameter effect analysis indicators, and generate monitoring parameter effect evaluation formulas.

[0246] Using the network status data corresponding to the current monitoring parameter set and the indicator data related to the monitoring parameter set as inputs to the monitoring parameter effect evaluation formula, the score of the current monitoring parameter set is calculated to determine whether the monitoring effect of the monitoring parameter set has reached the monitoring effect threshold.

[0247] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0248] Construct a set of monitoring parameters and metrics; the set of monitoring parameters and metrics includes: metric data related to the monitoring parameter set; the related metric data includes monitoring complexity and monitoring coverage.

[0249] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0250] Key indicators are identified and extracted from historical data using data mining or machine learning methods, and then added to the set of monitoring parameter indicators.

[0251] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0252] Based on the communication services described by the multimodal communication service network elements and the topology model, determine the current monitoring requirements and monitoring effect thresholds for the target network.

[0253] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0254] The decision analysis method is used to select monitoring parameter effect analysis indicators that meet the monitoring needs and objectives from the set of monitoring parameter indicators, and weight coefficients are assigned to the monitoring parameter effect analysis indicators.

[0255] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0256] The network status data corresponding to the current monitoring parameter set provided by the multimodal communication service network element and the topology model, as well as the indicator data related to the current monitoring parameter set, will be used as inputs to the monitoring parameter effect evaluation formula.

[0257] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0258] Calculate the score of the current monitoring parameter set to obtain the evaluation result of this round of monitoring parameter set;

[0259] The evaluation results of the current monitoring parameter set and the monitoring effect threshold are passed to the minimum monitoring parameter generation model so that the minimum monitoring parameter generation model can optimize the selection of the monitoring parameter set and determine the minimum monitoring parameter set of the target network through iteration.

[0260] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0261] The weight allocation for different communication services can be trained using machine learning methods to obtain a fitting scheme for the weight allocation of indicators that adapts to the monitoring needs of different communication services.

[0262] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0263] Based on the monitoring needs and objectives of communication services in different target networks, the indicators in the monitoring parameter indicator set are weighted using an indicator allocation weight model. The consistency of the current weight allocation is then checked to obtain the check results.

[0264] Adjust the weight allocation based on the test results to generate evaluation formulas for the monitoring parameters corresponding to different communication services.

[0265] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0266] Using the hierarchical analysis of indicators, a hierarchical relationship is established among the indicators in the monitoring parameter indicator set. The target layer in the hierarchical relationship is the score of the monitoring parameter set, the scheme layer is the monitoring parameter set, and the criterion layer contains multi-level indicators, including at least the first-level indicators and the second-level indicators corresponding to each first-level indicator.

[0267] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0268] For monitoring needs and objectives of different communication services, a subjective weighting method is used to assign weight coefficients to the primary indicators; the scores of the monitoring parameter set are obtained by weighted calculation of the primary indicators, and the primary indicators are obtained by weighted calculation of the secondary indicators.

[0269] The objective weighting method is used to assign weight coefficients to the secondary indicators.

[0270] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0271] The evaluation results are obtained by evaluating each set of monitoring parameters based on the evaluation formula for the effectiveness of monitoring parameters.

[0272] Based on the evaluation results and the corresponding feedback information, the weight allocation for different communication services is trained using machine learning methods to obtain an indicator weight allocation model adapted to different communication services.

[0273] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0274] The twin system event simulation module simulates and generates events based on the proposed scenario, and transmits the events to the network environment simulation system so that the network environment simulation system can generate events.

[0275] The event information is transmitted to the monitoring parameter analysis module so that the monitoring parameter analysis module can analyze and compare the event.

[0276] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0277] Based on the evaluation results provided by the monitoring parameter effect analysis module, the monitoring parameter relationship analysis module optimizes the relationship between monitoring parameters in the monitoring parameter set and passes it to the monitoring parameter relationship pattern repository. The monitoring parameter relationship analysis module, the monitoring parameter relationship pattern repository, and the monitoring parameter selection and aggregation module optimize and produce the minimum monitoring parameter set for the current target network based on the evaluation results of the monitoring parameter set.

[0278] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0279] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0280] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0281] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for evaluating the effectiveness of monitoring parameters, characterized in that, The method includes: Based on the communication services of the target network, determine the monitoring requirements and monitoring effect thresholds of the target network; Select monitoring parameter effect analysis indicators related to the monitoring requirements from the set of monitoring parameter indicators, assign weight coefficients to the monitoring parameter effect analysis indicators, and generate a monitoring parameter effect evaluation formula. Using the network status data corresponding to the current monitoring parameter set and the indicator data related to the monitoring parameter set as inputs to the monitoring parameter effect evaluation formula, the score of the current monitoring parameter set is calculated to determine whether the monitoring effect of the monitoring parameter set has reached the monitoring effect threshold.

2. The method according to claim 1, characterized in that, The method further includes: Construct the monitoring parameter indicator set; the monitoring parameter indicator set includes: indicator data related to the monitoring parameter set; the related indicator data includes monitoring complexity and monitoring coverage.

3. The method according to claim 1 or 2, characterized in that, The method further includes: Key indicators are identified and extracted from historical data using data mining or machine learning methods, and these key indicators are then added to the set of monitoring parameter indicators.

4. The method according to claim 1, characterized in that, The step of determining the monitoring requirements and monitoring effect thresholds of the target network based on the communication services of the target network includes: Based on the communication services described by the multimodal communication service network elements and the topology model, determine the current monitoring requirements and monitoring effect thresholds for the target network.

5. The method according to claim 1, characterized in that, The step of selecting monitoring parameter effect analysis indicators related to the monitoring requirements from the set of monitoring parameter indicators and assigning weight coefficients to the monitoring parameter effect analysis indicators includes: The decision analysis method is used to select monitoring parameter effect analysis indicators that meet the monitoring requirements from the set of monitoring parameter indicators, and weight coefficients are assigned to the monitoring parameter effect analysis indicators.

6. The method according to claim 1, characterized in that, The step of using network status data corresponding to the current monitoring parameter set and indicator data related to the monitoring parameter set as inputs to the monitoring parameter effect evaluation formula includes: The network status data corresponding to the current monitoring parameter set provided by the multimodal communication service network element and the topology model, as well as the indicator data related to the current monitoring parameter set, will be used as the input to the monitoring parameter effect evaluation formula.

7. The method according to claim 1, characterized in that, The step of calculating the score of the current monitoring parameter set and determining whether the monitoring effect of the monitoring parameter set reaches the monitoring effect threshold includes: Calculate the score of the current monitoring parameter set to obtain the evaluation result of this round of monitoring parameter set; The evaluation results of the current monitoring parameter set and the monitoring effect threshold are passed to the minimum monitoring parameter generation model so that the minimum monitoring parameter generation model can optimize the selection of the monitoring parameter set and determine the minimum monitoring parameter set of the target network through iteration.

8. The method according to claim 1, characterized in that, The method further includes: The weight allocation for different communication services can be trained using machine learning methods to obtain a fitting scheme for the weight allocation of indicators that adapts to the monitoring needs of different communication services.

9. The method according to claim 1 or 2, characterized in that, The method further includes: Based on the monitoring needs and objectives of communication services in different target networks, the indicators in the monitoring parameter indicator set are weighted using an indicator allocation weight model. The consistency of the current weight allocation is then checked to obtain the check results. The weight allocation is adjusted based on the test results to generate evaluation formulas for the monitoring parameters corresponding to different communication services.

10. The method according to claim 2, characterized in that, The monitoring parameter index set includes a target layer, a criterion layer, and a scheme layer. Constructing the monitoring parameter index set includes: Using the hierarchical analysis of indicators, a hierarchical relationship is established among the indicators in the monitoring parameter indicator set; the target layer in the hierarchical relationship is the score of the monitoring parameter set, the scheme layer is the monitoring parameter set, and the criterion layer contains multi-level indicators, including at least a primary indicator and a secondary indicator corresponding to each primary indicator.

11. The method according to claim 9, characterized in that, The step of assigning weights to the indicators in the monitoring parameter indicator set using an indicator allocation weight model includes: For monitoring needs and objectives of different communication services, a subjective weighting method is used to assign weight coefficients to the primary indicators; the score of the monitoring parameter set is obtained by weighted calculation of the primary indicators, and the primary indicators are obtained by weighted calculation of the secondary indicators; The objective weighting method is used to assign weight coefficients to the secondary indicators.

12. The method according to claim 9, characterized in that, After adjusting the weight allocation based on the test results and generating evaluation formulas for the monitoring parameters corresponding to different communication services, the method further includes: The monitoring parameter set is evaluated based on the aforementioned monitoring parameter effect evaluation formula to obtain the evaluation results; Based on the evaluation results and the corresponding feedback information, weight allocations are assigned to different communication services, and machine learning methods are used to train a weight allocation model that adapts to different communication services.

13. The method according to claim 1, characterized in that, The multimodal communication service network element and topology model includes a twin system event simulation module and a monitoring parameter analysis module. Before calculating the score of the current monitoring parameter set and determining whether the monitoring effect of the monitoring parameter set has reached the monitoring effect threshold, the method further includes: The twin system event simulation module simulates and generates events according to the proposed scenario, and transmits the events to the network environment simulation system so that the network environment simulation system can generate events. The event information is transmitted to the monitoring parameter analysis module so that the monitoring parameter analysis module can analyze and compare the event.

14. The method according to claim 7, characterized in that, The method further includes optimizing the selection of the monitoring parameter set by the minimum monitoring parameter generation model, and after iteratively determining the minimum monitoring parameter set of the target network, the method further includes: The monitoring parameter relationship analysis module optimizes the monitoring parameter relationships in the monitoring parameter set based on the evaluation results provided by the monitoring parameter effect analysis module, and then passes the optimization results to the monitoring parameter relationship pattern repository. The monitoring parameter relationship analysis module, the monitoring parameter relationship pattern repository, and the monitoring parameter selection and aggregation module optimize and produce the minimum monitoring parameter set for the current target network based on the evaluation results of the monitoring parameter set.

15. A monitoring parameter effect analysis device, characterized in that, The device includes: The determination module is used to determine the monitoring requirements and monitoring effect thresholds of the target network based on the communication services of the target network. The indicator weight allocation module is used to select monitoring parameter effect analysis indicators related to the monitoring requirements from the monitoring parameter indicator set, assign weight coefficients to the monitoring parameter effect analysis indicators, and generate monitoring parameter effect evaluation formulas. The effect evaluation module is used to calculate the score of the current monitoring parameter set by taking the network status data corresponding to the current monitoring parameter set and the indicator data related to the monitoring parameter set as inputs to the monitoring parameter effect evaluation formula, and to determine whether the monitoring effect of the monitoring parameter set has reached the monitoring effect threshold.

16. A communication device comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 14.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 14.

18. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 14.

Images