Network operation and maintenance method and device, storage medium and electronic equipment

Abstract

The invention provides a network operation and maintenance method and device, a storage medium and electronic equipment, and relates to the technical field of network intelligent operation management. The method comprises the following steps: determining a network operation and maintenance demand analysis result according to a network operation and maintenance target; the network operation and maintenance demand analysis result comprises a demand for artificial intelligence (AI) capability; the AI capability comprises an AI model and feature data; re-training the AI models of which the performance indexes do not meet a set threshold value in the original AI models, and determining the updated AI models; the retraining is that model parameters of the AI model are adjusted and then training is carried out by utilizing the characteristic data until the performance indexes of the AI model meet the set threshold value; determining a target AI model according to the network operation and maintenance demand analysis result and the updated AI model; and based on the feature data, executing the network operation and maintenance target by using the target AI model, and determining a network operation and maintenance result. According to the invention, unified management and application of the AI model are realized.

Description

Background Art

[0002] With the rapid development of intelligent network operation and maintenance, more and more AI (Artificial Intelligence) models are being applied to network operation and maintenance.

[0003] However, the predictive capabilities and performance indicators of each AI model differ, making it impossible to achieve unified management and shared use of AI models.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The present disclosure provides a network operation and maintenance method and device, a storage medium and an electronic device, which at least to a certain extent overcome the problem that the AI ​​model in the related technology cannot be uniformly managed and used.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0007] According to one aspect of the present disclosure, a network operation and maintenance method is provided, including:

[0008] Determine the network operation and maintenance demand analysis result according to the network operation and maintenance goal; the network operation and maintenance demand analysis result includes the demand for artificial intelligence (AI) capabilities; the AI ​​capabilities include: AI models and feature data;

[0009] Retraining the AI ​​model whose performance index in the original AI model does not meet the set threshold value to determine an updated AI model; the retraining is to adjust the model parameters of the AI ​​model and then use the feature data for training until the performance index of the AI ​​model meets the set threshold value;

[0010] Determine a target AI model according to the network operation and maintenance demand analysis result and the updated AI model;

[0011] Based on the feature data, the target AI model is used to execute the network operation and maintenance objectives and determine the network operation and maintenance results.

[0012] In some embodiments, determining the network operation and maintenance demand analysis result according to the network operation and maintenance goal includes:

[0013] According to the network operation and maintenance objectives, set the subject sample feature set and target set;

[0014] Determining a subject classification result according to the subject sample feature set and the target set;

[0015] Determining a semantic similarity calculation result according to the subject classification result;

[0016] A network operation and maintenance demand analysis result is determined according to the subject classification result and the semantic similarity calculation result.

[0017] In some embodiments, determining a subject classification result according to the subject sample feature set and the target set includes:

[0018] Vectorizing the subject sample feature set to determine a vectorized subject sample feature set; the vectorized subject sample feature set includes a plurality of subject sample feature vectors;

[0019] According to the target set, based on the subject classification algorithm, the maximum probability that the subject sample feature vector in the vectorized subject sample feature set belongs to the target set is predicted to determine the subject classification result.

[0020] In some embodiments, determining a semantic similarity calculation result according to the subject classification result includes:

[0021] Determining a first keyword of the subject classification result based on a keyword extraction algorithm;

[0022] Determine a second keyword of an AI capability management library; the AI ​​capability management library is used to manage existing AI capabilities; the AI ​​capabilities further include: tag data and application data;

[0023] A semantic similarity calculation result is determined according to the semantic similarity between the first keyword and the second keyword.

[0024] In some embodiments, determining a network operation and maintenance demand analysis result according to the subject classification result and the semantic similarity calculation result includes:

[0025] The maximum value between the subject classification result and the semantic similarity calculation result is determined as the network operation and maintenance demand analysis result.

[0026] In some embodiments, retraining the AI ​​model whose performance indicator in the original AI model does not meet the set threshold to determine the updated AI model includes:

[0027] Obtain performance parameters of the original AI model;

[0028] Determine the performance indicators of each AI model based on the performance parameters;

[0029] Determine the AI ​​model whose performance indicator does not meet the set threshold as a retraining AI model;

[0030] Retraining the retrained AI model using the feature data until a performance indicator of the retrained AI model reaches or exceeds the set threshold, and outputting the retrained AI model;

[0031] The original AI model and the retrained AI model are determined as updated AI models.

[0032] In some embodiments, determining a target AI model according to the network operation and maintenance demand analysis result and the updated AI model includes:

[0033] Using the network operation and maintenance demand analysis result as a matching index, matching it with the performance index corresponding to the updated AI model, and determining a first AI model set that meets the matching index;

[0034] The first AI model set is sorted according to other performance indicators except the matching indicator, and the optimal AI model in the sorting result is determined as the target AI model.

[0035] In some embodiments, based on the feature data, the target AI model is used to execute the network operation and maintenance goal, and the network operation and maintenance result is determined, including:

[0036] Sharing the target AI model and the feature data according to AI capability recommendation; the AI ​​capability recommendation includes at least one or more of the following combinations: standardized sharing, retraining sharing, inference call, model embedded reference, and image download;

[0037] Utilize the shared target AI model to execute the network operation and maintenance target and determine the network operation and maintenance result;

[0038] Among them, the standardized sharing is used to directly share the target AI model; the retraining sharing is used to adaptively retrain the target AI model according to usage requirements; the inference call is used to configure the target AI model in the cloud for model inference and call it through an API interface; the model embedded reference is used to encapsulate and package the target AI model and embed it into other systems different from the system where the AI ​​model is located; the image download is used to encapsulate the target AI model and the operating environment into other systems different from the system where the AI ​​model is located, and the other systems provide computing resources.

[0039] In some embodiments, after determining the network operation and maintenance demand analysis result, the method further includes:

[0040] The network operation and maintenance demand analysis results are sent to the user for verification.

[0041] In some embodiments, it also includes:

[0042] In response to the user's verification of the network operation and maintenance demand analysis result, submitting the network operation and maintenance demand analysis result for a subscription application;

[0043] In response to the subscription application being approved, the network operation and maintenance demand analysis result is downloaded to the local system in the form of a mirror or a model.

[0044] In some embodiments, the performance indicators include at least one or more combinations of the following: precision of the AI ​​model, recall rate of the AI ​​model, accuracy of the AI ​​model, error of the AI ​​model, and computational delay of the AI ​​model.

[0045] According to another aspect of the present disclosure, a network operation and maintenance device is provided, including:

[0046] A network operation and maintenance demand analysis result determination module is used to determine the network operation and maintenance demand analysis result according to the network operation and maintenance target;

[0047] An original AI model updating module, used to retrain an AI model whose performance index in the original AI model does not meet a set threshold, and determine an updated AI model; the retraining is to adjust the model parameters of the AI ​​model and then perform training using the feature data until the performance index of the AI ​​model meets the set threshold;

[0048] A target AI model determination module, used to determine a target AI model according to the network operation and maintenance demand analysis result and the updated AI model;

[0049] A network operation and maintenance result determination module is used to execute the network operation and maintenance target based on the feature data and use the target AI model to determine the network operation and maintenance result.

[0050] According to another aspect of the present disclosure, an electronic device is also provided, which includes: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned network operation and maintenance methods by executing the executable instructions.

[0051] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the network operation and maintenance method described in any one of the above is implemented.

[0052] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the network operation and maintenance method described above is implemented.

[0053] The network operation and maintenance method and device, storage medium and electronic device provided in the embodiments of the present disclosure first parse the network operation and maintenance target to obtain the network operation and maintenance demand analysis result, obtain the demand for AI capabilities, and then retrain the AI ​​model whose performance index does not meet the set threshold in the original AI model to update the original AI model, so that the performance index of each AI model in the updated AI model can reach the set threshold, which improves the application scope of the AI ​​model and facilitates the unified management and application of the AI ​​model; next, according to the network operation and maintenance demand analysis result and the updated AI model, the target AI model is determined, and finally based on the feature data, the target AI model is used to execute the network operation and maintenance target and determine the network operation and maintenance result. The present disclosure comprehensively utilizes the network operation and maintenance demand analysis result to screen the target AI model for the updated AI model, and can obtain the AI ​​model with the best performance index under the premise of meeting the network operation and maintenance requirements for AI performance indicators. The target AI model is used to execute the network operation and maintenance target, which can effectively improve the efficiency of network operation and maintenance engineers in network maintenance, improve network quality, meet user needs, and effectively improve user perception. At the same time, operators are one of the earliest companies to explore intelligent network operation and maintenance, with advantages in users, data, network technology and system platforms, which is conducive to the realization and application of technology. The present disclosure can be applied to a single system, and can interact with other internal and external systems through interfaces to provide efficient and high-quality AI models and feature data services, improve the efficiency of network operation and management, and enhance customer perception.

[0054] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0056] Figure 1 A schematic diagram showing the system architecture of a network operation and maintenance method in an embodiment of the present disclosure.

[0057] Figure 2 A schematic diagram of a network operation and maintenance method in an embodiment of the present disclosure is shown.

[0058] Figure 3 A schematic diagram of the process of determining a network operation and maintenance demand analysis result of a network operation and maintenance method in an embodiment of the present disclosure is shown.

[0059] Figure 4A schematic diagram of a process for determining subject classification results of a network operation and maintenance method in an embodiment of the present disclosure is shown.

[0060] Figure 5 A schematic diagram of a process for determining a semantic similarity calculation result of a network operation and maintenance method in an embodiment of the present disclosure is shown.

[0061] Figure 6 A demand analysis flow chart of a network operation and maintenance method in an embodiment of the present disclosure is shown.

[0062] Figure 7 A schematic diagram of the verification and audit process of a network operation and maintenance method in an embodiment of the present disclosure is shown.

[0063] Figure 8 A schematic diagram of an AI model retraining process of a network operation and maintenance method in an embodiment of the present disclosure is shown.

[0064] Fig. 9 A schematic diagram of the process of determining a target AI model of a network operation and maintenance method in an embodiment of the present disclosure is shown.

[0065] Fig.10 An interaction diagram of a network operation and maintenance method in an embodiment of the present disclosure is shown.

[0066] Fig.11 A flow chart of a network operation and maintenance method in an embodiment of the present disclosure is shown.

[0067] Fig.12 A schematic diagram of a network operation and maintenance device in an embodiment of the present disclosure is shown.

[0068] Fig.13 A structural block diagram of a computer device of a network operation and maintenance method in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0069] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0070] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0071] For ease of understanding, before introducing the embodiments of the present disclosure, several terms involved in the embodiments of the present disclosure are first explained as follows:

[0072] The specific implementation of the embodiment of the present disclosure is described in detail below with reference to the accompanying drawings.

[0073] Figure 1 FIG. 1 shows an exemplary application system architecture diagram to which a network operation and maintenance method according to an embodiment of the present disclosure can be applied. Figure 1 As shown, the system architecture may include a terminal device 101 , a network 102 and a server 103 .

[0074] The network 102 is a medium for providing a communication link between the terminal device 101 and the server 103, and can be a wired network or a wireless network.

[0075] Optionally, the wireless network or wired network described above uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a dedicated network or any combination of a virtual private network). In some embodiments, technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. are used to represent data exchanged through the network. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above data communication technologies.

[0076] The terminal device 101 can be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, wearable devices, augmented reality devices, virtual reality devices, etc.

[0077] Optionally, the client of the application installed in different terminal devices 101 is the same, or the client of the same type of application based on different operating systems. Based on the different terminal platforms, the specific form of the client of the application can also be different, for example, the application client can be a mobile client, a PC client, etc.

[0078] The server 103 may be a server that provides various services, such as a background management server that provides support for the device operated by the user using the terminal device 101. The background management server may analyze and process the received request and other data, and feed back the processing results to the terminal device.

[0079] Optionally, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0080] Those skilled in the art will know that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration, and any number of terminal devices, networks and servers may be provided according to actual needs, and the embodiments of the present disclosure do not limit this.

[0081] Under the above system architecture, a network operation and maintenance method is provided in an embodiment of the present disclosure, and the method can be executed by any electronic device with computing and processing capabilities.

[0082] In some embodiments, a network operation and maintenance method provided in an embodiment of the present disclosure may be executed by a terminal device of the above-mentioned system architecture; in other embodiments, a network operation and maintenance method provided in an embodiment of the present disclosure may be executed by a server in the above-mentioned system architecture; in other embodiments, a network operation and maintenance method provided in an embodiment of the present disclosure may be implemented by a terminal device and a server in the above-mentioned system architecture through interaction.

[0083] With the development of intelligent network operation and maintenance technology, a variety of platforms or systems for managing AI models have been developed. Such platforms or systems manage a variety of AI capability resources including AI models and feature label data. Due to the independence of platforms or systems, AI models and AI capability resources are repeatedly built and fragmented. A large number of AI models are scattered on a single platform or system due to their own different prediction management and performance indicators, and cannot be uniformly managed and called. Therefore, in order to facilitate the unified management of AI models, it is necessary to manage the AI ​​models scattered on a single platform or system, and realize the unified call management of AI models, so as to provide more efficient services and effectively improve the efficiency of network operation and management.

[0084] Figure 2 A flow chart of a network operation and maintenance method in an embodiment of the present disclosure is shown as follows: Figure 2 As shown, a network operation and maintenance method provided in an embodiment of the present disclosure includes the following steps:

[0085] S202: Determine a network operation and maintenance demand analysis result according to the network operation and maintenance target; the network operation and maintenance demand analysis result includes a demand for artificial intelligence (AI) capabilities; the AI ​​capabilities include: AI models and feature data;

[0086] S204: retraining the AI ​​model whose performance index in the original AI model does not meet the set threshold, and determining an updated AI model; the retraining is to adjust the model parameters of the AI ​​model and then perform training using the feature data until the performance index of the AI ​​model meets the set threshold;

[0087] S206: Determine a target AI model according to the network operation and maintenance demand analysis result and the updated AI model;

[0088] S208: Based on the feature data, use the target AI model to execute the network operation and maintenance goal and determine the network operation and maintenance result.

[0089] The disclosed embodiments implement demand analysis for network operation and maintenance by parsing network operation and maintenance goals, obtain demand for AI capabilities, and retrain the AI ​​models whose performance indicators do not meet the set thresholds in the original AI models to update the original AI models, so that the performance indicators of each AI model in the updated AI models can reach the set thresholds, thereby improving the application scope of the AI ​​models and facilitating the unified management and application of the AI ​​models. The network operation and maintenance demand analysis results are comprehensively utilized to screen the capabilities of the target AI models after the updated AI models, so that the AI ​​model with the best performance indicators can be obtained on the premise of meeting the network operation and maintenance requirements for AI performance indicators. Based on feature data, the target AI model is used to execute network operation and maintenance goals, which can effectively improve the efficiency of network operation and maintenance engineers in network maintenance, improve network quality, meet user needs, and effectively improve user perception.

[0090] Furthermore, when screening target AI models, AI models can be classified and managed according to specific network operation and maintenance scenarios, and based on the analysis of the needs of network operation and maintenance goals, accurate and effective recommendations of AI models can be achieved, realizing the accurate sharing of AI models and feature data in AI capabilities.

[0091] like Figure 3 As shown, in the embodiment, determining the network operation and maintenance demand analysis result according to the network operation and maintenance goal in the above step S202 may include:

[0092] S302: Setting a subject sample feature set and a target set according to a network operation and maintenance goal;

[0093] S304: Determine a subject classification result according to the subject sample feature set and the target set;

[0094] S306: Determine a semantic similarity calculation result according to the subject classification result;

[0095] S308: Determine a network operation and maintenance demand analysis result according to the subject classification result and the semantic similarity calculation result.

[0096] In the embodiment, the network operation and maintenance goals are generally formed in the form of text, for example, they can be recorded in the form of a manuscript. In order to accurately understand the network operation and maintenance goals and obtain accurate AI capability requirements, it is necessary to analyze the network operation and maintenance goals. First, the network operation and maintenance goals set the subject sample feature set X = {x1, x2, ... x n} and the target set Y = {y1,y2,…y m}, then the subject is classified according to the subject sample feature set and the target set to obtain the subject classification result, then the semantic similarity is calculated on the subject classification result to obtain the semantic similarity calculation result, and finally the network operation and maintenance demand analysis result is determined according to the subject classification result and the semantic similarity calculation result.

[0097] By classifying the subjects and calculating the semantic similarity of network operation and maintenance goals, accurate demand analysis can be performed. The network operation and maintenance demand analysis results obtained by combining the above two methods are more in line with the actual demand for AI capabilities in network operation and maintenance, and improve the accuracy of demand analysis.

[0098] like Figure 4 As shown, in the embodiment, determining the subject classification result according to the subject sample feature set and the target set in the above step S304 may include:

[0099] S402: vectorizing the subject sample feature set to determine a vectorized subject sample feature set; the vectorized subject sample feature set includes a plurality of subject sample feature vectors;

[0100] S404: According to the target set, based on the subject classification algorithm, predict the maximum probability that the subject sample feature vector in the vectorized subject sample feature set belongs to the target set, and determine the subject classification result.

[0101] When performing subject classification, we first need to transform the subject sample feature set X = {x1, x2, ... x n} vectorized, and obtained the vectorized subject sample feature set, that is, the subject sample features in the vectorized subject sample feature set are all subject sample feature vectors obtained after vectorization. Next, based on the subject classification algorithm, it is predicted that the subject sample feature vectors in the vectorized subject sample feature set belong to the target set Y = {y1, y2, ...y mThe subject classification algorithm mentioned above can adopt the Naive Bayes algorithm or other types of subject classification algorithms.

[0102] In the embodiment, the subject classification result is determined in the following manner:

[0103]

[0104]

[0105] Among them, max(P(y i |x)) is the subject classification result, which is used to indicate that the subject sample feature vector x in the vectorized subject sample feature set belongs to the target set Y = {y1, y2, ... m}; P(y i ) is the element y in the target set in the classification process using the classifier i The probability of P(x|y i ) is used to calculate the element y in the target set for the item to be classified. i , and element y i The probability that the subject sample feature vector x is included in is the joint probability; P(x) is the joint probability; P(x k |y i ) is used to calculate the element y in the target set for the item to be classified. i , and element y i Contains the subject sample feature vector x k The probability of; n is the number of subject sample feature vectors x in the vectorized subject sample feature set; k is the number of products.

[0106] When determining the subject classification result, if P(y i |x)=max{P(y1|x)×P(y2|x)…×P(y i |x)}, then the subject sample feature vector x belongs to the target set Y = {y1,y2,…y m}, each element y i , output P(y i |x), and obtain the subject classification result based on the Naive Bayes algorithm.

[0107] like Figure 5 As shown, in the embodiment, determining the semantic similarity calculation result according to the subject classification result in the above step S306 may include:

[0108] S502: Determine the first keyword of the subject classification result based on a keyword extraction algorithm;

[0109] S504: Determine a second keyword of an AI capability management library; the AI ​​capability management library is used to manage existing AI capabilities; the AI ​​capabilities further include: tag data and application data;

[0110] S506: Determine a semantic similarity calculation result according to the semantic similarity between the first keyword and the second keyword.

[0111] When performing similarity matching to calculate semantic similarity, it is first necessary to calculate the first keyword n1 of the subject classification result based on the keyword extraction algorithm. The above-mentioned keyword extraction algorithm can be the Term Frequency-Inverse Document Frequency (TF-IDF) algorithm, or other keyword extraction algorithms can be used.

[0112] Among them, regarding the TF-IDF algorithm, term frequency (TF) represents the frequency of occurrence of keywords (entries) in a text, and inverse document frequency (IDF) means that the IDF of a specific word can be obtained by dividing the total number of documents by the number of documents containing the word, and then taking the logarithm of the quotient.

[0113] Specifically, the first keyword may be calculated and extracted in the following manner:

[0114]

[0115]

[0116] TF-IDF=TF(t)×IDF(t)(5)

[0117] Among them, TF(t) is the word frequency; n t is the number of times the selected word appears in the file; n k is the number of times all words appear in the document; IDF(t) is the inverse document frequency; n n is the number of files containing the word; N is the total number of files; TF-IDF is the calculation result of the TF-IDF algorithm, that is, the extracted keywords.

[0118] After calculating the first keyword n1, the second keyword n2 is determined from the AI ​​capability management library, which includes various data such as original AI models, feature data, label data, and application data, and unified management of AI capabilities is achieved by using the AI ​​capability management library; the demand for AI capabilities in network operation and maintenance goals can be the demand for AI models. For example, when it is necessary to use AI models to predict link traffic, the AI ​​capability requirement is an AI model that can predict link traffic; the original AI models in the AI ​​capability management library include multiple AI models of various types, which can realize applications of different types and scenarios; for example, in a project, it is necessary to train an existing AI model in multiple scenarios, then AI can The power requirement is the need for feature data in multiple scenarios; the feature data in the AI ​​capability management library can be used to train AI models, and the feature data can be data from multiple types of scenarios; for another example, if an AI model lacks labeled label data during initial training, then the AI ​​capability requirement is label data; the label data in the AI ​​capability management library can be used for initial training of the AI ​​model, for example, the label data can be divided into training sets and verification sets for initial training of the AI ​​model; for another example, if an AI model lacks auxiliary running programs, then the AI ​​capability requirement is application data; the application data in the AI ​​capability management library can be an auxiliary program for the operation of the AI ​​model, for example, it can be the operating environment of the AI ​​model. In another embodiment, a keyword extraction algorithm can also be used to calculate the second keyword of the AI ​​capability management library.

[0119] Calculate the semantic similarity between the first keyword n1 and the second keyword n2 to obtain a semantic similarity calculation result. Specifically, the semantic similarity calculation result can be calculated in the following manner:

[0120]

[0121] Among them, Sim(n1, n2) is the result of semantic similarity calculation; n1 is the first keyword; n2 is the second keyword; is the semantic distance; α is an adjustment parameter, which in an example can represent the value of the semantic distance when the similarity is 0.5.

[0122] In an embodiment, determining the network operation and maintenance demand analysis result according to the subject classification result and the semantic similarity calculation result in the above step S308 may include:

[0123] The maximum value between the subject classification result and the semantic similarity calculation result is determined as the network operation and maintenance demand analysis result.

[0124] In the embodiment, the maximum value function is used to calculate the maximum value between the subject classification result and the semantic similarity calculation result, for example, {max(max(P(y i |x)), (Sim(n1, n2)))} calculated is determined as the network operation and maintenance demand analysis result.

[0125] like Figure 6 As shown, the network operation and maintenance target is analyzed for demand to obtain the network operation and maintenance demand analysis result, and the main process includes the following steps:

[0126] First, according to the network operation and maintenance goals, the subject sample feature set and the target set are set;

[0127] On the one hand, based on the naive Bayes algorithm, P(x|y i ), and then calculate max(P(y i |x)) to obtain the subject classification result;

[0128] On the other hand, based on the TF-IDF algorithm, the semantic similarity Sim(n1, n2) between the first keyword n1 and the second keyword n2 is calculated. In one way, the maximum value of Sim(n1, n2) is used as the result of semantic similarity calculation. In another way, the value of Sim(n1, n2) can be directly used as the result of semantic similarity calculation.

[0129] Finally, the maximum value of the subject classification result and the semantic similarity calculation result is comprehensively calculated to obtain the network operation and maintenance demand analysis result.

[0130] like Figure 7 As shown, in the embodiment, after the above step S202 determines the network operation and maintenance demand analysis result, it also includes:

[0131] S702: Send the network operation and maintenance demand analysis result to the user for verification;

[0132] S704: In response to the user verifying that the network operation and maintenance demand analysis result is passed, submitting the network operation and maintenance demand analysis result to a subscription application;

[0133] S706: In response to the subscription application being approved, the network operation and maintenance demand analysis result is downloaded to the local system in the form of a mirror or a model.

[0134] In order to ensure the accuracy of the network operation and maintenance demand analysis results obtained through demand analysis, the network operation and maintenance demand analysis results can be sent to the user for verification, and the correctness of the demand analysis can be further confirmed by manual verification by the user; if the network operation and maintenance demand analysis results meet the user's requirements, then the user verifies and approves the network operation and maintenance demand analysis results; this verification method can be displayed to the user in the display interface. After the user verifies and approves, the user clicks the pass button on the display interface, and in response to the user's verification and approval of the network operation and maintenance demand analysis results, the network operation and maintenance demand analysis results are submitted for a subscription application; the subscription application is used to subscribe to the network operation and maintenance target, so that there is no need to repeat the demand analysis for the same network operation and maintenance target. After receiving the subscription application, the administrator reviews it according to the user's authority or the calling authority of the AI ​​model. In response to the approval of the subscription application, the network operation and maintenance demand analysis results are downloaded to the local system in the form of a mirror or model, which can be directly used by the local system.

[0135] like Figure 8 As shown, in the embodiment, in the above S204, retraining the AI ​​model whose performance index in the original AI model does not meet the set threshold, and determining the updated AI model may include:

[0136] S802: Obtaining performance parameters of the original AI model;

[0137] S804: Determine a performance indicator of each AI model according to the performance parameter;

[0138] S806: Determine the AI ​​model whose performance indicator does not meet the set threshold as a retraining AI model;

[0139] S808: Retraining the retrained AI model using the feature data until the performance indicator of the retrained AI model reaches or exceeds the set threshold, and outputting the retrained AI model;

[0140] S8010: Determine the combination of the original AI model and the retrained AI model as an updated AI model.

[0141] In the embodiment, due to the different application scenarios, computing targets and computing objects of the model in the original AI model, the original AI model includes multiple AI models of multiple types. In order to achieve unified management and use of AI models, it is necessary to retrain the AI ​​models that do not meet the requirements to achieve standardized management of the original AI models. First, the performance parameters of the original AI model are obtained, and the performance indicators of each AI model are calculated. The performance indicators include at least one or more of the following combinations: the precision of the AI ​​model, the recall rate of the AI ​​model, the accuracy of the AI ​​model, the error of the AI ​​model, and the calculation delay of the AI ​​model. The above performance indicators can be used for value evaluation, for example, the effect of the AI ​​model can be evaluated to obtain value evaluation indicators.

[0142] Taking the error of the AI ​​model as an example, the error can be calculated based on algorithms such as the mean absolute error (MAE), mean square error (MSE), root mean square error (RMSE), and mean absolute percentage error (MAPE).

[0143] Taking the mean absolute percentage error method MAPE as an example, the error is calculated as follows:

[0144]

[0145] Wherein, MAPE is the mean absolute percentage error; Y i is the actual value, F i is the predicted value, and g is the number of predicted values.

[0146] Next, a threshold is set, and the AI ​​model whose performance index does not meet the set threshold is determined as a retrained AI model, which needs to be retrained; next, the retrained AI model is retrained using feature data until the performance index of the retrained AI model reaches or exceeds the set threshold, and the retrained AI model is output; in one way, a learning rate is set, and the retrained model is trained using feature data, and the parameters of the model are adjusted until the performance index of the retrained AI model reaches or exceeds the set threshold. At this time, the goal of retraining is achieved, and the retrained AI model is output. The original original AI model and the retrained AI model are combined to construct a new set as an updated AI model. In this way, the updated AI model obtained through the combined update process includes both the original AI model and the retrained AI model, which can better cover a variety of network operation and maintenance scenarios, improve the application scope of the AI ​​model, and facilitate the unified management and application of the AI ​​model. Another way is to eliminate the AI ​​models that need to be retrained from the original AI models, determine the remaining original AI models, combine the remaining original AI models with the retrained AI models, and build a new AI model as an updated AI model. In this way, the scale of the updated AI model will be reduced, reducing the difficulty of AI model management.

[0147] like Fig. 9 As shown, in the embodiment, determining the target AI model according to the network operation and maintenance demand analysis result and the updated AI model in the above step S206 may include:

[0148] S902: Using the network operation and maintenance demand analysis result as a matching index, matching it with the performance index corresponding to the updated AI model, and determining a first AI model set that meets the matching index;

[0149] S904: The first AI model set is sorted according to other performance indicators except the matching indicator, and the optimal AI model in the sorting result is determined as the target AI model.

[0150] In the embodiment, when the target AI model is screened, the network operation and maintenance demand analysis result is first used as a matching indicator to match the performance indicator corresponding to the updated AI model, and the first AI model set that meets the matching indicator is determined; for example, the network operation and maintenance goal is to predict the result of the IP new city L3 layer link traffic for the next 5 minutes, and the accuracy of the result prediction needs to be greater than 90%. Then the network operation and maintenance demand analysis result obtained after the demand analysis is the IP link traffic prediction with an accuracy of greater than 90%. The IP link traffic prediction with an accuracy of greater than 90% is used as a matching indicator to match the performance indicator corresponding to the updated AI model. The matching object is a model that can perform traffic prediction, and it is necessary to examine the AI ​​model. The accuracy of the model is measured, and the AI ​​models for traffic prediction with an accuracy greater than 90% are screened out to obtain a first AI model set that meets the matching index; next, the AI ​​models in the first AI model set are sorted according to other performance indicators except the accuracy. For example, the AI ​​models can be sorted according to the accuracy of the AI ​​model, the recall rate of the AI ​​model, the error of the AI ​​model, and the calculation delay of the AI ​​model. The top 10 in each of the above sortings are taken, and the AI ​​model that appears in the top 10 in each of the above sortings and is ranked highest is taken as the optimal AI model. The obtained optimal AI model has the highest accuracy, the lowest recall rate, the smallest error, and the lowest calculation delay. Therefore, the above optimal AI model can be determined as the target AI model.

[0151] In an embodiment, in the above step S208, based on the feature data, using the target AI model to execute the network operation and maintenance target and determine the network operation and maintenance result may include:

[0152] Sharing the target AI model and the feature data according to AI capability recommendation; the AI ​​capability recommendation includes at least one or more of the following combinations: standardized sharing, retraining sharing, inference calling, model embedded reference, and image download;

[0153] Utilize the shared target AI model to execute the network operation and maintenance target and determine the network operation and maintenance result;

[0154] Among them, the standardized sharing is used to directly share the target AI model; the retraining sharing is used to adaptively retrain the target AI model according to usage requirements; the inference call is used to configure the target AI model in the cloud for model inference, through the API interface call; the model embedded reference is used to package and pack the target AI model and embed it in other systems different from the system where the AI ​​model is located; the image download is used to package the target AI model and the operating environment into other systems different from the system where the AI ​​model is located, and the computing power resources are provided by the other systems. Further, the standardized sharing is used to directly share the feature data; the retraining sharing is used to modify the feature data to adapt to retraining according to usage requirements; the inference call is used to configure the feature data in the cloud for model inference, through the API interface call; the model embedded reference is used to package and pack the feature data and embed it in other systems different from the system where the AI ​​model is located; the image download is used to package the feature data and the operating environment into other systems different from the system where the AI ​​model is located, and the computing power resources are provided by the other systems.

[0155] In an embodiment, after obtaining the target AI model, the target AI model and feature data are shared according to the AI ​​capability recommendation, so that the use of the AI ​​model in different scenarios can be realized, and multi-scenario sharing of AI capabilities can be realized, which can adapt to a variety of network operation and maintenance scenarios. Next, the shared target AI model is used to execute the network operation and maintenance goals and determine the network operation and maintenance results. The results of the network operation and maintenance demand analysis can be comprehensively utilized to screen the capabilities of the target AI model of the updated AI model. On the premise of meeting the network operation and maintenance requirements for AI performance indicators, the AI ​​model with the best performance indicators can be obtained. The target AI model is used to execute the network operation and maintenance goals, which can effectively improve the efficiency of network operation and maintenance engineers in network maintenance, improve network quality, meet user needs, and effectively improve user perception.

[0156] Standardized sharing means that the target AI model can be directly shared without modification to execute network operation and maintenance goals; retraining sharing means that the target AI model needs to be reused for different regions, professions, and scenarios, and is to adaptively retrain the target AI model according to usage requirements. After retraining, it can adapt to different regions, professions, and scenarios to achieve precise adaptation of network operation and maintenance goals; reasoning call means configuring the AI ​​model in the cloud and providing an API interface (Application Programming Interface) to the outside world, which is used to configure the target AI model in the cloud for model reasoning and call the edge end through the API interface; model embedded reference means packaging the target AI model and embedding it in other systems different from the system where the AI ​​model is located, realizing the migration of AI models between different systems. In this sharing mode, the computing power resources of the system where the AI ​​model is located are still used; image download means packaging the target AI model and the operating environment into other systems different from the system where the AI ​​model is located, and computing power resources are provided by other systems. The above-mentioned multiple ways of AI capability recommendation can select one method according to specific needs, or combine multiple ways of AI capability recommendation according to specific needs. For example, if the requirement is to retrain according to the application scenario, choose retraining sharing. For example, if the requirement is to retrain in other systems according to the application scenario, you can choose a combination of retraining sharing and model embedded reference.

[0157] Operators are one of the earliest companies to explore intelligent network operation and maintenance. They have advantages in users, data, network technology and system platforms, which are conducive to the implementation and application of technology. The present disclosure can be applied to a single system, and can interact with other internal and external systems through interfaces to provide efficient and high-quality AI model services, improve the efficiency of network operation and management, and enhance customer perception.

[0158] like Fig.10 As shown, a network operation and maintenance method provided by the present disclosure can be independently run in a separate system, and by interacting with other systems through interfaces, it can provide efficient and high-quality intelligent services, improve the efficiency of telecommunications network operation and management, and enhance customer perception. Furthermore, a network operation and maintenance method of an embodiment of the present disclosure can be deployed on a cloud server, an edge server, an independent server, a server, or a client. The present disclosure can run on a variety of device terminals to achieve accurate recommendations of AI models.

[0159] The present disclosure also provides a specific example of a network operation and maintenance method, such as Fig.11As shown in the flowchart, the network operation and maintenance goal is input, the user application intention is analyzed based on the demand analysis algorithm, the network operation and maintenance demand analysis result is obtained, and the network operation and maintenance demand analysis result is fed back to the user for verification; if the network operation and maintenance goal is that the user wants to predict the IP New City L3 layer link traffic for the next 5 minutes, the network operation and maintenance demand analysis result is the AI ​​capability requirement for the IP link traffic prediction model, and the calculation delay is less than 30s; after the user verifies that the analyzed network operation and maintenance demand analysis result is correct, the user submits a subscription application to the AI ​​management end, and after the review is passed, the network operation and maintenance demand analysis result is downloaded to the local system in the form of a mirror or model; based on Evaluate the effect of AI models based on performance indicators; compare the performance indicators of AI models with the set threshold (such as the calculated accuracy of 90%), and retrain AI models below the threshold until they are greater than or equal to the threshold; (if the threshold is 95%, it is greater than the threshold, no retraining is required; if the threshold is 85%, it is less than the threshold, retraining is required); based on user intent, share the optimal (highest accuracy / lowest computing latency, etc.) target AI model that has passed the AI ​​model effect evaluation through standardized sharing / retraining sharing / inference call / model embedded reference / mirror download. (If the user's needs require the computing latency to be less than 30s, the AI ​​model with the highest accuracy, lowest recall, and smallest error under the 30s latency condition is recommended).

[0160] The present invention classifies and analyzes the needs of user input network operation and maintenance goals based on the Naive Bayes algorithm and the TF-IDF comprehensive algorithm, and recommends the optimal AI model based on the analyzed demand results and the intelligent recommendation algorithm based on the value assessment method. It can recommend the optimal AI capabilities based on the user's demand for AI capability calls, the highest accuracy / lowest error / minimum computing delay, etc. The user input demand classification and analysis method based on the Naive Bayes + TF-IDF algorithm can analyze the needs based on the optimal results of the two algorithms, which can better meet user needs and improve the accuracy of analysis. It can recommend the most effective AI capabilities on the premise of meeting user needs.

[0161] The present disclosure can add the policy decision verification capability of the cloud network operation system, improve the performance of the entire system, and ensure user perception. It can also help improve the intelligence level of the cloud network operation management system and improve the overall intelligence level.

[0162] It should be noted that the acquisition, storage, use, and processing of data in the technical solution disclosed in this disclosure are in compliance with the relevant provisions of national laws and regulations, and various types of data such as personal identity data, operation data, behavioral data, etc. related to individuals, customers, and groups obtained in the embodiments of this disclosure have all been authorized.

[0163] Based on the same inventive concept, the present disclosure also provides a network operation and maintenance device, as described in the following embodiments. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0164] Fig.12 A schematic diagram of a network operation and maintenance device in an embodiment of the present disclosure is shown. Fig.12 As shown, the device comprises:

[0165] The network operation and maintenance demand analysis result determination module 1201 is used to determine the network operation and maintenance demand analysis result according to the network operation and maintenance target; the network operation and maintenance demand analysis result includes the demand for artificial intelligence AI capabilities; the AI ​​capabilities include: AI model, feature data;

[0166] The original AI model updating module 1202 is used to retrain the AI ​​model whose performance index in the original AI model does not meet the set threshold, and determine an updated AI model; the retraining is to adjust the model parameters of the AI ​​model and then use the feature data to train until the performance index of the AI ​​model meets the set threshold;

[0167] A target AI model determination module 1203 is used to determine a target AI model according to the network operation and maintenance demand analysis result and the updated AI model;

[0168] The network operation and maintenance result determination module 1204 is used to execute the network operation and maintenance target based on the feature data and use the target AI model to determine the network operation and maintenance result.

[0169] It should be noted here that the above-mentioned network operation and maintenance demand analysis result determination module 1201, the original AI model update module 1202, the target AI model determination module 1203 and the network operation and maintenance result determination module 1204 correspond to S202 to S208 in the method embodiment, and the above-mentioned modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned method embodiment. It should be noted that the above-mentioned modules as part of the device can be executed in a computer system such as a set of computer executable instructions.

[0170] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0171] Refer to the following Fig.13 1300 according to this embodiment of the present disclosure is described. Fig.13 The electronic device 1300 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0172] like Fig.13 As shown, the electronic device 1300 is in the form of a general computing device. The components of the electronic device 1300 may include but are not limited to: at least one processing unit 1310, at least one storage unit 1320, and a bus 1330 connecting different system components (including the storage unit 1320 and the processing unit 1310).

[0173] Among them, the storage unit stores a program code, and the program code can be executed by the processing unit 1310, so that the processing unit 1310 executes the steps described in the above "Exemplary Method" section of this specification according to various exemplary embodiments of the present disclosure. For example, the processing unit 1310 can execute the following steps of the above method embodiment: according to the network operation and maintenance goal, determine the network operation and maintenance demand analysis result; the network operation and maintenance demand analysis result includes the demand for artificial intelligence AI capabilities; the AI ​​capabilities include: AI model, feature data; retrain the AI ​​model whose performance indicators in the original AI model do not meet the set threshold, and determine the updated AI model; the retraining is to adjust the model parameters of the AI ​​model and then use the feature data for training until the performance indicators of the AI ​​model meet the set threshold; determine the target AI model according to the network operation and maintenance demand analysis result and the updated AI model; based on the feature data, use the target AI model to execute the network operation and maintenance goal and determine the network operation and maintenance result.

[0174] The storage unit 1320 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 13201 and / or a cache storage unit 13202 , and may further include a read-only storage unit (ROM) 13203 .

[0175] The storage unit 1320 may also include a program / utility 13204 having a set (at least one) of program modules 13205, such program modules 13205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0176] Bus 1330 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0177] The electronic device 1300 may also communicate with one or more external devices 1340 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1300, and / or communicate with any device that enables the electronic device 1300 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1350. Furthermore, the electronic device 1300 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 1360. As shown, the network adapter 1360 communicates with other modules of the electronic device 1300 via a bus 1330. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1300, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0178] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0179] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart may be implemented as a computer program product, which includes: a computer program, which implements the above network operation and maintenance method when executed by a processor.

[0180] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. A program product capable of implementing the above method of the present disclosure is stored thereon. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to cause the terminal device to execute the steps according to various exemplary implementations of the present disclosure described in the above “Exemplary Method” section of this specification.

[0181] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0182] In the present disclosure, a computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0183] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0184] In a specific implementation, the program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0185] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0186] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0187] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0188] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A network operation and maintenance method, characterized in that: include: Determine the results of network operation and maintenance requirements analysis based on network operation and maintenance objectives; The network operation and maintenance demand analysis results include the demand for artificial intelligence (AI) capabilities; The AI ​​capabilities include: AI models and feature data; Retraining the AI ​​model whose performance index in the original AI model does not meet the set threshold value to determine an updated AI model; the retraining is to adjust the model parameters of the AI ​​model and then use the feature data for training until the performance index of the AI ​​model meets the set threshold value; Determine a target AI model according to the network operation and maintenance demand analysis result and the updated AI model; Based on the feature data, the target AI model is used to execute the network operation and maintenance objectives and determine the network operation and maintenance results.

2. The network operation and maintenance method according to claim 1, characterized in that: According to the network operation and maintenance objectives, determine the results of the network operation and maintenance requirements analysis, including: According to the network operation and maintenance objectives, set the subject sample feature set and target set; Determining a subject classification result according to the subject sample feature set and the target set; Determining a semantic similarity calculation result according to the subject classification result; A network operation and maintenance demand analysis result is determined according to the subject classification result and the semantic similarity calculation result.

3. The network operation and maintenance method according to claim 2, characterized in that: Determining a subject classification result according to the subject sample feature set and the target set includes: Vectorizing the subject sample feature set to determine a vectorized subject sample feature set; the vectorized subject sample feature set includes a plurality of subject sample feature vectors; According to the target set, based on the subject classification algorithm, the maximum probability that the subject sample feature vector in the vectorized subject sample feature set belongs to the target set is predicted to determine the subject classification result.

4. The network operation and maintenance method according to claim 2, characterized in that: Determining a semantic similarity calculation result according to the subject classification result includes: Determining a first keyword of the subject classification result based on a keyword extraction algorithm; Determine a second keyword of an AI capability management library; the AI ​​capability management library is used to manage existing AI capabilities; the AI ​​capabilities further include: tag data and application data; A semantic similarity calculation result is determined according to the semantic similarity between the first keyword and the second keyword.

5. The network operation and maintenance method according to claim 2, characterized in that: Determining a network operation and maintenance demand analysis result according to the subject classification result and the semantic similarity calculation result, including: The maximum value between the subject classification result and the semantic similarity calculation result is determined as the network operation and maintenance demand analysis result.

6. The network operation and maintenance method according to claim 1, characterized in that: Retrain the AI ​​models whose performance indicators in the original AI models do not meet the set thresholds, and determine the updated AI models, including: Obtain performance parameters of the original AI model; Determine the performance indicators of each AI model based on the performance parameters; Determine the AI ​​model whose performance indicator does not meet the set threshold as a retraining AI model; Retraining the retrained AI model using the feature data until a performance indicator of the retrained AI model reaches or exceeds the set threshold, and outputting the retrained AI model; The original AI model and the retrained AI model are combined to determine an updated AI model.

7. The network operation and maintenance method according to claim 1, characterized in that: Determining a target AI model according to the network operation and maintenance demand analysis result and the updated AI model includes: Using the network operation and maintenance demand analysis result as a matching index, matching it with the performance index corresponding to the updated AI model, and determining a first AI model set that meets the matching index; The first AI model set is sorted according to other performance indicators except the matching indicator, and the optimal AI model in the sorting result is determined as the target AI model.

8. The network operation and maintenance method according to claim 1, characterized in that: Based on the feature data, the target AI model is used to execute the network operation and maintenance target, and a network operation and maintenance result is determined, including: Sharing the target AI model and the feature data according to AI capability recommendation; the AI ​​capability recommendation includes at least one or more of the following combinations: standardized sharing, retraining sharing, inference calling, model embedded reference, and image download; Utilize the shared target AI model to execute the network operation and maintenance target and determine the network operation and maintenance result; Among them, the standardized sharing is used to directly share the target AI model; the retraining sharing is used to adaptively retrain the target AI model according to usage requirements; the inference call is used to configure the target AI model in the cloud for model inference and call it through an API interface; the model embedded reference is used to encapsulate and package the target AI model and embed it into other systems different from the system where the AI ​​model is located; the image download is used to encapsulate the target AI model and the operating environment into other systems different from the system where the AI ​​model is located, and the other systems provide computing resources.

9. The network operation and maintenance method according to claim 1, characterized in that: After determining the results of the network operation and maintenance requirements analysis, it also includes: The network operation and maintenance demand analysis results are sent to the user for verification.

10. The network operation and maintenance method according to claim 9, characterized in that: Also includes: In response to the user's verification of the network operation and maintenance demand analysis result, submitting the network operation and maintenance demand analysis result for a subscription application; In response to the subscription application being approved, the network operation and maintenance demand analysis result is downloaded to the local system in the form of a mirror or a model.

11. The network operation and maintenance method according to any one of claims 1 to 10, characterized in that: The performance indicators include at least one or more combinations of the following: precision of the AI ​​model, recall rate of the AI ​​model, accuracy of the AI ​​model, error of the AI ​​model, and calculation delay of the AI ​​model.

12. A network operation and maintenance device, characterized in that: include: A network operation and maintenance demand analysis result determination module is used to determine the network operation and maintenance demand analysis result according to the network operation and maintenance target; The network operation and maintenance demand analysis result includes the demand for artificial intelligence (AI) capabilities; the AI ​​capabilities include: AI models and feature data; An original AI model updating module, used to retrain an AI model whose performance index in the original AI model does not meet a set threshold, and determine an updated AI model; the retraining is to adjust the model parameters of the AI ​​model and then perform training using the feature data until the performance index of the AI ​​model meets the set threshold; A target AI model determination module, used to determine a target AI model according to the network operation and maintenance demand analysis result and the updated AI model; A network operation and maintenance result determination module is used to execute the network operation and maintenance target based on the feature data and use the target AI model to determine the network operation and maintenance result.

13. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the network operation and maintenance method according to any one of claims 1 to 11 by executing the executable instructions.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the network operation and maintenance method according to any one of claims 1 to 11 is implemented.

Images