Network operation and maintenance method and device

Abstract

The invention discloses a network operation and maintenance method and device and belongs to the field of network operation and maintenance. In the application, after an operation and maintenance request including first content expressed in a non-text form is acquired, at least one operation and maintenance task is acquired by adopting a multi-modal large language model according to the first content, and a request result matched with the operation and maintenance request is acquired according to the at least one operation and maintenance task. According to the method, the operation and maintenance request is processed by adopting the multi-modal large language model, and the multi-modal large language model can understand and process contents expressed in a non-text form, so that the flexibility of network operation and maintenance is relatively high.

Description

Technical Field

[0001] This application relates to the field of network operation and maintenance, and in particular to a network operation and maintenance method and apparatus. Background Technology

[0002] In today's information society, the efficiency and user experience of network operations and maintenance (O&M) are crucial for the operations of various enterprises and organizations. Large language models (LLMs) have already played a vital role in the field of network O&M. LLM-based network O&M products can understand user-inputted O&M questions and generate answers that match those questions, assisting O&M personnel in performing network O&M and improving efficiency. These LLM-based products can also be generative artificial intelligence (GenAI) products. However, currently, LLM applications in network O&M can only understand and process textual information (such as O&M questions expressed in text form), resulting in limited flexibility in LLM-based network O&M. Summary of the Invention

[0003] This application provides a network operation and maintenance method and apparatus. The technical solution of this application is as follows.

[0004] Firstly, a network operation and maintenance method is provided. This method includes: obtaining an operation and maintenance request, the request including first content, which is expressed in a non-textual form; obtaining at least one operation and maintenance task based on the first content using a multimodal large language model; and obtaining a request result matching the operation and maintenance request based on the at least one operation and maintenance task. The operation and maintenance request can be an operation and maintenance problem or a configuration request related to network operation and maintenance. The content expressed in a non-textual form is also called non-textual content or non-textual information.

[0005] The multimodal large language model used in this application can understand and process the non-textual content (i.e., non-textual content) included in operation and maintenance requests. Therefore, using this multimodal large language model to process operation and maintenance requests offers high flexibility and helps improve the flexibility of network operation and maintenance.

[0006] Optionally, the non-textual form includes at least one of image or audio forms. For example, the first content is content expressed in image form; or, the first content is content expressed in audio form; or, the first content includes content expressed in image form and content expressed in audio form. Content expressed in image form can be a single image or a video segment (also called a video clip) comprising multiple frames.

[0007] The multimodal large language model employed in this application can understand and process the non-textual content included in maintenance requests, which includes at least one of image or audio formats. Therefore, this multimodal large language model can understand and process the image-based and / or audio-based content included in maintenance requests. In other words, this multimodal large language model can understand and process images and / or videos and / or audio included in maintenance requests, thus offering high flexibility in processing maintenance requests.

[0008] Optionally, obtaining the operation and maintenance request includes: receiving first content input by the user in the network operation and maintenance interface; or, obtaining the first content based on a region selection instruction triggered by the user in the network operation and maintenance interface, wherein the first content is an image of the target region selected by the region selection instruction. That is, this application supports user input of content expressed in non-text form (i.e., non-text content, such as images and / or videos and / or audio), and also supports obtaining content expressed in image form (i.e., non-text content) based on the target region selected by the user.

[0009] Optionally, the multimodal large language model is used to: encode the first content to obtain a non-text encoded vector, convert the non-text encoded vector into a semantic vector, and obtain the at least one operation and maintenance task based on the semantic vector. Here, both the non-text encoded vector and the semantic vector are vectors of the first content, both are used to represent the first content, and the semantic vector represents the semantics expressed by the first content.

[0010] Optionally, the multimodal large language model includes a multimodal encoder, a modality converter, and a language sub-model. The multimodal encoder encodes first content to obtain a non-text encoded vector. The modality converter converts the non-text encoded vector into a semantic vector. The language sub-model obtains the at least one operational task based on the semantic vector. The method further includes: acquiring training data, which is data related to network operation and maintenance, and is expressed in non-textual form; and training the modality converter in the multimodal large language model based on the training data. For example, the training data is input into the multimodal large language model, the content output by the multimodal large language model based on the training data is obtained, and the parameters of the modality converter in the multimodal large language model are adjusted based on the content output by the multimodal large language model based on the training data to train the modality converter. Specifically, the multimodal encoder and the language sub-model in the multimodal large language model are frozen, and the parameters of the modality converter in the multimodal large language model are adjusted based on the content output by the multimodal large language model based on the training data. Data expressed in a non-textual form is also called non-textual data, which includes images, videos, audio, etc.

[0011] The technical solution provided in this application trains a modality converter in a multimodal large language model based on training data expressed in non-textual form. This can improve the accuracy of the modality converter in converting non-textual encoded vectors into semantic vectors, thereby improving the accuracy of the content output by the multimodal large language model (e.g., operation and maintenance tasks).

[0012] Optionally, the at least one operation and maintenance task includes a query task. Obtaining a request result matching the operation and maintenance request based on the at least one operation and maintenance task includes: encoding the first content according to the query task to obtain a non-text encoded vector; converting the non-text encoded vector into a semantic vector; obtaining a knowledge vector matching the semantic vector from an operation and maintenance knowledge base; and determining the knowledge corresponding to the knowledge vector as the request result matching the operation and maintenance request. Here, both the non-text encoded vector and the semantic vector are vectors of the first content, both used to represent the first content, and the semantic vector's semantics are the semantics expressed by the first content. The operation and maintenance knowledge base includes knowledge vectors matching the semantic vector, where each knowledge vector is a semantic vector of the knowledge corresponding to the knowledge vector, and the knowledge vector can express the semantics of the knowledge corresponding to the knowledge vector.

[0013] The technical solution provided in this application stores knowledge vectors in an operations and maintenance knowledge base. This allows the retrieval of a request result matching the operations and maintenance request from the knowledge base based on the semantic vector of the operations and maintenance request (e.g., the semantic vector of the content expressed in non-textual form included in the operations and maintenance request). The solution only requires matching the semantic vector of the operations and maintenance request with the knowledge vectors in the knowledge base to determine the knowledge vector matching the semantic vector of the operations and maintenance request. The knowledge corresponding to the knowledge vector matching the semantic vector of the operations and maintenance request is then identified as the request result matching the operations and maintenance request. This improves the efficiency and accuracy of retrieving the request result matching the operations and maintenance request.

[0014] Optionally, the operation and maintenance request also includes second content, which is content expressed in text form. Based on the first content, at least one operation and maintenance task is obtained using a multimodal large language model, including: obtaining the at least one operation and maintenance task using a multimodal large language model based on the first and second content. The content expressed in text form is also called text content or text information.

[0015] The multimodal large language model employed in this application can understand and process both the non-textual content (i.e., non-textual content) and the textual content (i.e., textual content) included in the operation and maintenance request. In other words, this multimodal large language model can understand and process operation and maintenance requests that include both non-textual and textual content. Therefore, using this multimodal large language model to process operation and maintenance requests offers high flexibility, contributing to improved network operation and maintenance flexibility. Since the operation and maintenance request includes both non-textual and textual content—that is, it includes at least two modalities of content—the content of the operation and maintenance request can be called multimodal content or multimodal information, and the operation and maintenance request can be called a multimodal operation and maintenance request. In other words, the multimodal large language model employed in this application can understand and process multimodal operation and maintenance requests, thus offering high flexibility.

[0016] Optionally, obtaining an operation and maintenance request includes: receiving first content and second content input by the user in the network operation and maintenance interface; or, receiving the second content input by the user in the network operation and maintenance interface, and obtaining the first content based on the preceding content of the second content, wherein the preceding content of the second content is the content of the user's interaction with the network operation and maintenance interface before the user inputs the second content, and the preceding content of the second content includes the first content; or, obtaining the first content based on a region selection instruction triggered by the user in the network operation and maintenance interface, and receiving the second content input by the user in the network operation and maintenance interface, wherein the first content is an image of the target region selected by the region selection instruction. That is, this application supports user input of operation and maintenance requests including non-text content and text content, supports obtaining operation and maintenance requests including non-text content and text content based on the text content input by the user and the preceding content of the text content, and supports obtaining operation and maintenance requests including images (i.e., non-text content) and text content based on the image of the target region selected by the user (the image is non-text content) and the text content input by the user.

[0017] Optionally, the multimodal large language model is used to: encode the first content and the second content to obtain a multimodal encoding vector, convert the multimodal encoding vector into a semantic vector, and obtain the at least one operation and maintenance task based on the semantic vector. Here, both the multimodal encoding vector and the semantic vector are vectors of the first content and the second content. The multimodal encoding vector is used to represent the first content and the second content, and the semantic vector is used to represent the first content and the second content. The semantic meaning of the semantic vector is the semantic meaning expressed by the combination of the first content and the second content.

[0018] Optionally, the multimodal large language model includes a multimodal encoder, a modality converter, and a language sub-model. The multimodal encoder encodes first content and second content to obtain a multimodal encoded vector. The modality converter converts the multimodal encoded vector into a semantic vector. The language sub-model obtains the at least one operation and maintenance task based on the semantic vector. The method further includes: acquiring training data, which is data related to network operation and maintenance, including first data content and second data content. The first data content is expressed in a non-textual form (i.e., the first data content is non-textual content), and the second data content is expressed in a textual form (i.e., the second data content is textual content); and training the modality converter in the multimodal large language model based on the training data. For example, the training data is input into the multimodal large language model, the output content of the multimodal large language model based on the training data is obtained, and the parameters of the modality converter in the multimodal large language model are adjusted based on the output content of the multimodal large language model based on the training data to train the modality converter. Specifically, the multimodal encoder and the language sub-model in the multimodal large language model are frozen, and the parameters of the modality converter in the multimodal large language model are adjusted according to the content output by the multimodal large language model based on the training data.

[0019] Since the training data includes both non-textual and textual content—that is, it contains content in at least two modalities—it can be called multimodal content or multimodal information, and thus, multimodal training data. This application uses multimodal training data to train the modality converter in a multimodal large language model, which improves the accuracy of the modality converter in converting multimodal encoded vectors into semantic vectors, thereby improving the accuracy of the content output by the multimodal large language model.

[0020] Optionally, the at least one operation and maintenance task includes a query task. Obtaining a request result matching the operation and maintenance request based on the at least one operation and maintenance task includes: encoding the first content and the second content according to the query task to obtain a multimodal encoding vector; converting the multimodal encoding vector into a semantic vector; obtaining a knowledge vector matching the semantic vector from the operation and maintenance knowledge base; and determining the knowledge corresponding to the knowledge vector as the request result matching the operation and maintenance request. Here, both the multimodal encoding vector and the semantic vector are vectors of the first content and the second content. The multimodal encoding vector is used to represent the first content and the second content, and the semantic vector is used to represent the first content and the second content. The semantic meaning of the semantic vector is the semantic meaning expressed by the combination of the first content and the second content. The operation and maintenance knowledge base includes a knowledge vector matching the semantic vector. The knowledge vector is the semantic vector of the knowledge corresponding to the knowledge vector, and the knowledge vector can express the semantic meaning of the knowledge corresponding to the knowledge vector.

[0021] The technical solution provided in this application stores knowledge vectors in an operations and maintenance knowledge base. This allows the retrieval of a request result matching the operations and maintenance request from the knowledge base based on the semantic vector of the operations and maintenance request (e.g., the semantic vectors of the content expressed in non-textual form and the content expressed in textual form included in the operations and maintenance request). This process only requires matching the semantic vector of the operations and maintenance request with the knowledge vectors in the knowledge base to determine the knowledge vector matching the semantic vector of the operations and maintenance request. Furthermore, the knowledge corresponding to the knowledge vector matching the semantic vector of the operations and maintenance request is determined as the request result matching the operations and maintenance request, thereby improving the efficiency and accuracy of retrieving the request result matching the operations and maintenance request.

[0022] Optionally, the maintenance request is for a maintenance issue related to the target network, and the first content is content related to the target network. Obtaining a request result matching the maintenance request based on the at least one maintenance task includes: obtaining network information of the target network based on the at least one maintenance task; and determining the request result matching the maintenance request based on the network information of the target network. For example, obtaining the network information of the target network by calling an application programming interface (API) based on the at least one maintenance task. Or,

[0023] This operation and maintenance request is a configuration request for the target network. The first part describes content related to the target network. Based on at least one operation and maintenance task, a request result matching the operation and maintenance request is obtained, including: configuring the target network according to the at least one operation and maintenance task, and determining the request result matching the operation and maintenance request based on the configuration result of the target network. For example, the target network is configured by calling an API according to the at least one operation and maintenance task.

[0024] Optionally, the first content is content specific to the target network, including: the first content is an image of the display interface provided by the target device, which is a network management device, a network operation and maintenance device, or a network device in the target network.

[0025] Optionally, the target device is a network operation and maintenance device of the target network, and the display interface is a network operation and maintenance interface of the target network, which displays a device distribution map of the target network; the first content is an image of the target area in the display interface, the target area includes at least a part of the device distribution map, the target area displays the target network device in the target network, and the target network device is a device in the target network that is in an abnormal state, the abnormal state includes at least one of the following: fault state, alarm state, offline state, and unregistered state.

[0026] Optionally, the method further includes: validating the request result matching the maintenance request using the multimodal large language model; and outputting the request result if the validation is successful. For example, based on the content of the maintenance request, the multimodal large language model is used to validate the request result; and if the validation is successful, the request result is displayed. Further, based on the content of the maintenance request and the execution process by which the network maintenance device obtains the request result, the multimodal large language model is used to validate the request result.

[0027] The verification of the request result matching the maintenance request in this application refers to a factual consistency verification of the request result. If the verification is successful, it indicates that the factual consistency of the request result is strong (i.e., the request result is highly consistent with the facts). If the verification fails, it indicates that the factual consistency of the request result is weak (i.e., the request result is weakly consistent with the facts). This application outputs the request result when the verification is successful; that is, it outputs a request result with strong factual consistency, which improves the reliability of the output request result, avoids outputting request results with weak factual consistency, and eliminates illusions.

[0028] Secondly, a network operation and maintenance device is provided, comprising at least one functional module for executing the network operation and maintenance method provided by the first aspect or any optional method thereof. The at least one functional module may be implemented based on software, hardware, or a combination of both, and may be arbitrarily combined or divided based on a specific implementation.

[0029] Thirdly, a network operation and maintenance device is provided, including a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program stored in the memory so that the network operation and maintenance device performs the network operation and maintenance method provided as described in the first aspect or any alternative method of the first aspect.

[0030] Fourthly, a network operation and maintenance system is provided, which includes network operation and maintenance equipment and an artificial intelligence (AI) server; the AI ​​server is deployed with a multimodal large language model, and the network operation and maintenance equipment includes network operation and maintenance devices as provided in the second or third aspect above. The network operation and maintenance device is used to obtain at least one operation and maintenance task based on a first content expressed in non-textual form included in the operation and maintenance request, using the multimodal large language model, and to obtain a request result matching the operation and maintenance request based on the at least one operation and maintenance task.

[0031] Fifthly, a computer-readable storage medium is provided, wherein a computer program is stored therein, which, when executed, implements the network operation and maintenance method provided as described in the first aspect or any alternative method of the first aspect.

[0032] In a sixth aspect, a computer program product is provided, comprising a program or code that, when executed, implements the network operation and maintenance method provided as described in the first aspect or any alternative method of the first aspect.

[0033] In a seventh aspect, a chip is provided, the chip including programmable logic circuitry and / or program instructions, the chip being used to implement the network operation and maintenance method provided as described in the first aspect or any alternative method of the first aspect.

[0034] The technical effects of the second to seventh aspects mentioned above can be referred to the technical effects of the first aspect and its optional implementation methods, and will not be elaborated here. Attached Figure Description

[0035] Figure 1 This is a schematic diagram illustrating an application scenario provided in an embodiment of this application;

[0036] Figure 2 This is a schematic diagram illustrating another application scenario provided by an embodiment of this application;

[0037] Figure 3 This is a schematic diagram illustrating another application scenario provided in the embodiments of this application;

[0038] Figure 4 This is a flowchart of a network operation and maintenance method provided in an embodiment of this application;

[0039] Figure 5 This is a schematic diagram of a network operation and maintenance interface provided in an embodiment of this application;

[0040] Figure 6 This is a schematic diagram of another network operation and maintenance interface provided in an embodiment of this application;

[0041] Figure 7 This is a schematic diagram of another network operation and maintenance interface provided in the embodiments of this application;

[0042] Figure 8 This is a schematic diagram of yet another network operation and maintenance interface provided in the embodiments of this application;

[0043] Figure 9 This is a schematic diagram of yet another network operation and maintenance interface provided in the embodiments of this application;

[0044] Figure 10 This is a flowchart of another network operation and maintenance method provided in the embodiments of this application;

[0045] Figure 11 This is a schematic diagram of yet another network operation and maintenance interface provided in the embodiments of this application;

[0046] Figure 12 This is a schematic diagram of yet another network operation and maintenance interface provided in the embodiments of this application;

[0047] Figure 13 This is a flowchart of another network operation and maintenance method provided in the embodiments of this application;

[0048] Figure 14 This is a flowchart of another network operation and maintenance method provided in the embodiments of this application;

[0049] Figure 15 This is a flowchart of yet another network operation and maintenance method provided in the embodiments of this application;

[0050] Figure 16 This is a schematic diagram of a network operation and maintenance device provided in an embodiment of this application;

[0051] Figure 17 This is a schematic diagram of another network operation and maintenance device provided in the embodiments of this application. Detailed Implementation

[0052] The embodiments of this application will now be described in further detail with reference to the accompanying drawings.

[0053] In today's information society, network operation and maintenance efficiency and user experience are crucial for the operations of various enterprises and organizations. Large language models (LLMs) have already played a vital role in network and security products. LLMs, also known as large models, are a type of artificial intelligence (AI) model based on deep learning. LLM-based products can understand and generate natural language text information. Training LLMs with training data can improve their understanding and predictive abilities, enabling LLM-based products to generate more accurate and natural text information. LLMs can be applied to fields such as natural language processing, machine translation, and text generation.

[0054] In the field of network operations and maintenance (O&M), LLM-based O&M products can understand user-inputted O&M questions and generate answers that match the questions, helping O&M personnel quickly understand network status, improve O&M efficiency, and optimize user experience. LLM has demonstrated powerful capabilities in automated fault diagnosis, intelligent customer service, and network configuration. However, currently, LLM applications in network O&M can only understand and process text information, and cannot understand or process non-text information such as images, videos, and audio, nor can they understand or process multimodal information. This results in relatively poor flexibility for current LLM-based network O&M. Text information refers to information expressed in text form. Non-text information refers to information expressed in non-text form, such as images and videos. Multimodal information refers to information that includes at least two modalities (i.e., at least two forms of expression). When non-text information includes at least two forms of expression (e.g., including images and audio), this non-text information is also considered multimodal information. For example, in the field of network operations and maintenance (O&M), it is currently necessary to manually convert O&M problems, which are not expressed in text form, into text form before inputting them into a dialog box for interaction with an LLM (Local Management Module). This conversion process is time-consuming and inefficient, leading to low overall network O&M efficiency. Furthermore, O&M problems may be difficult to convert into text form (some problems are inherently difficult to express in words), and the conversion process can easily result in the loss of important information, leading to lower accuracy of answers generated from the converted text-based O&M problems. As another example, the process of generating answers using an LLM may require searching an O&M knowledge base. Since LLM can only understand and process text information, the construction of this knowledge base requires manual conversion of non-textual knowledge (i.e., knowledge expressed in non-text form) into text knowledge (i.e., knowledge expressed in text form). This results in low efficiency in building the O&M knowledge base, and the information lost during the conversion process can have a disastrous impact on the O&M knowledge base.

[0055] This application provides a technical solution that introduces a multimodal large language model into the field of network operations and maintenance (O&M) to assist O&M personnel in performing network O&M. Since the multimodal large language model can understand and process multimodal information, its introduction into the network O&M field can improve the flexibility and efficiency of network O&M, as well as the accuracy of results obtained based on multimodal information (e.g., answers matching O&M questions), making network O&M more intelligent and broadening its application scope.

[0056] Specifically, this application provides a network operation and maintenance method and apparatus, wherein the network operation and maintenance method is executed by a network operation and maintenance device. After obtaining an operation and maintenance request including first content expressed in a non-textual form, the network operation and maintenance device uses a multimodal large language model to obtain at least one operation and maintenance task based on the first content, and then obtains a request result matching the operation and maintenance request based on the at least one operation and maintenance task. It is evident that the multimodal large language model used in this application can understand and process the non-textual content (i.e., non-textual content) included in the operation and maintenance request. That is, the multimodal large language model can understand and process operation and maintenance requests including non-textual content. Therefore, using this multimodal large language model to process operation and maintenance requests offers high flexibility, helping to improve the flexibility of network operation and maintenance. The operation and maintenance request can be an operation and maintenance problem or a configuration request related to network operation and maintenance. When the operation and maintenance request is an operation and maintenance problem, the request result matching the operation and maintenance request can be an answer matching the operation and maintenance problem (i.e., a response to the operation and maintenance problem). The non-textual form can include at least one of image or audio forms. For example, the first content may be content expressed in the form of an image; or, the first content may be content expressed in the form of audio; or, the first content may include content expressed in the form of an image and content expressed in the form of audio. Content expressed in the form of an image may be a single image or a video segment comprising multiple frames. For example, the first content may include at least one of an image, video, and audio.

[0057] In this embodiment, the network operation and maintenance equipment can be a single server, a server cluster consisting of several servers, or a cloud computing service center. The multimodal large language model can be deployed within the network operation and maintenance equipment or in other devices communicatively connected to it. These other devices can be a single server (e.g., an AI server), a server cluster consisting of several servers (e.g., AI servers), or a cloud computing service center. The AI ​​server is an AI-related server, including but not limited to: servers capable of deploying AI models, servers capable of providing AI computation, and servers capable of performing AI training.

[0058] As an example, please refer to Figure 1This illustration shows a schematic diagram of an application scenario provided by an embodiment of this application. The application scenario includes a network operation and maintenance device 110 and an AI server 120, with the network operation and maintenance device 110 and AI server 120 communicatively connected. A multimodal large language model is deployed in the AI ​​server 120, and the network operation and maintenance device 110 can access the multimodal large language model in the AI ​​server 120 to obtain operation and maintenance tasks using the multimodal large language model. For example, the network operation and maintenance device 110 is used to: obtain an operation and maintenance request A, which includes first content expressed in non-textual form; obtain at least one operation and maintenance task based on the first content using the multimodal large language model in the AI ​​server 120; and obtain a request result matching the operation and maintenance request A based on the at least one operation and maintenance task. The multimodal large language model is used to: encode the first content to obtain a non-textual encoded vector; convert the non-textual encoded vector into a semantic vector; and obtain the at least one operation and maintenance task based on the semantic vector. Figure 1 As shown, the multimodal large language model includes a multimodal encoder, a modality converter, and a language sub-model, which are sequentially connected. The multimodal encoder encodes the first content to obtain a non-textual encoded vector. The modality converter converts the non-textual encoded vector into a semantic vector. The language sub-model obtains the at least one operational task based on the semantic vector. The multimodal encoder, modality converter, and language sub-model can all be modules based on a transformer architecture. The multimodal encoder is also called a multimodal semantic alignment encoder, and the language sub-model is also called a generative language model. The non-textual form can include at least one of image or audio forms. Content expressed in image form can be a single image or a video segment including multiple frames. The first content can include at least one of image, video, and audio.

[0059] As another example, please refer to Figure 2 This illustration shows a schematic diagram of another application scenario provided by an embodiment of this application. Figure 1 Based on the application scenarios shown, Figure 2In the application scenario shown, the multimodal large language model also includes a text encoder and a content recognizer. The content recognizer, the text encoder, and the language sub-model are connected sequentially, as are the content recognizer, the multimodal encoder, the modality converter, and the language sub-model. The content recognizer is used to identify the type of the content of the maintenance request (e.g., whether the maintenance request includes content expressed in non-textual form, which is also called non-textual content or non-textual information). If the content recognizer determines that the maintenance request includes content expressed in non-textual form, it inputs the content of the maintenance request into the multimodal encoder. The multimodal encoder encodes the content of the maintenance request to obtain a content vector (e.g., if the content of the maintenance request is non-textual content, the content vector is a non-textual encoded vector). The modality converter converts the content vector (e.g., the non-textual encoded vector) into a semantic vector. The language sub-model obtains the maintenance task based on the semantic vector. When the content recognizer determines that the content of the maintenance request is expressed in text form (content expressed in text form is also called text content or text information), the content recognizer inputs the content of the maintenance request into the text encoder. The text encoder is used to encode the content of the maintenance request to obtain a text encoding vector. The text encoding vector is a semantic vector. The language sub-model is used to obtain the maintenance task based on the text encoding vector.

[0060] As another example, please refer to Figure 3 The diagram illustrates another application scenario provided by the embodiments of this application. Figure 3 A multimodal large language model is like this Figure 1 The multimodal large language model shown is used as an example for illustration. Figure 3As shown, the network operation and maintenance device 110 includes an intelligent agent. This intelligent agent is an intelligent assistant (or intelligent tool) based on AI technology to assist in network operation and maintenance. The intelligent agent can provide the ability to interact with users (e.g., dialog boxes) and interact with multimodal large language models (e.g., generating prompt words that the large model can recognize). The intelligent agent can be implemented using software, hardware, or a combination of both. The intelligent agent is used to: obtain an operation and maintenance request A based on user operations; obtain at least one operation and maintenance task based on the first content (expressed in non-textual form) included in the operation and maintenance request A using the multimodal large language model in the AI ​​server 120; and obtain a request result matching the operation and maintenance request A based on the at least one operation and maintenance task. Specifically, the intelligent agent is used to: obtain a first prompt word template from a prompt word template library based on the first content included in the operation and maintenance request A; generate a first prompt word based on the first prompt word template and the first content; send the first prompt word to the AI ​​server 120; receive at least one operation and maintenance task obtained by the AI ​​server 120 based on the first prompt word; and obtain a request result matching the operation and maintenance request A based on the at least one operation and maintenance task. The AI ​​server 120 is configured to: receive a first prompt word sent by the intelligent agent; input the first prompt word into the multimodal large language model; enable the multimodal large language model to obtain at least one maintenance task based on the first prompt word and output the at least one maintenance task; and send the at least one maintenance task to the intelligent agent. Specifically, the multimodal large language model is configured to: extract first content from the first prompt word; encode the first content to obtain a non-text encoded vector; convert the non-text encoded vector into a semantic vector; and obtain the at least one maintenance task based on the semantic vector. In one embodiment, the at least one maintenance task includes a query task, and the intelligent agent is configured to query the maintenance knowledge base according to the query task to obtain a request result matching maintenance request A. For example, the intelligent agent is configured to: encode the first content included in maintenance request A according to the query task to obtain a non-text encoded vector; convert the non-text encoded vector into a semantic vector; obtain a knowledge vector matching the semantic vector from the maintenance knowledge base; and determine the knowledge corresponding to the knowledge vector as a request result matching maintenance request A. The maintenance knowledge base includes the knowledge vector. In another embodiment, maintenance request A is a maintenance issue for the target network, and the first content is content related to the target network. For example, the first content is an image of the display interface provided by the target device. The target device is the network management device of the target network, the network maintenance device of the target network (e.g., network maintenance device 110), or a network device in the target network. The intelligent agent is used to: obtain network information of the target network according to the at least one maintenance task, and determine the request result that matches maintenance request A according to the network information of the target network.In another embodiment, maintenance request A is a configuration request for a target network, and the first content is content for the target network, such as an image of the display interface provided by the target device. The intelligent agent is used to: configure the target network according to the at least one maintenance task, and determine a request result that matches maintenance request A based on the configuration result of the target network.

[0061] In specific implementation methods, such as Figure 3 As shown, the intelligent agent includes a management module and a capability engine. The management module is used to: obtain an operation and maintenance request A based on user actions; obtain at least one operation and maintenance task using a multimodal large language model in the AI ​​server 120 based on the first content included in the operation and maintenance request A; and obtain a request result matching the operation and maintenance request A by calling the capability engine based on the at least one operation and maintenance task. Specifically, the management module is used to: obtain a first prompt word template from a prompt word template library based on the first content included in the operation and maintenance request A; generate a first prompt word based on the first prompt word template and the first content; send the first prompt word to the AI ​​server 120; receive at least one operation and maintenance task obtained by the AI ​​server 120 based on the first prompt word; and obtain a request result matching the operation and maintenance request A by calling the capability engine based on the at least one operation and maintenance task. Figure 3 As shown, the management module includes an interface management unit, an engine management unit, and a prompt word management unit. The capability engine includes a knowledge engine and a business engine. The interface management unit is used to obtain maintenance request A based on user operations. The prompt word management unit is used to: obtain a first prompt word template from the prompt word template library according to the first content included in maintenance request A; generate a first prompt word according to the first prompt word template and the first content; send the first prompt word to the AI ​​server 120; and receive at least one maintenance task obtained by the AI ​​server 120 according to the first prompt word. The engine management unit is used to call the capability engine to obtain a request result matching maintenance request A according to the at least one maintenance task. In one embodiment, the at least one maintenance task includes a query task, and the engine management unit is used to call the knowledge engine to query the maintenance knowledge base according to the query task to obtain a request result matching maintenance request A. In another embodiment, maintenance request A is a maintenance problem for a target network, and the first content is content for the target network. The engine management unit is used to: call the business engine to obtain network information of the target network according to the at least one maintenance task, and determine a request result matching maintenance request A according to the network information of the target network. In another embodiment, maintenance request A is a configuration request for the target network. The intelligent agent is used to: call the business engine to configure the target network according to the at least one maintenance task, and determine the request result that matches maintenance request A based on the configuration result of the target network.

[0062] Among the optional implementation methods, such as Figure 3As shown, the capability engine also includes a validation engine and a drawing engine. The engine management unit is further used to: call the validation engine to perform legality validation on the first content. For example, to validate whether the first content contains sensitive information. The prompt word management unit is used to: if the validation engine successfully validates the first content, retrieve a first prompt word template from the prompt word template library based on the first content, and generate a first prompt word based on the first prompt word template and the first content. The engine management unit is also used to: call the drawing engine to draw a chart based on the request results matching the maintenance request A, so as to display the request results matching the maintenance request A in chart form. In an optional implementation, the capability engine also includes a rendering engine (…). Figure 3 The rendering engine (also called the renderer) is not shown in the diagram. The engine management unit is also used to: invoke the rendering engine to render based on the request result that matches the operation and maintenance request A.

[0063] In an optional implementation, maintenance request A further includes second content expressed in text form (i.e., maintenance request A includes first content expressed in non-text form and second content expressed in text form). Network maintenance device 110 is used to: obtain at least one maintenance task based on the first and second content using a multimodal large language model in AI server 120; and obtain a request result matching maintenance request A based on the at least one maintenance task. The multimodal large language model is used to: encode the first and second content to obtain a multimodal encoding vector; convert the multimodal encoding vector into a semantic vector; and obtain the at least one maintenance task based on the semantic vector. Figure 3As shown, the multimodal large language model includes a multimodal encoder, a modality converter, and a language sub-model, which are sequentially connected. The multimodal encoder encodes first content and second content to obtain a multimodal encoded vector. The modality converter converts the multimodal encoded vector into a semantic vector. The language sub-model obtains the at least one operation and maintenance task based on the semantic vector. Specifically, the network operation and maintenance device 110 includes an intelligent agent, which is used to: obtain at least one operation and maintenance task based on the multimodal large language model in the AI ​​server 120 according to the operation and maintenance request A (including first and second content); and obtain a request result matching the operation and maintenance request A based on the at least one operation and maintenance task. In a specific implementation, the intelligent agent is used to: obtain a first prompt word template from a prompt word template library based on the first and second content included in the maintenance request A; generate a first prompt word based on the first prompt word template, the first content, and the second content; send the first prompt word to the AI ​​server 120; receive at least one maintenance task obtained by the AI ​​server 120 based on the first prompt word; and obtain a request result matching the maintenance request A based on the at least one maintenance task. The AI ​​server 120 is used to: receive the first prompt word sent by the intelligent agent; input the first prompt word into the multimodal large language model; enable the multimodal large language model to obtain the at least one maintenance task based on the first prompt word and output the at least one maintenance task; and send the at least one maintenance task to the intelligent agent. In a specific implementation, the multimodal large language model is used to: extract the first and second content from the first prompt word; encode the first and second content to obtain a multimodal encoding vector; convert the multimodal encoding vector into a semantic vector; and obtain the at least one maintenance task based on the semantic vector. In one embodiment, the at least one operation and maintenance task includes a query task. The intelligent agent is configured to: encode the first and second content included in the operation and maintenance request A according to the query task to obtain a multimodal encoding vector; convert the multimodal encoding vector into a semantic vector; obtain a knowledge vector matching the semantic vector from the operation and maintenance knowledge base; and determine the knowledge corresponding to the knowledge vector as a request result matching the operation and maintenance request A. In another embodiment, the operation and maintenance request A is an operation and maintenance problem for a target network, and the first content is content related to the target network. The intelligent agent is configured to: obtain network information of the target network according to the at least one operation and maintenance task; and determine a request result matching the operation and maintenance request A based on the network information of the target network. In yet another embodiment, the operation and maintenance request A is a configuration request for a target network. The intelligent agent is configured to: call the business engine to configure the target network according to the at least one operation and maintenance task; and determine a request result matching the operation and maintenance request A based on the configuration result of the target network.

[0064] In specific implementation methods, such as Figure 3As shown, the intelligent agent includes a management module and a capability engine. The management module is used to: obtain at least one maintenance task based on the first and second content included in maintenance request A using a multimodal large language model in AI server 120; and obtain a request result matching maintenance request A by calling the capability engine based on the at least one maintenance task. Specifically, the management module is used to: obtain a first prompt word template from the prompt word template library based on the first and second content included in maintenance request A; generate a first prompt word based on the first prompt word template, the first content, and the second content; send the first prompt word to AI server 120; receive at least one maintenance task obtained by AI server 120 based on the first prompt word; and obtain a request result matching maintenance request A by calling the capability engine based on the at least one maintenance task. Figure 3 As shown, the management module includes an interface management unit, an engine management unit, and a prompt word management unit. The capability engine includes a knowledge engine and a business engine. The interface management unit is used to obtain maintenance request A based on user operations. The prompt word management unit is used to: obtain a first prompt word template from the prompt word template library based on the first and second content included in maintenance request A; generate a first prompt word based on the first prompt word template, the first content, and the second content; send the first prompt word to the AI ​​server 120; and receive at least one maintenance task obtained by the AI ​​server 120 based on the first prompt word. The engine management unit is used to call the capability engine to obtain a request result matching maintenance request A based on the at least one maintenance task. In one embodiment, the at least one maintenance task includes a query task, and the engine management unit is used to call the knowledge engine to query the maintenance knowledge base based on the query task to obtain a request result matching maintenance request A. In another embodiment, maintenance request A is a maintenance problem for a target network, and both the first and second content are content related to the target network. The engine management unit is used to: call the business engine to obtain network information of the target network based on the at least one maintenance task; and determine a request result matching maintenance request A based on the network information of the target network. In another embodiment, maintenance request A is a configuration request for the target network. The first content and the second content are both content for the target network. The engine management unit is used to: call the business engine to configure the target network according to the at least one maintenance task, and determine the request result that matches maintenance request A based on the configuration result of the target network.

[0065] Among the optional implementation methods, such as Figure 3As shown, the capability engine also includes a validation engine and a drawing engine. The engine management unit is further used to: invoke the validation engine to perform legality validation on the first and second content. For example, to validate whether the first and second content contain sensitive information. The prompt word management unit is used to: if the validation engine successfully validates the first and second content, retrieve a first prompt word template from the prompt word template library based on the first and second content, and generate a first prompt word based on the first prompt word template, the first content, and the second content.

[0066] In this embodiment of the application, the network operation and maintenance device 110 can obtain the operation and maintenance request A based on the user's operation in the following three ways.

[0067] The first implementation method: Network operation and maintenance device 110 generates operation and maintenance request A based on the user input.

[0068] In a specific implementation, network maintenance device 110 provides a network maintenance interface. The user inputs content into this interface, and network maintenance device 110 receives the user's input and generates maintenance request A based on it. In a specific implementation, such as... Figure 3 As shown, the network operation and maintenance device 110 includes an interface management unit that receives user input and generates an operation and maintenance request A based on the user input. For example, this interface management unit includes a dialog management unit (…). Figure 3 (Not shown in the image), the dialogue management unit receives the content input by the user, and generates an operation and maintenance request A based on the content input by the user.

[0069] In one example, maintenance request A includes first content expressed in non-text form. The user inputs this first content into the network maintenance interface, and network maintenance device 110 receives the user's input. Network maintenance device 110 then generates maintenance request A based on this first content. In a specific implementation, such as... Figure 3 As shown, the network operation and maintenance device 110 includes an interface management unit. This interface management unit receives first content input by the user and generates an operation and maintenance request A based on the first content. For example, this interface management unit includes a dialog management unit (…). Figure 3 (Not shown in the image), the dialogue management unit receives the first content input by the user, and the dialogue management unit generates an operation and maintenance request A based on the first content.

[0070] In another example, maintenance request A includes a first piece of content expressed in non-text form and a second piece of content expressed in text form. The user inputs the first and second pieces of content into the network maintenance interface. Network maintenance device 110 receives the first and second pieces of content input by the user and generates maintenance request A based on the first and second pieces of content. In a specific implementation, such as... Figure 3 As shown, the network operation and maintenance device 110 includes an interface management unit. This interface management unit receives first and second content input by the user, and generates an operation and maintenance request A based on the first and second content. For example, the interface management unit includes a dialog management unit (…). Figure 3 (Not shown in the image), the dialogue management unit receives the first content and the second content input by the user, and generates an operation and maintenance request A based on the first content and the second content.

[0071] The second implementation method: Network operation and maintenance device 110 generates operation and maintenance request A based on the content input by the user and the preceding content of the user input.

[0072] In a specific implementation, network maintenance device 110 provides a network maintenance interface. The user inputs content into this interface, and network maintenance device 110 receives the user's input. Network maintenance device 110 also obtains the context of the user's input and generates maintenance request A based on the user's input and its context. In a specific implementation, such as... Figure 3 As shown, the network operation and maintenance device 110 includes an interface management unit. This interface management unit receives user input, obtains the context of the user input, and generates an operation and maintenance request A based on the user input and its context. For example, this interface management unit includes a dialogue management unit (…). Figure 3 (Not shown in the image), the dialogue management unit receives the content input by the user, obtains the preceding content of the user input, and generates an operation and maintenance request A based on the user input and the preceding content of the user input.

[0073] In one example, maintenance request A includes a first content expressed in non-text form and a second content expressed in text form. The user inputs the second content into the network maintenance interface. Network maintenance device 110 receives the second content input by the user. Network maintenance device 110 obtains the preceding content of the second content from the dialogue context. Network maintenance device 110 obtains the first content based on the preceding content of the second content. Network maintenance device 110 generates maintenance request A based on the first and second content. The preceding content of the second content refers to the content the user interacted with the network maintenance interface before inputting the second content, and the preceding content of the second content includes the first content. In a specific implementation, such as... Figure 3 As shown, the network operation and maintenance device 110 includes an interface management unit. This interface management unit receives second content input by the user, obtains the preceding text of the second content from the dialogue context, obtains first content based on the preceding text of the second content, and generates an operation and maintenance request A based on the first and second content. For example, this interface management unit includes a dialogue management unit (…). Figure 3 (Not shown in the diagram), the dialogue management unit receives the second content input by the user, obtains the preceding content of the second content from the dialogue context, obtains the first content based on the preceding content of the second content, and generates an operation and maintenance request A based on the first content and the second content.

[0074] The third implementation method: Network operation and maintenance device 110 generates operation and maintenance request A based on the area selection command triggered by the user and the content entered by the user.

[0075] In a specific implementation, network maintenance device 110 provides a network maintenance interface. The user triggers a region selection command and inputs content within this interface. Network maintenance device 110 receives the user-triggered region selection command and the user-input content, and then generates maintenance request A based on these two commands. In a specific implementation, such as... Figure 3 As shown, the network operation and maintenance device 110 includes an interface management unit. This interface management unit receives a region selection command triggered by a user and content input by the user. The interface management unit generates an operation and maintenance request A based on the user-triggered region selection command and the user-input content. For example, this interface management unit includes a dialog management unit (…). Figure 3 (Not shown in the image), this dialog management unit receives user input.

[0076] In one example, maintenance request A includes a first content expressed in non-text form and a second content expressed in text form. A user triggers a region selection command and inputs the second content in the network maintenance interface. Network maintenance device 110 receives the region selection command and the second content. Based on the region selection command, network maintenance device 110 obtains the first content and generates maintenance request A according to the first and second content. The first content is an image of the target region selected by the region selection command. For example, the network maintenance interface displays a device distribution map (also called a digital map) of the target network. The target region includes at least a portion of the device distribution map (e.g., the target region is a portion of the device distribution map). The target region displays target network devices in the target network, which are devices in an abnormal state. This abnormal state includes at least one of the following: fault state, alarm state, offline state, and unregistered state. In specific implementation methods, such as... Figure 3 As shown, the network operation and maintenance device 110 includes an interface management unit. This interface management unit receives a region selection command triggered by a user and second content input by the user. The interface management unit generates an operation and maintenance request A based on the user-triggered region selection command and the second content input by the user. For example, the interface management unit includes a dialogue management unit, which receives the second content input by the user.

[0077] It is understandable that the first implementation method described above can be considered as the user inputting maintenance request A into the network maintenance device 110, and the network maintenance device 110 receiving the maintenance request A input by the user. Figure 3 The illustration may depict a scenario where network maintenance equipment 110 receives maintenance request A input by a user.

[0078] In this embodiment, the network operation and maintenance device 110 can implement the network operation and maintenance method provided in this embodiment by running an executable program. The executable program is presented in the form of an application installation package. After installing the application installation package, the network operation and maintenance device 110 can implement the network operation and maintenance method provided in this embodiment by running the executable program. The network operation and maintenance device 110 has a human-computer interaction component, through which the user interacts with the network operation and maintenance device 110. For example, the user inputs an operation and maintenance request A to the network operation and maintenance device 110 through the human-computer interaction component. The human-computer interaction component includes an input component and an output component. The input component includes a touch component, a touch display component, an audio component, a keyboard, a mouse, etc. The output component includes a display component, a touch display component, an audio component, etc., which are not limited in this embodiment.

[0079] It should be understood that the above descriptions of application scenarios are for illustrative purposes only and are not intended to limit the technical solutions of the embodiments of this application. For example, in some embodiments, the operation and maintenance knowledge base can be located in network operation and maintenance equipment. In other embodiments, the business front-end and business back-end are located in the same device, but the business front-end and business back-end are located in different devices from the intelligent agent. As those skilled in the art will know, the application scenarios of the embodiments of this application can be appropriately adjusted as business needs change, and the embodiments of this application do not list them all.

[0080] The above is an introduction to the application scenarios of this application. The following describes the method embodiments of this application.

[0081] Please refer to Figure 4 The diagram illustrates a flowchart of a network operation and maintenance method provided in an embodiment of this application. This network operation and maintenance method is executed by a network operation and maintenance device. The network operation and maintenance device may be... Figures 1 to 3 Network maintenance device 110 in the illustrated application scenario. See also... Figure 4 The network operation and maintenance method includes the following steps S401 to S403.

[0082] S401. Obtain maintenance request A. Maintenance request A includes first content, which is expressed in a non-textual form.

[0083] Operation and maintenance request A can be an operation and maintenance problem or a configuration request related to network operation and maintenance. Operation and maintenance request A includes first content expressed in non-text form, and may also include second content expressed in text form. That is, operation and maintenance request A includes first content expressed in non-text form, or operation and maintenance request A includes first content expressed in non-text form and second content expressed in text form. Non-text form can include image or audio form. Content expressed in image form can be a single image or a video containing multiple frames. For example, the first content expressed in image form can be a screenshot of a problem ticket, or a screenshot of the display interface showing an abnormal device, or a video recording demonstrating the problem with the abnormal device. An abnormal device refers to a device in an abnormal state, which includes at least one of the following: fault state, alarm state, offline state, and unregistered state.

[0084] Network maintenance equipment can obtain maintenance request A based on user operations. Specifically, the network maintenance equipment can provide a network maintenance interface, which is a display interface for user interaction with the equipment. The network maintenance equipment obtains maintenance request A based on user operations within this interface. The implementation method for obtaining maintenance request A varies depending on its content. The following describes two scenarios for how the network maintenance equipment obtains maintenance request A based on user operations.

[0085] The first scenario: Operation and maintenance request A includes the first content but does not include the second content.

[0086] In the first scenario, the implementation methods for network operation and maintenance equipment to obtain operation and maintenance request A based on user operations include the following two embodiments.

[0087] First embodiment: The user enters the first content in the network operation and maintenance interface provided by the network operation and maintenance equipment. The network operation and maintenance equipment receives the first content entered by the user in the network operation and maintenance interface and generates operation and maintenance request A based on the first content entered by the user.

[0088] In its specific implementation, the network operation and maintenance interface includes a dialog box where the user enters initial information. For an example, please refer to [link / reference needed]. Figure 5 This illustration shows a schematic diagram of a network operation and maintenance interface provided in an embodiment of this application. The network operation and maintenance interface includes a first display area Q1, in which a dialog box is displayed, allowing the user to input first content. The dialog box includes an input box 501, a send control 502, an audio input control 503, and a dialog display area 504. The dialog display area 504 displays the input prompt message "Hello, how can I help you?" The user can input the first content to the network operation and maintenance device through the input box 501 and / or the audio input control 503 according to this input prompt message. For example, if the first content is expressed in image form, the user inputs the first content in the input box 501 according to the input prompt message and clicks the send control 502 to input the first content to the network operation and maintenance device. As another example, if the first content is expressed in audio form, the user inputs the first content to the network operation and maintenance device through the audio input control 503 according to the input prompt message.

[0089] For example, the first content is image 1, and the user is in such a position... Figure 5 After entering image 1 (i.e., the first content) in the dialog box shown, the network operation and maintenance interface changes as follows. Figure 6 As shown. Combined with Figure 5 See also Figure 6The dialog box displays image 1, which can be a screenshot of a problem ticket. Image 1 shows the message "AP7 in the lobby is faulty, please diagnose the cause." The network maintenance equipment receives image 1 from the user in the dialog box and generates maintenance request A based on image 1. Maintenance request A includes image 1.

[0090] Among the optional implementation methods, such as Figure 5 As shown, the first display area Q1 also includes a file upload control 505 and an operation control bar 506, with a close control 507 displayed in the operation control bar 506. Users can input information in file format using the file upload control 505; for example, users can input video files, audio files, etc., to the network maintenance equipment. Users can close the dialog box using the close control 507. Furthermore, this network maintenance interface can be the network maintenance interface of the target network, such as... Figure 5 As shown, the network operation and maintenance interface also includes a second display area Q2, which displays a device distribution map of the target network. According to... Figure 5 As can be seen, the target network includes access points (APs) 1-9. APs 1-9 are all network devices in the target network. AP1 is located in the manager's office, APs 2-3 are located in office 1, AP4 is located in the finance office, APs 5-6 are located in office 2, and APs 7-9 are located in the lobby. This device distribution map is also known as a digital map.

[0091] In an optional implementation, the first display area Q1 can be collapsed or expanded. The network maintenance device collapses or expands the first display area Q1 based on user operations in the second display area Q2. For example, if a user clicks on the icon of an AP in the second display area Q2, the network maintenance device will expand the first display area Q1. After the network maintenance device expands the first display area Q1, both the first display area Q1 and the second display area Q2 will be displayed in the network maintenance interface. If the user clicks on the AP icon again, the network maintenance device will collapse the first display area Q1. After the network maintenance device collapses the first display area Q1, the second display area Q2 will be displayed in the network maintenance interface, and the first display area Q1 will not be displayed.

[0092] The second embodiment: A user triggers a region selection command in the network operation and maintenance interface provided by the network operation and maintenance equipment. The network operation and maintenance equipment receives the region selection command, obtains first content based on the region selection command, and generates operation and maintenance request A based on the first content. Here, the first content is an image of the target region selected by the region selection command, and the target region is a region in the network operation and maintenance interface.

[0093] In the specific implementation, the network operation and maintenance interface displays a device distribution map of the target network. The target area includes at least a portion of this device distribution map, and the target area displays the target network devices within the target network. The target network devices are those devices in an abnormal state within the target network. This abnormal state includes at least one of the following: fault state, alarm state, offline state, or unregistered state. For an example, please refer to [reference needed]. Figure 5 The second display area Q2 of the network operation and maintenance interface displays a device distribution map of the target network. Users can select a target area in Q2 to trigger an area selection command. For example, a user can draw a "focus box" in Q2 to trigger the area selection command; the area selected by the "focus box" is the target area. For instance, if a user determines that AP7 in the target network is in an abnormal state (e.g., the user determines that the network quality near AP7 is poor, thus determining that AP7 is in an abnormal state), such as... Figure 7 As shown, the user draws a rectangular "focus box" in the second display area Q2 to trigger a region selection command. The region X selected by the "focus box" is the target region, which includes a portion of the device distribution map of the target network. The target region displays AP7 in the target network. The network maintenance device receives the region selection command and obtains an image of the target region X (i.e., the first content) based on the command. The network maintenance device then generates a maintenance request A based on the image of the target region X, which includes the image of the target region X. The user can trigger the region selection command by drawing a "focus box" in the second display area Q2 using the network maintenance device's mouse, touch display component, or other human-computer interaction components.

[0094] In an optional implementation, before the user triggers a region selection command in the network operation and maintenance interface, the first display area Q1 is in a collapsed state, and the network operation and maintenance interface displays the second display area Q2 but does not display the first display area Q1. After the user triggers a region selection command in the network operation and maintenance interface, the network operation and maintenance device expands the first display area Q1. After the network operation and maintenance device expands the first display area Q1, the network operation and maintenance interface displays both the first display area Q1 and the second display area Q2. This embodiment of the application does not limit this.

[0095] The second scenario: Operation and maintenance request A includes both the first and second parts. That is, operation and maintenance request A includes multimodal content.

[0096] In the second scenario, the implementation methods for network operation and maintenance equipment to obtain operation and maintenance request A based on user operations include the following three embodiments.

[0097] First embodiment: The user inputs first content and second content in the network operation and maintenance interface provided by the network operation and maintenance equipment. The network operation and maintenance equipment receives the first content and second content input by the user in the network operation and maintenance interface and generates operation and maintenance request A based on the first content and second content input by the user.

[0098] In its specific implementation, the network operation and maintenance interface includes a dialog box where the user enters first and second information. As an example, the network operation and maintenance interface is as follows: Figure 5 As shown, the first display area Q1 of the network operation and maintenance interface displays a dialog box where the user inputs first and second content. For example, if the first content is expressed as an image, the user inputs the first and second content in input box 501 and clicks the send control 502 to input the first and second content to the network operation and maintenance device. As another example, if the first content is expressed as audio, the user inputs the first content to the network operation and maintenance device through audio input control 503, and then inputs the second content in input box 501 and clicks the send control 502 to input the second content to the network operation and maintenance device.

[0099] For example, the first content is image 1, and the user is in such a position... Figure 5 After entering Image 1 (i.e., the first content) and the text "Please diagnose the root cause" (i.e., the second content) in the dialog box shown, the network maintenance interface changes as follows: Figure 8 As shown. Combined with Figure 5 See also Figure 8 The dialog box displays image 1 and the text "Please diagnose the root cause" in the dialog box display area 504. Image 1 can be a screenshot of a problem ticket. The network operation and maintenance device receives image 1 and the text "Please diagnose the root cause" entered by the user in the dialog box. The network operation and maintenance device generates operation and maintenance request A based on image 1 and the text "Please diagnose the root cause". Operation and maintenance request A includes image 1 and the text "Please diagnose the root cause".

[0100] The second embodiment: A user triggers a region selection command in the network operation and maintenance interface provided by the network operation and maintenance equipment, and the user inputs second content in the network operation and maintenance interface. The network operation and maintenance equipment receives the region selection command and the second content, obtains the first content based on the region selection command, and generates an operation and maintenance request A based on the first content and the second content. Here, the first content is an image of the target region selected by the region selection command, and the target region is a region in the network operation and maintenance interface.

[0101] In the specific implementation, the network operation and maintenance interface displays a device distribution map of the target network. The target area includes at least a portion of the device distribution map, and the target area displays the target network devices within the target network. The target network devices are those devices in an abnormal state within the target network. This abnormal state includes at least one of the following: fault state, alarm state, offline state, or unregistered state. Specifically, the network operation and maintenance interface includes a dialog box where the user inputs second information, and the network operation and maintenance device receives this second information from the user in the dialog box. As an example, the network operation and maintenance interface is as follows: Figure 5 As shown, the network operation and maintenance interface includes a first display area Q1 and a second display area Q2. The first display area Q1 displays a dialog box, and the second display area Q2 displays a device distribution diagram of the target network. Users can select a target area in the second display area Q2 to trigger an area selection command. Users can also enter secondary information in the dialog box. For example, a user can draw a "focus box" in the second display area Q2 to trigger the area selection command; the area selected by the "focus box" is the target area. For instance, if a user determines that AP7 in the target network is in an abnormal state (e.g., the user determines that the network quality near AP7 is poor, thus determining that AP7 is in an abnormal state), such as... Figure 9 As shown, the user draws a rectangular "focus box" in the second display area Q2 to trigger a region selection command. The region X selected by the "focus box" is the target region, which displays AP7 in the target network. After triggering the region selection command in the second display area Q2, the user enters the text "Why is the network in this area bad?" (i.e., the second content) in the dialog box. The network maintenance device receives the region selection command and the text "Why is the network in this area bad?" (i.e., the second content). Based on the region selection command, the network maintenance device obtains an image of the target region X (i.e., the first content). The network maintenance device generates a maintenance request A based on the image of the target region X and the text "Why is the network in this area bad?". Maintenance request A includes the image of the target region X and the text "Why is the network in this area bad?". The user can trigger the region selection command by drawing a "focus box" in the second display area Q2 using the network maintenance device's mouse, touch display component, or other human-computer interaction components.

[0102] In an optional implementation, before the user triggers a region selection command in the network operation and maintenance interface, the first display area Q1 is collapsed, and the network operation and maintenance interface displays a second display area Q2 but not the first display area Q1. After the user triggers a region selection command in the network operation and maintenance interface, the network operation and maintenance device expands the first display area Q1 (i.e., expands the dialog box). After the network operation and maintenance device expands the first display area Q1, the user enters second content in the dialog box in the first display area Q1; this embodiment of the application does not limit this.

[0103] The third embodiment: The user inputs second content into the network operation and maintenance interface provided by the network operation and maintenance equipment. The network operation and maintenance equipment receives the second content input by the user, obtains the preceding content of the second content, obtains the first content based on the preceding content of the second content, and generates operation and maintenance request A based on the first content and the second content. Here, the preceding content of the second content refers to the content that the user interacted with the network operation and maintenance interface before inputting the second content, and the preceding content of the second content includes the first content.

[0104] In its specific implementation, the network operation and maintenance interface includes a dialog box where the user inputs second information, and the network operation and maintenance device receives this second information. As an example, the network operation and maintenance interface is as follows: Figure 5 As shown, a dialog box is displayed in the first display area Q1 of the network operation and maintenance interface, where the user enters second content. For example, let's call the preceding operation and maintenance request A operation and maintenance request B. Assume operation and maintenance request B includes an image of the target area X and the text "Why is the network in this area not working well?". After the network operation and maintenance device generates operation and maintenance request B (the implementation method of generating operation and maintenance request B can be referred to the second embodiment in the second case), the network operation and maintenance device processes operation and maintenance request B to obtain a request result matching operation and maintenance request B, and outputs the request result matching operation and maintenance request B. After the network operation and maintenance device outputs the request result matching operation and maintenance request B, the user enters the text "How should I handle this problem?" in the network operation and maintenance interface to continue asking questions. The network operation and maintenance device receives the text "How should I handle this problem?" entered by the user, obtains the image of the target area X based on the preceding content of the text "How should I handle this problem?", and generates operation and maintenance request A based on the image of the target area X and the text "How should I handle this problem?". For example, network maintenance equipment generates maintenance request A based on the image of target area X, the request result matching maintenance request B, and the text "How should I handle this problem".

[0105] It should be noted that the above description of S401 and Figures 6 to 9Taking an image as the first content as an example. If the first content is video, the user can enter the first content in input box 501 and click send control 502 to input the first content to the network maintenance device. Alternatively, the user can input a video file (i.e., the first content) to the network maintenance device through file upload control 505. If the first content is audio, the user can input the first content to the network maintenance device through audio input control 503. Alternatively, the user can input an audio file (i.e., the first content) to the network maintenance device through file upload control 505. Further details are omitted here.

[0106] S402. Based on the first content, use a multimodal large language model to obtain at least one operation and maintenance task.

[0107] The network operation and maintenance device inputs first content into the multimodal large language model, enabling the multimodal large language model to obtain (e.g., generate) at least one operation and maintenance task based on the first content, and outputs the at least one operation and maintenance task. The network operation and maintenance device then obtains the at least one operation and maintenance task output by the multimodal large language model. Specifically, the multimodal large language model is used to: encode the first content to obtain a non-text encoded vector; convert the non-text encoded vector into a semantic vector; and obtain (e.g., generate) the at least one operation and maintenance task based on the semantic vector. In a specific implementation, the multimodal large language model includes a multimodal encoder, a modality converter, and a language sub-model, which are sequentially connected. The multimodal encoder encodes the first content to obtain the non-text encoded vector. The modality converter converts the non-text encoded vector into the semantic vector. The language sub-model obtains the at least one operation and maintenance task based on the semantic vector. Here, the semantics of the semantic vector refers to the semantics that the language sub-model can understand. The language sub-model is used to understand the semantics of the semantic vector and to orchestrate tasks based on the semantics of the semantic vector to obtain at least one operational task. For ease of description, the non-text encoded vector obtained by encoding the first content by the multimodal large language model (specifically, the multimodal encoder) is called the non-text encoded vector P1, and the semantic vector obtained by transforming the non-text encoded vector P1 by the multimodal large language model (specifically, the modality converter) is called the semantic vector Y1. Both the non-text encoded vector P1 and the semantic vector Y1 are vectors of the first content. Both the non-text encoded vector P1 and the semantic vector Y1 are used to represent the first content, and the semantics of the semantic vector Y1 is the semantics expressed by the first content.

[0108] In a specific implementation, the network operation and maintenance device generates a first prompt based on the first content. The first prompt includes the first content. The network operation and maintenance device inputs the first prompt into the multimodal large language model, causing the multimodal large language model to generate at least one operation and maintenance task based on the first prompt. Specifically, the network operation and maintenance device retrieves a first prompt template from a prompt template library based on the first content, and the first prompt template matches the first content. The network operation and maintenance device generates a first prompt based on the first content and the first prompt template. The network operation and maintenance device inputs the first prompt into the multimodal large language model, causing the multimodal large language model to generate at least one operation and maintenance task based on the first prompt. The multimodal large language model is used to: extract the first content from the first prompt, encode the first content to obtain a non-text encoded vector, convert the non-text encoded vector into a semantic vector, and obtain the at least one operation and maintenance task based on the semantic vector.

[0109] In an optional implementation, before the network operation and maintenance device obtains at least one operation and maintenance task using a multimodal large language model based on the first content—for example, before the network operation and maintenance device obtains a first prompt word template from a prompt word template library based on the first content—the network operation and maintenance device performs a legality check on the first content. For example, the network operation and maintenance device checks whether the first content contains sensitive information. If the network operation and maintenance device successfully verifies the first content, it obtains at least one operation and maintenance task based on the first content using a multimodal large language model. Specifically, the network operation and maintenance device includes a verification engine, which it calls to perform a legality check on the first content.

[0110] In an optional implementation, the maintenance request A further includes a second content expressed in text form; that is, maintenance request A includes a first content expressed in non-text form and a second content expressed in text form. The network maintenance device uses a multimodal large language model to obtain at least one maintenance task based on the first and second content. Specifically, the network maintenance device inputs the first and second content into the multimodal large language model, causing the model to obtain (e.g., generate) at least one maintenance task based on the first and second content, and outputs the at least one maintenance task. The network maintenance device then obtains the at least one maintenance task output by the multimodal large language model. The multimodal large language model is used to: encode the first and second content to obtain a multimodal encoding vector; convert the multimodal encoding vector into a semantic vector; and obtain (e.g., generate) the at least one maintenance task based on the semantic vector. In a specific implementation, the multimodal large language model is used to: encode a first piece of content to obtain a non-text encoded vector, encode a second piece of content to obtain a text encoded vector, and obtain (e.g., generate) the multimodal encoded vector based on the non-text encoded vector and the text encoded vector. For example, the multimodal large language model concatenates the non-text encoded vector and the text encoded vector to obtain the multimodal encoded vector. In a specific implementation, the multimodal large language model includes a multimodal encoder, a modality converter, and a language sub-model, which are sequentially connected. The multimodal encoder is used to encode the first and second pieces of content to obtain the multimodal encoded vector. For example, the multimodal encoder is used to: encode the first piece of content to obtain a non-text encoded vector, encode the second piece of content to obtain a text encoded vector, and obtain the multimodal encoded vector based on the non-text encoded vector and the text encoded vector. The modality converter is used to convert the multimodal encoded vector into the semantic vector. The language sub-model is used to obtain the at least one operational task based on the semantic vector. For ease of description, the non-text encoded vector obtained by encoding the first content by the multimodal large language model (specifically, the multimodal encoder) is called the non-text encoded vector P1, the text encoded vector obtained by encoding the second content by the multimodal large language model (specifically, the multimodal encoder) is called the text encoded vector W1, the multimodal encoded vector obtained by the multimodal large language model (specifically, the multimodal encoder) based on the non-text encoded vector P1 and the text encoded vector W1 is called the multimodal encoded vector D1, and the semantic vector obtained by the multimodal large language model (specifically, the modality converter) from the multimodal encoded vector D1 is called the semantic vector Y2.Non-text encoded vector P1 is the vector of the first content, used to represent the first content; text encoded vector W1 is the vector of the second content, used to represent the second content; multimodal encoded vector D1 is the vector of the first content and the second content, used to represent the first content and the second content; semantic vector Y2 is the vector of the first content and the second content, used to represent the first content and the second content, and the semantics of semantic vector Y2 is the semantics expressed by the combination of the first content and the second content.

[0111] In a specific implementation, the network operation and maintenance device generates a first prompt word based on the first content and the second content. The first prompt word includes both the first content and the second content. The network operation and maintenance device inputs the first prompt word into the multimodal large language model, causing the multimodal large language model to generate at least one operation and maintenance task based on the first prompt word. Specifically, the network operation and maintenance device obtains a first prompt word template from a prompt word template library based on the first content and the second content. The first prompt word template matches both the first content and the second content. The network operation and maintenance device generates the first prompt word based on the first content, the second content, and the first prompt word template. The network operation and maintenance device inputs the first prompt word into the multimodal large language model, causing the multimodal large language model to generate at least one operation and maintenance task based on the first prompt word, and outputs the at least one operation and maintenance task. The multimodal large language model is used to: extract the first content and the second content from the first prompt word; encode the first content and the second content to obtain a multimodal encoding vector; convert the multimodal encoding vector into a semantic vector; and obtain the at least one operation and maintenance task based on the semantic vector.

[0112] In an optional implementation, before the network operation and maintenance device obtains at least one operation and maintenance task using a multimodal large language model based on the first and second content, for example, before the network operation and maintenance device obtains the first prompt word template from the prompt word template library based on the first and second content, the network operation and maintenance device performs a legality check on the first and second content. For example, the network operation and maintenance device checks whether the first and second content contain sensitive information. If the network operation and maintenance device successfully verifies the first and second content, it obtains at least one operation and maintenance task using a multimodal large language model based on the first and second content. In a specific implementation, the network operation and maintenance device includes a verification engine, which it calls to perform legality checks on the first and second content.

[0113] In an optional implementation, the at least one operation and maintenance task is a plurality of operation and maintenance tasks, the at least one operation and maintenance task is arranged in sequence, the arrangement order of the at least one operation and maintenance task is the execution order of the at least one operation and maintenance task, and the at least one operation and maintenance task constitutes a task sequence.

[0114] It should be noted that, for ease of description, in this embodiment, both the prompt word template obtained by the network operation and maintenance device based on the first content and the prompt word template obtained by the network operation and maintenance device based on the first content and the second content are referred to as the first prompt word template. Similarly, both the prompt word generated by the network operation and maintenance device based on the first content and the prompt word generated by the network operation and maintenance device based on the first content and the second content are referred to as the first prompt word. Those skilled in the art will readily understand that the first prompt word template obtained by the network operation and maintenance device based on the first content and the first prompt word template obtained by the network operation and maintenance device based on the first content and the second content may be different, and the first prompt word generated by the network operation and maintenance device based on the first content and the second content may be different. This embodiment does not limit this.

[0115] S403. Obtain the request result that matches the operation and maintenance request A based on the at least one operation and maintenance task.

[0116] The network operation and maintenance (O&M) device executes at least one O&M task to obtain a request result matching O&M request A. For example, the at least one O&M task may include a query task, whereby the network O&M device queries the O&M knowledge base to obtain a request result matching O&M request A. As another example, if O&M request A is an O&M issue concerning a target network, the network O&M device obtains network information of the target network based on the at least one O&M task, and then determines a request result matching O&M request A based on the network information. Yet another example is if O&M request A is a configuration request for a target network, where the network O&M device configures the target network based on the at least one O&M task, and then determines a request result matching O&M request A based on the configuration result of the target network. The following three embodiments illustrate how the network O&M device obtains a request result matching O&M request A based on the at least one O&M task.

[0117] First embodiment: The at least one operation and maintenance task includes a query task. The network operation and maintenance device queries the operation and maintenance knowledge base according to the query task to obtain a request result matching operation and maintenance request A. Specifically, the network operation and maintenance device queries the operation and maintenance knowledge base according to the content of the query task and operation and maintenance request A to obtain a request result matching operation and maintenance request A. As mentioned above, operation and maintenance request A includes first content but does not include second content, or operation and maintenance request A includes both first and second content. Depending on the content of operation and maintenance request A, the implementation method of the network operation and maintenance device querying the operation and maintenance knowledge base according to the content of the query task and operation and maintenance request A differs. The implementation methods of the network operation and maintenance device querying the operation and maintenance knowledge base according to the content of the query task and operation and maintenance request A are described below in two cases.

[0118] Scenario 1: Maintenance request A includes the first content but does not include the second content. The network maintenance device queries the maintenance knowledge base based on the query task and the first content to obtain the request result matching maintenance request A. The maintenance knowledge base is used to indicate the association between the first content and the request result, so that the network maintenance device can query the maintenance knowledge base based on the first content to obtain the request result.

[0119] In its specific implementation, the operations and maintenance (O&M) knowledge base includes semantic vectors of knowledge. For ease of description, these semantic vectors are referred to as knowledge vectors. Each knowledge vector in the O&M knowledge base corresponds to a piece of knowledge, which can be knowledge related to network O&M. This knowledge can include O&M experience recorded in network O&M manuals, general O&M knowledge, etc. Each knowledge vector is obtained by encoding the knowledge it corresponds to, or by encoding and transforming the knowledge it corresponds to. For example, the O&M knowledge base includes knowledge vectors corresponding to textual knowledge. The knowledge vector corresponding to each piece of textual knowledge is a text-encoded vector (which is also a semantic vector) obtained by encoding the textual knowledge using a text encoder. As another example, the O&M knowledge base includes knowledge vectors corresponding to non-textual knowledge. The knowledge vector corresponding to each piece of non-textual knowledge is a semantic vector obtained by encoding the non-textual knowledge using a multimodal encoder, then transforming the non-textual encoded vector using a modal converter. For example, the operation and maintenance knowledge base includes knowledge vectors corresponding to multimodal knowledge. Each knowledge vector corresponding to a multimodal knowledge item is a semantic vector obtained by encoding the multimodal knowledge using a multimodal encoder, followed by transformation of the multimodal encoded vector using a modality converter. Textual knowledge is knowledge expressed in textual form, while non-textual knowledge is knowledge expressed in non-textual form. Multimodal knowledge includes knowledge expressed in at least two forms (i.e., at least two modalities). When non-textual knowledge includes knowledge expressed in at least two forms (e.g., including images and audio), this non-textual knowledge is also multimodal knowledge. The text encoder used to encode textual knowledge can be the same as the text encoder in the multimodal large language model. Similarly, the multimodal encoder used to encode non-textual knowledge and the multimodal encoder used to encode multimodal knowledge can be the same as the multimodal encoder in the multimodal large language model. The modality converter used to transform the non-textual encoded vector of non-textual knowledge and the modality converter used to transform the multimodal encoded vector of multimodal knowledge can be the same as the modality converter in the multimodal large language model. In an optional implementation, the operations and maintenance knowledge base also includes the knowledge corresponding to the knowledge vectors. For example, the operations and maintenance knowledge base includes the correspondence between knowledge vectors and knowledge.

[0120] In a specific implementation, the network operation and maintenance device encodes the first content according to the query task to obtain a non-text encoded vector; the network operation and maintenance device converts the non-text encoded vector into a semantic vector; the network operation and maintenance device retrieves a knowledge vector matching the semantic vector from the operation and maintenance knowledge base; the network operation and maintenance device retrieves the knowledge corresponding to the knowledge vector; and the network operation and maintenance device determines the knowledge corresponding to the knowledge vector as the request result matching the operation and maintenance request A. Specifically, the network operation and maintenance device uses a multimodal encoder to encode the first content according to the query task to obtain the non-text encoded vector; the network operation and maintenance device uses a modality converter to convert the non-text encoded vector into a semantic vector; the network operation and maintenance device retrieves the knowledge vector matching the semantic vector from the operation and maintenance knowledge base based on the semantic vector; and the network operation and maintenance device retrieves the knowledge corresponding to the knowledge vector based on the correspondence between the knowledge vector and the knowledge corresponding to the knowledge vector. The multimodal encoder and modality converter can be the same as those in the aforementioned multimodal large language model. For ease of description, the non-text encoded vector obtained by the network operation and maintenance equipment from encoding the first content is called the non-text encoded vector P1. The semantic vector obtained by the network operation and maintenance equipment from transforming the non-text encoded vector P1 is called the semantic vector Y1. The knowledge vector in the operation and maintenance knowledge base that matches the semantic vector Y1 is called the knowledge vector Z1. The knowledge corresponding to the knowledge vector Z1 is called knowledge Z1. Both the non-text encoded vector P1 and the semantic vector Y1 are vectors of the first content. Both the non-text encoded vector P1 and the semantic vector Y1 are used to represent the first content, and the semantics of the semantic vector Y1 is the semantics expressed by the first content. The knowledge vector Z1 is the semantic vector of knowledge Z1. The knowledge vector Z1 is obtained by encoding knowledge Z1, or by encoding and transforming knowledge Z1. The knowledge vector Z1 is used to represent knowledge Z1. For example, if knowledge Z1 is textual knowledge, the knowledge vector Z1 is a text encoded vector (i.e., a semantic vector) obtained by encoding knowledge Z1 using a text encoder. For example, if knowledge Z1 is non-textual knowledge, the knowledge vector Z1 is a semantic vector obtained by encoding knowledge Z1 using a multimodal encoder to obtain a non-textual encoded vector, and then transforming this non-textual encoded vector using a modal converter. Alternatively, if knowledge Z1 is multimodal knowledge, the knowledge vector Z1 is a semantic vector obtained by encoding knowledge Z1 using a multimodal encoder to obtain a multimodal encoded vector, and then transforming this multimodal encoded vector using a modal converter.

[0121] In the specific implementation, the network operation and maintenance device obtains the distance between semantic vector Y1 and each knowledge vector in the operation and maintenance knowledge base. Based on this distance, the network operation and maintenance device obtains the knowledge vector Z1 that matches semantic vector Y1. The distance between semantic vector Y1 and each knowledge vector in the operation and maintenance knowledge base is used to characterize the correlation between the first content and the knowledge corresponding to that knowledge vector. For example, the knowledge vector Z1 that matches semantic vector Y1 is the knowledge vector in the operation and maintenance knowledge base with the smallest distance to semantic vector Y1. That is, the network operation and maintenance device determines the knowledge vector with the smallest distance to semantic vector Y1 in the operation and maintenance knowledge base as the knowledge vector Z1 that matches semantic vector Y1, and obtains the knowledge vector Z1 that matches semantic vector Y1 from the operation and maintenance knowledge base.

[0122] In an optional implementation, the network operations and maintenance (O&M) device obtains the distance between semantic vector Y1 and each knowledge vector in the O&M knowledge base. The O&M device then scores each knowledge vector based on this distance. Finally, based on these scores, the O&M device obtains the knowledge vector Z1 that matches semantic vector Y1. Knowledge vectors with smaller distances to semantic vector Y1 receive higher scores. For example, the knowledge vector Z1 that matches semantic vector Y1 is the highest-scoring knowledge vector in the O&M knowledge base. In other words, the O&M device identifies the highest-scoring knowledge vector in the O&M knowledge base as the knowledge vector Z1 that matches semantic vector Y1, and retrieves this matching knowledge vector from the O&M knowledge base.

[0123] In an optional implementation, the network operations and maintenance (O&M) device queries the O&M knowledge base using retrieval enhancement technology based on the query task and the first content to obtain request results matching the O&M request A. For example, the network O&M device retrieves the O&M knowledge base using retrieval enhancement technology based on the semantic vector Y1 to obtain the knowledge vector Z1 matching the semantic vector Y1 from the O&M knowledge base.

[0124] The second scenario: Maintenance request A includes a first element and a second element. The network maintenance device queries the maintenance knowledge base based on the query task, the first element, and the second element to obtain the request result matching maintenance request A. The maintenance knowledge base is used to indicate the association between the first element, the second element, and the request result, so that the network maintenance device can query the maintenance knowledge base based on the first element and the second element to obtain the request result.

[0125] For an introduction to the operation and maintenance knowledge base, please refer to the first scenario mentioned above. This section mainly describes the implementation method of the network operation and maintenance device querying the operation and maintenance knowledge base based on the query task, the first content, and the second content. Specifically, the network operation and maintenance device encodes the first and second content according to the query task to obtain a multimodal encoding vector; the network operation and maintenance device converts this multimodal encoding vector into a semantic vector; the network operation and maintenance device retrieves a knowledge vector matching the semantic vector from the operation and maintenance knowledge base; the network operation and maintenance device retrieves the knowledge corresponding to the knowledge vector; and the network operation and maintenance device determines the knowledge corresponding to the knowledge vector as the request result matching operation and maintenance request A. Specifically, the network operation and maintenance device, according to the query task, uses a multimodal encoder to encode the first and second content to obtain the multimodal encoding vector; the network operation and maintenance device uses a modality converter to convert the multimodal encoding vector into a semantic vector; the network operation and maintenance device retrieves the operation and maintenance knowledge base based on the semantic vector to obtain a knowledge vector matching the semantic vector; and the network operation and maintenance device retrieves the knowledge corresponding to the knowledge vector based on the correspondence between the knowledge vector and the knowledge corresponding to the knowledge vector. The multimodal encoder and modality converter described above can be the same as those in the multimodal large language model. Specifically, the network operation and maintenance device encodes the first content using the multimodal encoder to obtain a non-text encoded vector based on the query task; the network operation and maintenance device then encodes the second content using the same multimodal encoder to obtain a text encoded vector based on the query task; finally, the network operation and maintenance device obtains (e.g., generates) the multimodal encoded vector based on the non-text encoded vector and the text encoded vector. For ease of description, the non-text encoded vector obtained by the network operation and maintenance equipment from encoding the first content is called non-text encoded vector P1; the text encoded vector obtained by the network operation and maintenance equipment from encoding the second content is called text encoded vector W1; the multimodal encoded vector obtained by the network operation and maintenance equipment based on non-text encoded vector P1 and text encoded vector W1 is called multimodal encoded vector D1; the semantic vector obtained by the network operation and maintenance equipment from multimodal encoded vector D1 is called semantic vector Y2; the knowledge vector in the operation and maintenance knowledge base that matches semantic vector Y2 is called knowledge vector Z2; and the knowledge corresponding to knowledge vector Z2 is called knowledge Z2. Non-text encoded vector P1 is the vector of the first content and is used to represent the first content. Text encoded vector W1 is the vector of the second content and is used to represent the second content. Multimodal encoded vector D1 is the vector of the first content and the second content and is used to represent both. Semantic vector Y2 is a vector of the first content and the second content. Semantic vector Y2 is used to represent the first content and the second content. The semantics of semantic vector Y2 is the semantics expressed by the combination of the first content and the second content.Knowledge vector Z2 is the semantic vector of knowledge Z2. Knowledge vector Z2 is obtained by encoding knowledge Z2, or by encoding and transforming knowledge Z2. Knowledge vector Z2 is used to represent knowledge Z2. For example, if knowledge Z2 is textual knowledge, knowledge vector Z2 is a text-encoded vector (i.e., a semantic vector) obtained by encoding knowledge Z2 using a text encoder. As another example, if knowledge Z2 is non-textual knowledge, knowledge vector Z2 is a semantic vector obtained by encoding knowledge Z2 into a non-textual encoded vector using a multimodal encoder, and then transforming that non-textual encoded vector using a modality converter. Yet another example is multimodal knowledge, knowledge vector Z2 is a semantic vector obtained by encoding knowledge Z2 into a multimodal encoded vector using a multimodal encoder, and then transforming that multimodal encoded vector using a modality converter.

[0126] In the specific implementation, the network operation and maintenance device obtains the distance between semantic vector Y2 and each knowledge vector in the operation and maintenance knowledge base. Based on this distance, the network operation and maintenance device obtains the knowledge vector Z2 that matches semantic vector Y2. The distance between semantic vector Y2 and each knowledge vector in the operation and maintenance knowledge base is used to characterize the correlation between the first content and the knowledge corresponding to that knowledge vector. For example, the knowledge vector Z2 that matches semantic vector Y2 is the knowledge vector in the operation and maintenance knowledge base with the smallest distance to semantic vector Y2. That is, the network operation and maintenance device determines the knowledge vector with the smallest distance to semantic vector Y2 in the operation and maintenance knowledge base as the knowledge vector Z2 that matches semantic vector Y2, and obtains the knowledge vector Z2 that matches semantic vector Y2 from the operation and maintenance knowledge base.

[0127] In an optional implementation, the network operations and maintenance (O&M) device obtains the distance between semantic vector Y2 and each knowledge vector in the O&M knowledge base. The O&M device then scores each knowledge vector based on this distance. Finally, based on these scores, the O&M device obtains the knowledge vector Z2 that matches semantic vector Y2. Knowledge vectors with smaller distances to semantic vector Y2 receive higher scores. For example, the knowledge vector Z2 that matches semantic vector Y2 is the highest-scoring knowledge vector in the O&M knowledge base. In other words, the O&M device identifies the highest-scoring knowledge vector in the O&M knowledge base as the knowledge vector Z2 that matches semantic vector Y2, and retrieves this matching knowledge vector from the O&M knowledge base.

[0128] In an optional implementation, the network operation and maintenance device queries the operation and maintenance knowledge base using retrieval enhancement technology based on the query task, the first content, and the second content to obtain request results matching the operation and maintenance request A. For example, the network operation and maintenance device retrieves the operation and maintenance knowledge base using retrieval enhancement technology based on semantic vector Y2 to obtain knowledge vector Z1 matching semantic vector Y2 from the operation and maintenance knowledge base.

[0129] In an optional implementation, before querying the maintenance knowledge base according to the query task, the network maintenance device obtains the maintenance knowledge base. For example, the network maintenance device builds the maintenance knowledge base, or the network maintenance device obtains the maintenance knowledge base built by other devices. This application embodiment uses the construction of the maintenance knowledge base by the network maintenance device as an example. The network maintenance device obtains at least one piece of knowledge, and the network maintenance device constructs the maintenance knowledge base based on the at least one piece of knowledge. The at least one piece of knowledge is knowledge related to network maintenance, and may include textual knowledge, non-textual knowledge, multimodal knowledge, etc., which are not limited in this application embodiment.

[0130] In a specific implementation, the operation and maintenance knowledge base includes knowledge vectors. Each knowledge vector in the knowledge base corresponds to a piece of knowledge. Each knowledge vector is obtained by encoding the knowledge it corresponds to, or by encoding and transforming the knowledge it corresponds to. The network operation and maintenance device obtains the knowledge vector corresponding to each of the at least one piece of knowledge to obtain at least one knowledge vector corresponding to the at least one piece of knowledge. The network operation and maintenance device constructs the operation and maintenance knowledge base based on the at least one knowledge vector. In a specific implementation, for each piece of text knowledge in the at least one piece of knowledge: the network operation and maintenance device uses a text encoder to encode the text knowledge to obtain the text encoding vector (i.e., semantic vector) corresponding to the text knowledge. The network operation and maintenance device determines the text encoding vector corresponding to the text knowledge as the knowledge vector corresponding to the text knowledge. For each non-textual knowledge item in the at least one knowledge item: the network operation and maintenance device uses a multimodal encoder to encode the non-textual knowledge to obtain a non-textual encoding vector corresponding to the non-textual knowledge. The network operation and maintenance device then uses a modality converter to transform the non-textual encoding vector to obtain a semantic vector corresponding to the non-textual knowledge. The network operation and maintenance device then determines the semantic vector corresponding to the non-textual knowledge as the knowledge vector corresponding to the non-textual knowledge. For each multimodal knowledge item in the at least one knowledge item: the network operation and maintenance device uses a multimodal encoder to encode the multimodal knowledge to obtain a multimodal encoding vector corresponding to the multimodal knowledge. The network operation and maintenance device then uses a modality converter to transform the multimodal encoding vector to obtain a semantic vector corresponding to the multimodal knowledge. The network operation and maintenance device then determines the semantic vector corresponding to the multimodal knowledge as the knowledge vector corresponding to the multimodal knowledge. In an optional implementation, the operation and maintenance knowledge base includes a correspondence between knowledge vectors and knowledge. The network operation and maintenance device constructs the operation and maintenance knowledge base based on the at least one knowledge item and the at least one knowledge vector corresponding to the at least one knowledge item. For example, network operation and maintenance equipment establishes a correspondence between at least one piece of knowledge and at least one knowledge vector.

[0131] In this embodiment of the application, when constructing the operation and maintenance knowledge base, there is no need to convert knowledge expressed in non-textual form (e.g., non-textual knowledge, multimodal knowledge) into knowledge expressed in textual form (i.e., textual knowledge). Therefore, the process of constructing this operation and maintenance knowledge base is simple and highly efficient. Furthermore, it avoids the problem of losing important information due to knowledge format conversion, thus enriching the knowledge in the operation and maintenance knowledge base. The operation and maintenance knowledge base constructed in this embodiment of the application can cover knowledge in many forms of expression, and has a wide range of knowledge coverage.

[0132] The second embodiment: Operation and maintenance request A is an operation and maintenance problem for the target network. The at least one operation and maintenance task is a task used to obtain network information of the target network. The network operation and maintenance device obtains the network information of the target network according to the at least one operation and maintenance task, and determines the request result that matches operation and maintenance request A according to the network information of the target network.

[0133] In this second embodiment, the first content included in the maintenance request A is content targeting the target network.

[0134] In one example, the first content is an image of a display interface provided by the target device, which is a network management device, a network maintenance device, or a network device within the target network. For instance, if the target device is a network maintenance device, the display interface provided by the target device is a network maintenance interface for the target network, displaying a device distribution map of the target network. The first content is an image of a target area within this display interface, which includes at least a portion of the device distribution map. The target area displays target network devices within the target network, which are devices in an abnormal state within the target network. This abnormal state includes at least one of the following: fault state, alarm state, offline state, or unregistered state. For example, the display interface is as follows: Figure 7 or Figure 9 The network operation and maintenance interface shown has a device distribution map of the target network in the second display area Q2. The first content is the image of the target area X in the display interface. The target area displays AP7 in the target network. AP7 is a device in the target network that is in an abnormal state.

[0135] In another example, the first content is a screenshot of an issue form showing the problem occurring on the target network. For example, the first content is as follows: Figure 6 Image 1, as shown, displays the message "Lobby AP7 malfunction, please diagnose the cause," indicating a problem in the target network. For example, the first content is as follows... Figure 8 Image 1 shown in the diagram, which displays a "Lobby AP7 Failure," represents a problem occurring in the target network.

[0136] In another example, the first content is a video or audio recording of a problem occurring on the target network.

[0137] In an optional implementation, the maintenance request A further includes second content. In this second embodiment, the second content is also content specific to the target network. For example, the second content is a textual description related to the first content. Since the first content is content specific to the target network, the second content is also content specific to the target network. Figure 9As shown, the first content is an image of the target region X, and the second content is the text "Why is the network in this region poor?" For example... Figure 8 As shown, the first content is image 1, and the second content is the text "Please diagnose the cause".

[0138] In a specific implementation, the network operation and maintenance (O&M) device executes at least one O&M task to obtain network information of the target network. Based on this network information, the O&M device determines the request result that matches O&M request A. For example, if O&M request A represents a problem occurring in the target network, the O&M device determines the root cause of this problem based on the target network information. Alternatively, the O&M device may perform diagnostic analysis on the target network based on the network information to determine the root cause of the O&M problem.

[0139] In an optional implementation, the at least one maintenance task is arranged sequentially, and the order of the at least one maintenance task is the execution order of the at least one maintenance task. The network maintenance device executes the at least one maintenance task according to the order of the at least one maintenance task to obtain network information of the target network. Specifically, each of the at least one maintenance task includes an interface identifier and interface call parameters. The interface identifier of each maintenance task indicates an interface (e.g., an application program interface, API), and the interface call parameters of each maintenance task are the call parameters of the interface indicated by the interface identifier. For each of the at least one maintenance task: the network maintenance device calls the interface indicated by the interface identifier of the maintenance task according to the interface call parameters to obtain network information of the target network. For example, the first maintenance task includes an interface identifier of a first interface and interface call parameters of the first interface. The network maintenance device calls the first interface according to the interface call parameters of the first interface to obtain network information of the target network.

[0140] The third embodiment: Operation and maintenance request A is a configuration request for the target network. The at least one operation and maintenance task is a task used to configure the target network. The network operation and maintenance device configures the target network according to the at least one operation and maintenance task. The network operation and maintenance device determines the request result that matches operation and maintenance request A based on the configuration result of the target network.

[0141] In this second embodiment, the first content included in the maintenance request A is content related to the target network. For example, the first content is an image of a display interface provided by the target device, which is the network management device, network maintenance device, or network device within the target network. Another example is a screenshot of a problem ticket, showing a problem occurring on the target network. Yet another example is a video or audio recording related to a problem occurring on the target network.

[0142] In a specific implementation, the network operation and maintenance device executes at least one operation and maintenance task to configure the target network. Based on the configuration result of the target network, the network operation and maintenance device determines the request result that matches operation and maintenance request A. For example, the network operation and maintenance device determines the configuration result of the target network as the request result that matches operation and maintenance request A.

[0143] In an optional implementation, the at least one maintenance task is arranged sequentially, and the order of the at least one maintenance task is the execution order of the at least one maintenance task. The network maintenance device executes the at least one maintenance task according to the order of the at least one maintenance task to configure the target network. Specifically, each of the at least one maintenance task includes an interface identifier and interface call parameters. The interface identifier in each maintenance task indicates an interface, and the interface call parameters in each maintenance task are the call parameters of the interface indicated by the interface identifier. For each of the at least one maintenance task: the network maintenance device calls the interface indicated by the interface identifier in the maintenance task according to the interface call parameters to configure the target network.

[0144] It should be noted that, Figures 5 to 9 as well as Figures 11 to 12 Taking the second display area Q2 of the network operation and maintenance interface, which displays a device distribution diagram of the target network, as an example, in practical applications, the second display area Q2 can display any possible information, such as charts related to network operation and maintenance. Furthermore, Figures 5 to 9 as well as Figures 11 to 12 The dialog box in the first display area Q1 shown is only an example. The first display area Q1 can display any possible information, components, etc. related to the dialog, and this application embodiment does not limit it.

[0145] In summary, the technical solution provided in this application embodiment involves a network operation and maintenance device acquiring an operation and maintenance request that includes first content expressed in a non-textual form. Based on the first content, it uses a multimodal large language model to obtain at least one operation and maintenance task, and then obtains a request result matching the operation and maintenance request based on the at least one operation and maintenance task. It is evident that the multimodal large language model used in this application embodiment can understand and process the non-textual content included in the operation and maintenance request. That is, the multimodal large language model can understand and process operation and maintenance requests that include non-textual content. Therefore, using this multimodal large language model to process operation and maintenance requests offers high flexibility, contributing to improved network operation and maintenance flexibility. This application embodiment eliminates the need to convert non-textual information into textual information, thus solving problems such as low efficiency, complex operation, and low accuracy caused by information format conversion. It simplifies the user's operation of triggering operation and maintenance requests and improves the efficiency and accuracy of the network operation and maintenance device in processing operation and maintenance requests.

[0146] After obtaining the request result matching maintenance request A, the network maintenance device can output the request result matching maintenance request A. In this embodiment, the network maintenance device can verify the request result matching maintenance request A. If the verification of the request result matching maintenance request A is successful, the network maintenance device outputs the request result matching maintenance request A.

[0147] As an example, please refer to Figure 10 This illustrates a flowchart of another network operation and maintenance method provided in an embodiment of this application. For example... Figure 10 As shown, in Figure 4 Based on the illustrated embodiment, the network operation and maintenance method further includes the following steps S404 to S405.

[0148] S404. Use a multimodal large language model to verify the request results that match the operation and maintenance request A.

[0149] For ease of description, the request result that matches the operation and maintenance request A will be referred to as request result A1.

[0150] The network operation and maintenance equipment inputs the request result A1 into a multimodal large language model, which then verifies the request result A1 and outputs the verification result. The verification result can be either successful or unsuccessful. Specifically, the network operation and maintenance equipment generates a second prompt word based on the request result A1. This second prompt word is a verification prompt word used to request the verification of the request result A1. The second prompt word includes the request result A1. The network operation and maintenance equipment inputs the second prompt word into the multimodal large language model, which then verifies the request result A1 included in the second prompt word and outputs the verification result.

[0151] In an optional implementation, the network operation and maintenance (O&M) device verifies the request result A1 using a multimodal large language model based on the content of O&M request A. The network O&M device inputs the content of O&M request A and request result A1 into the multimodal large language model, which then verifies request result A1 based on the content of O&M request A and outputs the verification result. For example, the network O&M device generates a second prompt word based on the content of O&M request A and request result A1. The second prompt word is a verification prompt word used to request the verification of request result A1, and includes the content of O&M request A and request result A1. The network O&M device inputs the second prompt word into the multimodal large language model, which then verifies request result A1 based on the content of O&M request A included in the second prompt word and outputs the verification result. The multimodal large language model is used to: obtain at least one verification task based on the content of O&M request A and request result A1; and verify request result A1 based on the at least one verification task. For example, the multimodal large language model is used to: invoke a simulation tool to obtain reference materials based on the at least one verification task, and verify the request result A1 based on the reference materials. Specifically, the multimodal large language model is used to: verify whether the request result A1 matches the reference materials; if the request result A1 matches the reference materials, output information indicating successful verification; if the request result A1 does not match the reference materials, output information indicating verification failure. In one embodiment, the at least one verification task includes a query task, and the multimodal large language model is used to: invoke a simulation tool to query the operation and maintenance knowledge base to obtain reference materials based on the query task. For another example, if the request result A1 is an operation and maintenance problem or a configuration request for the target network, the multimodal large language model is used to: invoke a simulation tool to obtain reference materials based on the knowledge graph of the target network based on the at least one verification task.

[0152] In an optional implementation, the network operation and maintenance device (O&M) performs an execution process of obtaining request result A1 based on the content of O&M request A, and then uses a multimodal large language model to verify request result A1. The execution process of obtaining request result A1 based on the content of O&M request A may include: the process of obtaining at least one O&M task based on the content of O&M request A using a multimodal large language model, and the process of obtaining request result A1 based on the at least one O&M task (i.e., the process of executing the at least one O&M task). Specifically, the O&M device obtains execution process information describing the process of obtaining request result A1 based on the content of O&M request A. The O&M device inputs the content of O&M request A, the execution process information, and request result A1 into the multimodal large language model, causing the multimodal large language model to verify request result A1 based on the content of O&M request A and the execution process information, and outputs the verification result. For example, a network operation and maintenance (O&M) device generates a second prompt word based on the content of O&M request A, the execution process information, and the request result A1. This second prompt word is a verification prompt word used to request verification of the request result A1. The second prompt word includes the content of O&M request A, the execution process information, and the request result A1. The network O&M device inputs the second prompt word into a multimodal large language model, which then verifies the request result A1 based on the content of O&M request A and the execution process information included in the second prompt word, and outputs the verification result. Specifically, the multimodal large language model is used to: obtain at least one verification task based on the content of O&M request A, the execution process information, and the request result A1; and verify the request result A1 based on the at least one verification task. For example, the multimodal large language model is used to: call a simulation tool to obtain reference materials based on the at least one verification task, and verify the request result A1 based on the reference materials.

[0153] As mentioned earlier, maintenance request A includes the first content but does not include the second content, or maintenance request A includes both the first and second content. Depending on the content of maintenance request A, the following describes the implementation methods for network maintenance equipment to verify request result A1 in two different scenarios.

[0154] The first scenario: Maintenance request A includes a first element but excludes a second element. The network maintenance device verifies the request result A1 using the multimodal large language model based on the first element. The network maintenance device inputs the first element and request result A1 into the multimodal large language model, which then verifies request result A1 based on the first element and outputs the verification result. For example, the network maintenance device generates a second prompt word based on the first element and request result A1. This second prompt word is a verification prompt word used to request verification of request result A1, and it includes both the first element and request result A1. The network maintenance device inputs the second prompt word into the multimodal large language model, which then verifies request result A1 based on the first element included in the second prompt word and outputs the verification result. The multimodal large language model is used to: obtain at least one verification task based on the first element and request result A1; obtain reference materials based on the at least one verification task; and verify request result A1 based on the reference materials. For example, the multimodal large language model is used to: call a simulation tool to obtain the reference material according to the at least one verification task; verify whether the verification request result A1 matches the reference material; if the request result A1 matches the reference material, output information indicating successful verification; if the reference material does not match the request result A1, output information indicating failed verification.

[0155] In an optional implementation, the network operation and maintenance device verifies the request result A1 using the multimodal large language model based on the first content and the execution process of obtaining the request result A1 based on the first content. The network operation and maintenance device obtains execution process information describing the execution process of obtaining the request result A1 based on the first content. The network operation and maintenance device inputs the first content, the execution process information, and the request result A1 into the multimodal large language model, enabling the multimodal large language model to verify the request result A1 based on the first content and the execution process information, and outputs the verification result. For example, the network operation and maintenance equipment generates a second prompt word based on the first content, the execution process information, and the request result A1. The second prompt word is a verification prompt word used to request verification of the request result A1. The second prompt word includes the first content, the execution process information, and the request result A1. The network operation and maintenance equipment inputs the second prompt word into the multimodal large language model, causing the multimodal large language model to verify the request result A1 included in the second prompt word based on the first content and the execution process information, and output the verification result. Specifically, the multimodal large language model is used to: obtain at least one verification task based on the first content, the execution process information, and the request result A1; obtain reference materials based on the at least one verification task; and verify the request result A1 based on the reference materials. For example, the multimodal large language model is used to: call a simulation tool to obtain the reference materials based on the at least one verification task; verify whether the request result A1 matches the reference materials; if the request result A1 matches the reference materials, output information indicating successful verification; if the reference materials do not match the request result A1, output information indicating verification failure.

[0156] The multimodal large language model obtains at least one verification task based on the first content and the request result A1, including: encoding the first content and the request result A1 to obtain an encoding vector (which can be a non-text encoding vector or a multimodal encoding vector, depending on the expression form of the first content and the expression form of the request result A1); converting the encoding vector into a semantic vector; and obtaining the at least one verification task based on the semantic vector. Alternatively, the multimodal large language model obtains at least one verification task based on the first content, execution process information, and the request result A1, including: encoding the first content, the execution process information, and the request result A1 to obtain an encoding vector (which can be a non-text encoding vector or a multimodal encoding vector, depending on the expression form of the first content, the expression form of the execution process information, and the expression form of the request result A1); converting the encoding vector into a semantic vector; and obtaining the at least one verification task based on the semantic vector. The multimodal large language model obtains the specific implementation process of at least one verification task based on the first content and the request result A1. The multimodal large language model obtains the specific implementation process of at least one verification task based on the first content, the execution process information and the request result A1. The implementation process of the multimodal large language model obtaining the above-mentioned at least one operation and maintenance task based on the first content in the foregoing embodiment can be referred to, and will not be elaborated here.

[0157] The second scenario: Maintenance request A includes a first content and a second content. The network maintenance device uses the multimodal large language model to verify the request result A1 based on the first and second content. The network maintenance device inputs the first content, the second content, and the request result A1 into the multimodal large language model, which then verifies the request result A1 based on the first and second content and outputs the verification result. For example, the network maintenance device generates a second prompt word based on the first content, the second content, and the request result A1. The second prompt word is a verification prompt word used to request the verification of the request result A1. The second prompt word includes the first content, the second content, and the request result A1. The network maintenance device inputs the second prompt word into the multimodal large language model, which then verifies the request result A1 included in the second prompt word based on the first and second content included in the second prompt word and outputs the verification result. The multimodal large language model is used to: obtain at least one verification task based on the first content, the second content, and the request result A1; obtain reference materials based on the at least one verification task; and verify the request result A1 based on the reference materials. For example, the multimodal large language model is used to: call a simulation tool to obtain the reference material according to the at least one verification task; verify whether the verification request result A1 matches the reference material; if the request result A1 matches the reference material, output information indicating successful verification; if the request result A1 does not match the reference material, output information indicating failed verification.

[0158] In an optional implementation, the network operation and maintenance device verifies the request result A1 using the multimodal large language model based on the first content, the second content, and the execution process of obtaining the request result A1 based on the first content and the second content. The network operation and maintenance device obtains execution process information describing the process by which it obtains the request result A1 based on the first content and the second content. The network operation and maintenance device inputs the first content, the second content, the execution process information, and the request result A1 into the multimodal large language model, causing the multimodal large language model to verify the request result A1 based on the first content, the second content, and the execution process information, and output the verification result. For example, a network maintenance device generates a second prompt word based on the first content, the second content, the execution process information, and the request result A1. The second prompt word is a verification prompt word used to request verification of the request result A1. The second prompt word includes the first content, the second content, the execution process information, and the request result A1. The network maintenance device inputs the second prompt word into the multimodal large language model, causing the multimodal large language model to verify the request result A1 included in the second prompt word based on the first content, the second content, and the execution process information, and output the verification result. Specifically, the multimodal large language model is used to: obtain at least one verification task based on the first content, the second content, the execution process information, and the request result A1; obtain reference materials based on the at least one verification task; and verify the request result A1 based on the reference materials. For example, the multimodal large language model is used to: call a simulation tool to obtain the reference material according to the at least one verification task; verify whether the verification request result A1 matches the reference material; if the request result A1 matches the reference material, output information indicating successful verification; if the reference material does not match the request result A1, output information indicating failed verification.

[0159] The multimodal large language model obtains at least one verification task based on the first content, the second content, and the request result A1, including: encoding the first content, the second content, and the request result A1 to obtain a multimodal encoding vector; converting the multimodal encoding vector into a semantic vector; and obtaining the at least one verification task based on the semantic vector. Alternatively, the multimodal large language model obtains at least one verification task based on the first content, the second content, execution process information, and the request result A1, including: encoding the first content, the second content, execution process information, and the request result A1 to obtain a multimodal encoding vector; converting the multimodal encoding vector into a semantic vector; and obtaining the at least one verification task based on the semantic vector. The multimodal large language model obtains the specific implementation process of at least one verification task based on the first content, the second content, and the request result A1. The specific implementation process of the multimodal large language model obtaining the specific implementation process of at least one verification task based on the first content, the second content, the execution process information, and the request result A1 can be referred to in the foregoing embodiment. The implementation process of the multimodal large language model obtaining the above-mentioned at least one operation and maintenance task based on the first content and the second content will not be elaborated here.

[0160] S405. If the verification of the request result matching the operation and maintenance request A is successful, output the request result matching the operation and maintenance request A.

[0161] The request result mentioned in S405 refers to the aforementioned request result A1.

[0162] In an optional implementation, the network operation and maintenance device presents the request result A1. In one embodiment, the network operation and maintenance device has a voice component, through which it broadcasts the request result A1. In another embodiment, the network operation and maintenance device has a display component, through which it displays the request result A1. For example, the network operation and maintenance device displays the request result A1 in a dialog box within the network operation and maintenance interface. Alternatively, the network operation and maintenance device may display the request result A1 in the dialog box within the network operation and maintenance interface in the form of a graph, table, or text.

[0163] In one optional implementation, the network operation and maintenance device uses a multimodal large language model to summarize the request result A1 to obtain the summary information of the request result A1, and then outputs the summary information of the request result A1. In another optional implementation, the network operation and maintenance device generates a third prompt word based on the request result A1. The third prompt word is a summary prompt word used to request a summary of the request result A1, and includes the request result A1. The network operation and maintenance device inputs the third prompt word into the multimodal large language model, which summarizes the request result A1 based on the third prompt word to obtain the summary information of the request result A1, and then outputs the summary information of the request result A1.

[0164] As an example, maintenance request A includes a first part and a second part, such as... Figure 9 As shown, the first content is an image of the target area X, and the second content is the text "Why is the network in this area poor?" Network maintenance equipment is used in... Figure 9 After the network operations and maintenance interface displays the request result A1 that matches the operations and maintenance request A, the interface changes as follows: Figure 11 The network operation and maintenance interface shown. Figure 11 As shown, the dialog box in the first display area Q1 of the network operation and maintenance interface displays the request result A1. Based on request result A1, it can be determined that there are three possible reasons why the network in target area X is poor (i.e., AP7's network is poor). Reason 1: There is an illegal device interference source near AP7. Illegal device interference sources may include illegal APs and interfering APs. Reason 2: There is an external interference source near AP7. External interference sources can be non-Wi-Fi interference sources, such as Bluetooth devices or microwave ovens. Reason 3: AP7 is experiencing interference from neighboring devices operating on the same frequency. Among the optional implementation methods, such as... Figure 11 As shown, the network operation and maintenance interface also displays a diagnostic control corresponding to each cause. Figure 11 In the "Diagnosis" section, users can click the diagnostic control corresponding to any cause to trigger a diagnostic command for further diagnosis of that cause. The network maintenance equipment can then perform further diagnosis based on this command. For example, if a user clicks the diagnostic control corresponding to cause 1, triggering a diagnostic command for cause 1, the network maintenance equipment will diagnose cause 1 according to the command and output the diagnostic result. After the network maintenance equipment displays the diagnostic result for cause 1 in the dialog box of the aforementioned network maintenance interface, the network maintenance interface changes as follows: Figure 12 The network operations and maintenance interface shown. See also... Figure 12It can be concluded that the diagnosis for cause 1 is: there is one illegitimate AP and one interfering AP in the network (i.e., the target network). The reason for the poor network condition in target area X (i.e., the poor network condition of AP7) may be: the presence of one illegitimate AP and one interfering AP in the target network. Whether this illegitimate AP and the interfering AP are near AP7 needs to be confirmed on-site. In other words, whether the specific cause of the poor network condition in target area X (i.e., the poor network condition of AP7) is caused by this illegitimate AP and the interfering AP needs to be confirmed on-site.

[0165] It should be noted that, in this embodiment, verifying request result A1 refers to performing a factual consistency check on request result A1. By verifying request result A1, the consistency between request result A1 and the facts can be determined. If the verification of request result A1 is successful, it indicates that the factual consistency of request result A1 is strong (i.e., the consistency between request result A1 and the facts is strong). If the verification of request result A1 fails, it indicates that the factual consistency of request result A1 is weak (i.e., the consistency between request result A1 and the facts is weak). This embodiment outputs request result A1 when the verification is successful, that is, it outputs a request result with strong factual consistency, which can improve the reliability of the output request result A1, avoid outputting a request result A1 with weak factual consistency, and eliminate illusions.

[0166] Furthermore, this embodiment of the application illustrates the example of a network operation and maintenance device successfully verifying request result A1. If the network operation and maintenance device fails to verify request result A1, it executes S402 to S403 again to obtain a request result matching the operation and maintenance request A and verifies the re-obtained request result. If the verification of the re-obtained request result is successful, the network operation and maintenance device outputs the request result. If the network operation and maintenance device fails to verify the request result obtained by executing S402 to S403 multiple times consecutively (e.g., twice), the network operation and maintenance device outputs the request result obtained from any single execution of S402 to S403. For example, the network operation and maintenance device outputs the request result obtained from the first execution of S401 to S403, or the request result obtained from the last and first execution of S401 to S403.

[0167] The multimodal language models involved in the above embodiments are pre-trained. These models can be trained by network maintenance equipment or by other devices (such as AI servers). This application uses the training of the multimodal language model by network maintenance equipment as an example. Please continue to refer to... Figure 10 The network operation and maintenance method also includes the following step S406. The network operation and maintenance equipment can execute S406 before executing S402.

[0168] S406. Train a multimodal large language model based on training data, which is data related to network operation and maintenance, and includes content expressed in non-textual form.

[0169] The network operations and maintenance (O&M) equipment acquires the training data and then trains the multimodal large language model based on it. Specifically, the O&M equipment inputs the training data into the multimodal large language model, enabling the model to perform inference calculations based on the training data and output inference data. This output inference data is the data obtained by the multimodal large language model through inference calculations based on the input data (e.g., the training data). The O&M equipment adjusts the parameters of the multimodal large language model based on the inference data output by the model from the training data to train the model. In essence, the inference data output by the multimodal large language model based on the training data is the data obtained and output after the model performs inference calculations based on the training data.

[0170] In an optional implementation, the network operation and maintenance equipment adjusts the parameters of the multimodal large language model (MLM) to train the MLM based on the labeled data corresponding to the training data and the inference data output by the MLM based on the training data. The labeled data corresponding to the training data is data obtained by labeling the training data, and it is the data that the MLM is expected to output after performing inference calculations based on the training data. The labeled data can be labeled manually or by a labeling device.

[0171] In the specific implementation, the network operation and maintenance equipment inputs the training data into the multimodal large language model, enabling the model to perform inference calculations based on the training data and output inference data. The network operation and maintenance equipment adjusts the parameters of the multimodal large language model based on the difference between the labeled data corresponding to the training data and the inference data output by the model. The network operation and maintenance equipment then inputs the adjusted training data into the multimodal large language model, enabling it to perform inference calculations based on the training data and output inference data. The network operation and maintenance equipment again adjusts the parameters of the multimodal large language model based on the difference between the labeled data corresponding to the training data and the inference data output by the model. This process of inputting training data into the multimodal large language model and adjusting its parameters is repeated to train the model until preset conditions are met. Each iteration of this process constitutes one iteration. The preset conditions include at least one of the following: the difference between the inference data output by the multimodal large language model based on the training data and the labeled data corresponding to the training data is less than a first preset difference; the number of training iterations (i.e., the number of iterations in the training process) reaches a preset number; and the difference between the inference data output by the multimodal large language model based on the training data on multiple consecutive occasions is less than a second preset difference. The second preset difference may be equal to or unequal to the first preset difference, and this embodiment does not limit this.

[0172] In an optional implementation, the multimodal large language model includes a multimodal encoder, a modality transducer, and a language sub-model, which are sequentially connected. The network operation and maintenance device trains the modality transducer in the multimodal large language model based on the training data. The aforementioned adjustment of the parameters of the multimodal large language model specifically refers to adjusting the parameters of the modality transducer. In an optional implementation, the network operation and maintenance device freezes the multimodal encoder and the language sub-model (i.e., does not adjust the parameters of the multimodal encoder and the language sub-model), and adjusts the parameters of the modality transducer.

[0173] In a specific implementation, the network operation and maintenance device inputs the training data into the multimodal large language model, enabling the multimodal large language model to perform inference calculations based on the training data and output inference data. The network operation and maintenance device freezes the multimodal encoder and the language sub-model in the multimodal large language model. The network operation and maintenance device adjusts the parameters of the modality converter in the multimodal large language model according to the inference data to train the modality converter. In an optional implementation, the network operation and maintenance device adjusts the parameters of the modality converter in the multimodal large language model according to the labeled data corresponding to the training data and the inference data output by the multimodal large language model based on the training data to train the modality converter.

[0174] In the specific implementation, the network operation and maintenance device inputs the training data into the multimodal large language model, enabling the multimodal large language model to perform inference calculations based on the training data and output inference data. The network operation and maintenance device freezes the multimodal encoder and the language sub-model in the multimodal large language model. The network operation and maintenance device adjusts the parameters of the modality converter in the multimodal large language model according to the difference between the labeled data corresponding to the training data and the inference data output by the multimodal large language model based on the training data. The network operation and maintenance device inputs the training data into the multimodal large language model after parameter adjustment, enabling the multimodal large language model after parameter adjustment to perform inference calculations based on the training data and output inference data. The network operation and maintenance device freezes the multimodal encoder and the language sub-model in the multimodal large language model. The network operation and maintenance device adjusts the parameters of the modality converter in the multimodal large language model again according to the difference between the labeled data corresponding to the training data and the inference data output by the multimodal large language model after parameter adjustment. The network operation and maintenance equipment repeatedly executes the process of inputting the training data into the multimodal large language model and adjusting the parameters of the modality converter to train the model until the above-mentioned preset conditions are met.

[0175] In this embodiment, the training data includes content expressed in non-textual form. For example, the training data is data expressed in non-textual form. As another example, the training data includes first data content and second data content, where the first data content is expressed in non-textual form and the second data content is expressed in textual form; that is, the training data is multimodal training data. Depending on the content of the training data, the implementation process of the multimodal large language model performing inference calculations based on this training data differs, and will be described in two embodiments below.

[0176] In one embodiment, the training data is expressed in a non-textual form. After the network operation and maintenance device inputs the training data into the multimodal large language model, the multimodal large language model encodes the training data to obtain a non-textual encoded vector. The multimodal large language model then converts the non-textual encoded vector into a semantic vector, and generates and outputs inference data based on the semantic vector. In an optional implementation, the multimodal large language model includes a multimodal encoder, a modality converter, and a language sub-model, which are sequentially connected. After the network operation and maintenance device inputs the training data into the multimodal large language model, the multimodal encoder encodes the training data to obtain a non-textual encoded vector. The modality converter converts the non-textual encoded vector into a semantic vector, and the language sub-model generates and outputs inference data based on the semantic vector. The network operation and maintenance device adjusts the parameters of the modality converter based on the inference data.

[0177] In another embodiment, the training data includes first data content and second data content. The first data content is expressed in non-text form, and the second data content is expressed in text form. After the network operation and maintenance device inputs the training data into the multimodal large language model, the multimodal large language model encodes the training data to obtain a multimodal encoding vector. The multimodal large language model converts the multimodal encoding vector into a semantic vector. The multimodal large language model generates inference data based on the semantic vector and outputs the inference data. Specifically, the multimodal large language model encodes the first data content to obtain a non-text encoding vector, encodes the second data content to obtain a text encoding vector, and obtains the multimodal encoding vector based on the non-text encoding vector and the text encoding vector. For example, the multimodal large language model concatenates the non-text encoding vector and the text encoding vector to obtain the multimodal encoding vector. In an optional implementation, the multimodal large language model includes a multimodal encoder, a modality converter, and a language sub-model, which are sequentially connected. After the network operation and maintenance device inputs the training data into the multimodal large language model, the multimodal encoder encodes the training data to obtain a multimodal encoded vector. The modality converter converts the multimodal encoded vector into a semantic vector. The language sub-model generates and outputs inference data based on the semantic vector. The network operation and maintenance device adjusts the parameters of the modality converter based on the inference data.

[0178] The technical solution provided in this application embodiment uses training data related to network operation and maintenance, including content expressed in non-textual form. Network operation and maintenance equipment trains a multimodal large language model based on this training data, which improves the accuracy and reliability of the multimodal large language model's inference calculations in the network operation and maintenance field, and further improves the accuracy and reliability of the multimodal large language model's inference calculations based on data including content expressed in non-textual form. For example, if the training data is expressed in non-textual form, the network operation and maintenance equipment trains the modality converter in the multimodal large language model based on this training data, which improves the accuracy and reliability of the modality converter's conversion of non-textual encoded vectors into semantic vectors, thereby improving the accuracy and reliability of the inference data output by the multimodal large language model. For example, the training data includes content expressed in non-textual form and content expressed in textual form. This training data is multimodal training data. The network operation and maintenance equipment trains the modality converter in the multimodal large language model based on the training data, which can improve the accuracy and reliability of the modality converter in converting multimodal encoded vectors into semantic vectors, thereby improving the accuracy and reliability of the inference data output by the multimodal large language model.

[0179] Please refer to Figure 13 This illustrates a flowchart of another network operation and maintenance method provided in an embodiment of this application. For example... Figure 13 As shown, the network operation and maintenance method includes steps S1301 to S1318. For the sake of brevity, in... Figure 13 The numbers S1301 to S1318 are not mentioned in the following description. Furthermore, the operations performed by the network maintenance equipment as described below are specifically performed by the intelligent agents within that network maintenance equipment.

[0180] like Figure 13 As shown, the user received a problem ticket for the target network. Figure 1 Next, input maintenance issue C1 into the network maintenance equipment. Maintenance issue C1 is a multimodal maintenance issue, which includes a problem statement. Figure 1 And the problem description, which is related to the problem statement. Figure 1The relevant text description is as follows: After receiving the maintenance question C1, the network maintenance equipment performs a validity check on the maintenance question C1 (e.g., checking whether the maintenance question C1 contains sensitive information). If the network maintenance equipment successfully verifies the maintenance question C1, it generates a prompt word C1, which includes the maintenance question C1. The network maintenance equipment inputs the prompt word C1 into a multimodal large language model. The multimodal large language model receives the prompt word C1 and obtains at least one maintenance task based on it (specifically, at least one maintenance task based on the maintenance question C1 in the prompt word C1). The multimodal large language model outputs this at least one maintenance task to the network maintenance equipment. The network maintenance equipment receives this at least one maintenance task and obtains the network information of the target network based on it. The network maintenance equipment obtains a matching result for the maintenance question C1 based on the network information of the target network. The network maintenance equipment presents the matching result for the maintenance question C1 to the user.

[0181] After the network maintenance equipment presents the user with the results matching maintenance issue C1, the user can continue to ask follow-up questions. For example... Figure 13 As shown, the user inputs the text "How should I handle this problem?" into the network maintenance equipment. After receiving the text "How should I handle this problem?", the network maintenance equipment recognizes that the text "How should I handle this problem?" is a follow-up question based on maintenance problem C1, and obtains the problem order segment based on the preceding content of the text "How should I handle this problem?". Figure 1 Network maintenance equipment is processed according to the problem report. Figure 1 The text "How should I handle this problem?" generates an operations and maintenance (O&M) issue C2. O&M issue C2 is a multimodal O&M issue, which includes a problem statement. Figure 1 The text "How should I handle this problem?" is a truncated version of the question. Figure 1 The relevant text description is as follows: The network maintenance equipment performs a validity check on maintenance question C2 (e.g., checking if maintenance question C2 contains sensitive information). After successful verification, the network maintenance equipment generates a prompt word C2, which includes maintenance question C2. The network maintenance equipment inputs the prompt word C2 into a multimodal large language model. The multimodal large language model receives the prompt word C2 and obtains a processing suggestion based on it (specifically, it obtains the processing suggestion based on the maintenance question C2 within the prompt word C2; this processing suggestion is the result of matching maintenance question C2). The multimodal large language model outputs this processing suggestion to the network maintenance equipment. The network maintenance equipment receives this processing suggestion. The network maintenance equipment presents the processing suggestion to the user.

[0182] After the network maintenance equipment presents the suggested solution to the user, the user can continue to ask follow-up questions. For example, after the network maintenance equipment presents the suggested solution, the user enters the text "Help me resolve this". After receiving the text "Help me resolve this", the network maintenance equipment recognizes that the text "Help me resolve this" is a follow-up question based on maintenance issue C2, and the network maintenance equipment obtains the issue segment based on the preceding content of the text "Help me resolve this". Figure 1 Network maintenance equipment is processed according to the problem report. Figure 1 The text "Help me handle this" generates an operations and maintenance issue C3. Operations and maintenance issue C3 is a multimodal issue, including a problem statement transcribed into the document. Figure 1 The text "Help me process this" is related to the issue statement. Figure 1 The relevant text description is as follows: The network operation and maintenance (O&M) equipment verifies the legitimacy of O&M question C3. After successful verification, the network O&M equipment generates a prompt word C3, which includes the O&M question C3. The network O&M equipment inputs the prompt word C3 into a multimodal large language model (MLM). The MLM receives the prompt word C3 and obtains at least one O&M task based on it (specifically, at least one O&M task based on the O&M question C3 in the prompt word C3). The MLM outputs this at least one O&M task to the network O&M equipment. The network O&M equipment processes the target network according to this at least one O&M task and presents the processing result (i.e., the result matching the O&M question C3) to the user. It should be noted that, for clarity of the accompanying drawings, the embodiments of this application do not include... Figure 13 The document shows the relevant content of maintenance issue C3, and only this part is described in text.

[0183] Please refer to Figure 14 This illustrates a flowchart of yet another network operation and maintenance method provided in an embodiment of this application. For example... Figure 14 As shown, the network operation and maintenance method includes steps S1401 to S1419. For brevity, in... Figure 14 The numbers S1401 to S1419 are not mentioned in the following description. Furthermore, the operations performed by the network maintenance equipment as described below are specifically performed by the intelligent agents within that network maintenance equipment.

[0184] like Figure 14As shown, the network operation and maintenance (O&M) device can display the network O&M interface of the target network. The user selects a target area and enters a problem description, which is a textual description related to the target area. The network O&M device acquires an image of the target area selected by the user and receives the user's input problem description. Based on the image of the target area and the problem description, the network O&M device generates an O&M problem E1. O&M problem E1 is a multimodal O&M problem, including the image of the target area and the problem description. The network O&M device performs a validity check on O&M problem E1 (e.g., checking whether O&M problem E1 contains sensitive information). After successful verification of O&M problem E1, the network O&M device generates a prompt word E1, which includes the O&M problem E1. The network O&M device inputs the prompt word E1 into the multimodal large language model. The multimodal large language model receives a prompt word E1 and retrieves at least one maintenance task based on it (specifically, at least one maintenance task based on the maintenance question E1 within the prompt word E1). The multimodal large language model outputs this at least one maintenance task to the network maintenance device. The network maintenance device receives this at least one maintenance task and retrieves network information for the target network based on it. The network maintenance device then retrieves a matching result for the maintenance question E1 based on the target network information. Finally, the network maintenance device presents this matching result to the user.

[0185] After the network maintenance equipment presents the user with the results matching maintenance issue E1, the user can continue to ask follow-up questions. For example... Figure 14As shown, the user inputs the text "How should I handle this problem?" into the network maintenance device. After receiving the text "How should I handle this problem?", the network maintenance device recognizes that the text "How should I handle this problem?" is a follow-up question based on maintenance problem E1. The network maintenance device then obtains an image of the target area based on the preceding content of the text "How should I handle this problem?". Based on the image of the target area and the text "How should I handle this problem?", the network maintenance device generates maintenance problem E2. Maintenance problem E2 is a multimodal maintenance problem, including the image of the target area and the text "How should I handle this problem?", where the text "How should I handle this problem?" is a textual description related to the image of the target area. The network maintenance device performs a validity check on maintenance problem E2 (e.g., checking whether maintenance problem E2 contains sensitive information). After successful validation of maintenance problem E2, the network maintenance device generates a prompt word E2, which includes maintenance problem E2. The network maintenance device then inputs the prompt word E2 into the multimodal large language model. The multimodal large language model receives the prompt word E2. Based on E2, it obtains a processing suggestion (specifically, a processing suggestion based on the maintenance problem E2 in the prompt word E1, which is a match between the suggestion and the problem). The multimodal large language model outputs this processing suggestion to the network maintenance equipment. The network maintenance equipment receives the suggestion and then presents it to the user.

[0186] After the network maintenance equipment presents the suggested solution to the user, the user can continue to ask follow-up questions. For example, after presenting the suggested solution, the user inputs the text "Help me with this." Upon receiving the text "Help me with this," the network maintenance equipment recognizes that the text "Help me with this" is a follow-up question based on maintenance question E2. The network maintenance equipment then obtains an image of the target area based on the preceding text "Help me with this." Based on the image of the target area and the text "Help me with this," the network maintenance equipment generates maintenance question E3. Maintenance question E3 is a multimodal maintenance question, including the image of the target area and the text "Help me with this," which is a textual description related to the image of the target area. The network maintenance equipment performs a validity check on maintenance question E3. After successful validation of maintenance question E3, the network maintenance equipment generates a prompt word E3, which includes maintenance question E3. The network maintenance equipment then inputs the prompt word E3 into the multimodal large language model. A multimodal large language model receives the prompt word E3 and obtains at least one maintenance task based on E3 (specifically, at least one maintenance task based on the maintenance question E3 in the prompt word E3). The multimodal large language model outputs the at least one maintenance task to the network maintenance device. The network maintenance device processes the target network according to the at least one maintenance task and presents the processing result (i.e., the result matching the maintenance question E3) to the user. It should be noted that, for clarity of the accompanying drawings, the embodiments of this application do not include... Figure 14 The document shows the relevant content of maintenance issue E3, and only describes this part of the content in text.

[0187] Please refer to Figure 15 This illustrates a flowchart of yet another network operation and maintenance method provided in an embodiment of this application. For example... Figure 15 As shown, the network operation and maintenance method includes steps S1501 to S1518. For the sake of brevity, in... Figure 15 The numbers S1501 to S1518 are not mentioned in the following description. Furthermore, the operations performed by the network maintenance equipment as described below are specifically performed by the intelligent agents within that network maintenance equipment.

[0188] like Figure 15 As shown, the user inputs maintenance question F1 into the network maintenance equipment. Maintenance question F1 is a multimodal maintenance question, which includes screenshots of the interface. Figure 1 The text reads "How do I configure this?". (Screenshot of the interface) Figure 1 These are screenshots related to network configuration. For example, a user learns about a certain network parameter from an operations and maintenance manual or someone else's experience and wants to configure the target network based on that parameter, but cannot find the interface for configuring that parameter. The user takes a screenshot of the interface related to that network parameter (e.g., the interface that displays the network parameter) to obtain the screenshot. Figure 1 The text "How do I configure this?" is related to the screenshot of the interface. Figure 1 The relevant text description is as follows: After receiving the maintenance question F1, the network maintenance equipment performs a validity check on F1 (e.g., checking whether F1 contains sensitive information). If the check is successful, the network maintenance equipment generates a prompt word F1, which includes the maintenance question F1. The network maintenance equipment inputs the prompt word F1 into a multimodal large language model. The multimodal large language model receives the prompt word F1 and retrieves at least one maintenance task based on it (specifically, at least one maintenance task based on the maintenance question F1 in the prompt word F1). This at least one maintenance task is a query task. The multimodal large language model outputs this at least one maintenance task to the network maintenance equipment. The network maintenance equipment receives this at least one maintenance task and queries the maintenance knowledge base to obtain results matching the maintenance question F1. The network maintenance equipment presents the results matching the maintenance question F1 to the user. For example, the results matched with maintenance question F1 are operation instructions, configuration documents, etc. related to the above network parameter configuration.

[0189] After the network maintenance equipment presents the user with the results matching the maintenance question F1, the user can continue to ask follow-up questions. For example... Figure 15 As shown, the user inputs the text "Help me configure this as xx" into the network maintenance device. After receiving the text "Help me configure this as xx", the network maintenance device recognizes that the text "Help me configure this as xx" is a follow-up question based on maintenance question F1. The network maintenance device then retrieves the interface screenshot based on the preceding content of the text "Help me configure this as xx". Figure 1 Network maintenance equipment is based on the interface screenshot. Figure 1 The configuration request F2 is generated along with the text "Help me configure this as xx". Configuration request F2 is a multimodal configuration request, and it includes a screenshot of the interface. Figure 1 The text "Help me configure this as xx" is related to the screenshot. Figure 1The relevant text description is as follows: The network operations and maintenance (O&M) device performs a validity check on configuration request F2 (e.g., checking if configuration request F2 contains sensitive information). After successful verification, the O&M device generates a prompt word F2 based on configuration request F2, which includes configuration request F2. The O&M device inputs prompt word F2 into a multimodal large language model (MLM). The MLM receives prompt word F2 and retrieves at least one O&M task based on it (specifically, at least one O&M task related to network configuration). The MLM outputs this at least one O&M task to the O&M device. The O&M device receives this at least one O&M task, configures the target network according to it, and presents the configuration result (i.e., the result matching configuration request F2) to the user.

[0190] Figure 15 After the network maintenance equipment presents the user with the results matching the maintenance issue F1, the user inputs information into the network maintenance equipment and the interface screenshot. Figure 1 The relevant text for network configuration is illustrated using follow-up questions as an example. In an optional embodiment, after the network operation and maintenance equipment presents the user with the results matching the operation and maintenance question F1, the user inputs information to the network operation and maintenance equipment that matches the interface screenshot. Figure 1 The relevant text used for querying allows for further follow-up questions. For example, a user inputs the text "I want to configure this parameter, what are the constraints?" into the network operations and maintenance (O&M) device. After receiving the text "I want to configure this parameter, what are the constraints?", the O&M device recognizes that the text "I want to configure this parameter, what are the constraints?" is a follow-up question based on the O&M question F1. The O&M device then retrieves the interface screenshot based on the preceding content of the text "I want to configure this parameter, what are the constraints?". Figure 1 Network maintenance equipment is based on the interface screenshot. Figure 1 The query request F3 is generated from the text "I want to configure this parameter, what are the constraints?". Query request F3 is a multimodal query request, which includes a screenshot of the interface. Figure 1 The text "I want to configure this parameter, what are the constraints?" is related to the screenshot of the interface. Figure 1The relevant text description is as follows: The network operation and maintenance equipment performs a validity check on query request F3 (e.g., checks whether query request F3 contains sensitive information). After the network operation and maintenance equipment successfully verifies query request F3, it generates a prompt word F3 based on query request F3, which includes the content of query request F3. The network operation and maintenance equipment inputs prompt word F3 into a multimodal large language model. The multimodal large language model receives prompt word F3 and obtains at least one operation and maintenance task based on prompt word F3 (specifically, it obtains at least one operation and maintenance task based on the content of query request F3 in prompt word F3). The multimodal large language model outputs the at least one operation and maintenance task to the network operation and maintenance equipment, which is a query task. The network operation and maintenance equipment queries the operation and maintenance knowledge base based on the at least one operation and maintenance task to obtain results that match query request F3. For example, the result that matches query request F3 is the constraint condition for configuring the above network parameters. It should be noted that, for clarity of the accompanying drawings, the embodiments of this application do not include... Figure 15 The content related to query request F3 is shown below, using only text description.

[0191] The above is a description of the method embodiments of this application. The following describes the apparatus embodiments of this application, which are used to execute the method of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the method embodiments.

[0192] This application provides a network operation and maintenance device, which includes at least one functional module, the at least one functional module being used to perform, for example... Figure 4 , Figure 10 , Figures 13 to 15 All or part of the steps of the method provided in any embodiment. The at least one functional module can be implemented based on software, hardware, or a combination of software and hardware, and the at least one functional module can be arbitrarily combined or divided based on a specific implementation. As an example, please refer to... Figure 16 This illustration shows a schematic diagram of a network operation and maintenance device 1600 provided in an embodiment of this application. The network operation and maintenance device 1600 can be a network operation and maintenance equipment or a functional component within a network operation and maintenance equipment, which can be a server. For example... Figure 16 As shown, the network operation and maintenance device 1600 includes a first acquisition module 1610, a second acquisition module 1620 and a third acquisition module 1630.

[0193] The first acquisition module 1610 is used to acquire an operation and maintenance request, which includes first content, which is expressed in a non-textual form. The functional implementation of the first acquisition module 1610 can be referred to the relevant description in S401 above.

[0194] The second acquisition module 1620 is used to acquire at least one operation and maintenance task based on the first content using a multimodal large language model. The functional implementation of the second acquisition module 1620 can be found in the relevant description in S402 above.

[0195] The third acquisition module 1630 is used to acquire a request result matching the operation and maintenance request based on the at least one operation and maintenance task. The functional implementation of the third acquisition module 1630 can be referred to the relevant description in S403 above.

[0196] In the optional implementation, the non-text form includes at least one of the image form or the audio form.

[0197] In an optional implementation, the first acquisition module 1610 is used to: receive the first content input by the user in the network operation and maintenance interface; or, acquire the first content based on the region selection instruction triggered by the user in the network operation and maintenance interface, wherein the first content is an image of the target region selected by the region selection instruction.

[0198] In an optional implementation, the multimodal large language model is used to: encode the first content to obtain a non-text encoded vector; convert the non-text encoded vector into a semantic vector; and obtain the at least one operation and maintenance task based on the semantic vector.

[0199] In an optional implementation, the multimodal large language model includes a multimodal encoder, a modality converter, and a language sub-model. The multimodal encoder encodes the first content to obtain a non-text encoded vector. The modality converter converts the non-text encoded vector into a semantic vector. The language sub-model obtains the at least one operational task based on the semantic vector. Please refer to [further details omitted]. Figure 16 The network operation and maintenance device 1600 further includes a training module 1640, used for: acquiring training data, which is data related to network operation and maintenance, and which is expressed in a non-textual form; and training the modality converter in the multimodal large language model based on the training data. The functional implementation of the training module 1640 can be found in the relevant description in S406 above.

[0200] In an optional implementation, the at least one operation and maintenance task includes a query task. The third acquisition module 1630 is used to: encode the first content according to the query task to obtain a non-text encoded vector; convert the non-text encoded vector into a semantic vector; obtain a knowledge vector that matches the semantic vector from the operation and maintenance knowledge base; and determine the knowledge corresponding to the knowledge vector as the request result that matches the operation and maintenance request.

[0201] In an optional implementation, the maintenance request also includes second content, which is content expressed in text form. The second acquisition module 1620 is used to: acquire the at least one maintenance task using the multimodal large language model based on the first content and the second content.

[0202] In an optional implementation, the first acquisition module 1610 is configured to: receive first content and second content input by the user in the network operation and maintenance interface; or, receive the second content input by the user in the network operation and maintenance interface, and acquire the first content based on the preceding content of the second content, wherein the preceding content of the second content is the content of the user's interaction with the network operation and maintenance interface before the user inputs the second content, and the preceding content of the second content includes the first content; or, acquire the first content based on the region selection instruction triggered by the user in the network operation and maintenance interface, and receive the second content input by the user in the network operation and maintenance interface, wherein the first content is an image of the target region selected by the region selection instruction.

[0203] In an optional implementation, the multimodal large language model is used to: encode the first content and the second content to obtain a multimodal encoding vector; convert the multimodal encoding vector into a semantic vector; and obtain the at least one operation and maintenance task based on the semantic vector.

[0204] In an optional implementation, the multimodal large language model includes a multimodal encoder, a modality converter, and a language sub-model. The multimodal encoder encodes the first and second content to obtain a multimodal encoded vector. The modality converter converts the multimodal encoded vector into a semantic vector. The language sub-model retrieves the at least one operational task based on the semantic vector. Please refer to [further details omitted]. Figure 16 The network operation and maintenance device 1600 also includes a training module 1640, used for: acquiring training data, which is data related to network operation and maintenance, including first data content and second data content, wherein the first data content is expressed in non-text form and the second data content is expressed in text form; and training the modality converter in the multimodal large language model based on the training data. The functional implementation of the training module 1640 can be referred to the relevant description in S406 above.

[0205] In an optional implementation, the at least one operation and maintenance task includes a query task. The third acquisition module 1630 is used to: encode the first content and the second content according to the query task to obtain a multimodal encoding vector; convert the multimodal encoding vector into a semantic vector; obtain a knowledge vector that matches the semantic vector from the operation and maintenance knowledge base; and determine the knowledge corresponding to the knowledge vector as a request result that matches the operation and maintenance request.

[0206] In the optional implementation, the maintenance request is for maintenance issues of the target network, the first content is for content related to the target network, and the third acquisition module 1630 is used to: acquire network information of the target network based on the at least one maintenance task; determine the request result matching the maintenance request based on the network information of the target network; or,

[0207] The operation and maintenance request is a configuration request for the target network. The first content is content for the target network. The third acquisition module 1630 is used to: configure the target network according to the at least one operation and maintenance task, and determine the request result that matches the operation and maintenance request based on the configuration result of the target network.

[0208] Please refer to the optional implementation methods. Figure 16 The network operation and maintenance device 1600 also includes: a verification module 1650 and an output module 1660.

[0209] The verification module 1650 is used to verify the request result using the multimodal large language model. The implementation of the verification module 1650 can be found in the relevant description in S404 above.

[0210] Output module 1660 is used to output the request result if the verification of the request result is successful. The implementation of the function of output module 1660 can be referred to the relevant description in S405 above.

[0211] In summary, the technical solution provided in this application embodiment involves a network operation and maintenance device obtaining an operation and maintenance request that includes first content expressed in a non-textual form. Based on the first content, it uses a multimodal large language model to obtain at least one operation and maintenance task, and then obtains a request result matching the operation and maintenance request based on the at least one operation and maintenance task. It is evident that the multimodal large language model used in this application embodiment can understand and process the non-textual content included in the operation and maintenance request. For example, the non-textual form includes at least one of image or audio forms; that is, the multimodal large language model can understand and process content expressed in image form and / or audio form. In other words, the multimodal large language model can understand and process operation and maintenance requests whose content includes images and / or audio. Therefore, using this multimodal large language model to process operation and maintenance requests offers high flexibility and helps improve the flexibility of network operation and maintenance.

[0212] The network operation and maintenance device provided in this application embodiment can also be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. The network operation and maintenance method provided in the above embodiments can also be implemented in software. When the network operation and maintenance method provided in the above embodiments is implemented in software, each module in the network operation and maintenance device can be a software module.

[0213] This application provides a network operation and maintenance device, which includes a memory and a processor. The memory stores computer programs. The processor executes the computer programs stored in the memory to cause the network operation and maintenance device to perform actions such as... Figure 4 , Figure 10 , Figures 13 to 15 All or part of the steps of the method provided in any embodiment. The network operation and maintenance device may be a network operation and maintenance equipment or a functional component of a network operation and maintenance equipment, and the network operation and maintenance equipment may be a server.

[0214] As an example, please refer to Figure 17 This illustration shows a schematic diagram of another network operation and maintenance device 1700 provided in an embodiment of this application. The network operation and maintenance device 1700 is a network operation and maintenance equipment or a functional component within a network operation and maintenance equipment. For example, the network operation and maintenance device 1700 is an intelligent agent within a network operation and maintenance equipment. Figure 17 As shown, the network operation and maintenance device 1700 includes a processor 1701, a memory 1702, a bus 1703, a communication interface 1704, and an input / output device 1705. The processor 1701, memory 1702, communication interface 1704, and input / output device 1705 are connected via the bus 1703. Figure 17 The processor 1701 and memory 1702 are described independently, but the processor 1701 and memory 1702 can also be integrated together, and this application embodiment does not limit this.

[0215] Memory 1702 is used to store computer programs, including operating systems and program code. Memory 1702 can be various types of storage media. For example, memory 1702 is random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM), flash memory, registers, optical disc storage, optical disc storage (including compressed discs, laser discs, optical discs, digital universal discs, Blu-ray discs, etc.), magnetic disks, or other magnetic storage devices.

[0216] Processor 1701 can be a general-purpose processor or a special-purpose processor. A general-purpose processor is a processor that performs specific steps and / or operations by reading and executing a computer program stored in memory. The general-purpose processor may use the computer program stored in memory during the execution of the aforementioned steps and / or operations. The computer program may be executed, for example, to implement the relevant functions of the aforementioned first acquisition module 1610, second acquisition module 1620, third acquisition module 1630, training module 1640, verification module 1650, and output module 1660. A general-purpose processor is, for example, but not limited to, a central processing unit (CPU). A special-purpose processor is a processor specifically designed to perform specific steps and / or operations. Special-purpose processors are, for example, but not limited to, application-specific integrated circuits (ASICs), complex programmable logical devices (CPLDs), field-programmable gate arrays (FPGAs), generic array logic (GALs), or any combination thereof. Processor 1701 may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. The processor 1701 includes at least one circuit to perform all or part of the steps of the network operation and maintenance method provided in the above embodiments.

[0217] Communication interface 1704 includes physical interfaces and logical interfaces for interconnecting devices within the network maintenance device 1700, and interfaces for interconnecting the network maintenance device 1700 with other devices and networks. The physical interface can be an Ethernet interface, Fast Ethernet (FE) interface, Gigabit Ethernet (GE) interface, Terabit Ethernet (TbE) interface, 400GE interface, Asynchronous Transfer Mode (ATM) interface, etc., which can be used to interconnect the network maintenance device 1700 with other devices. The logical interface can be an internal interface of the network maintenance device 1700, which can be used to interconnect devices within the network maintenance device 1700. It is easy to understand that communication interface 1704 can be used for communication between the network maintenance device 1700 and other devices. For example, communication interface 1704 is used for sending and receiving information between the network maintenance device 1700 and other devices and networks. The communication interface 1704 can also be any transceiver (such as a transmitter or receiver).

[0218] Input / output device 1705 includes an input / output (I / O) interface and devices such as a keyboard, mouse, and monitor connected to network operation and maintenance device 1700 via the I / O interface, and devices such as a monitor connected to processor 1701 via a bus. Processor 1701 can receive user input commands or data (e.g., operation and maintenance requests, images, audio, text, etc.) through input / output device 1705 and output processed data. For example, input / output device 1705 includes a monitor, which can be used to display intermediate and / or final results generated by processor 1701 executing the above-described network operation and maintenance method, such as displaying request results matching the operation and maintenance request. In some embodiments, input / output device 1705 also includes an audio device (or audio component), which can output intermediate and / or final results generated by processor 1701 executing the above-described network operation and maintenance method in audio form. Audio device is also called audio component, monitor is also called display component, and keyboard, mouse, monitor, audio device, etc., are also called human-computer interaction components; this application embodiment does not limit these terms.

[0219] Bus 1703 can be any type of communication bus used to interconnect internal devices of the network maintenance device 1700. For example, bus 1703 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. Bus 1703 can be divided into address bus, data bus, control bus, etc. This embodiment of the application illustrates the interconnection of the aforementioned devices inside the network maintenance device 1700 via bus 1703. The aforementioned devices inside the network maintenance device 1700 can also be connected to each other using other connection methods. For example, the aforementioned devices inside the network maintenance device 1700 can be interconnected through the logical interface inside the network maintenance device 1700, and this embodiment of the application does not limit this.

[0220] The aforementioned devices can be disposed on separate chips, or at least partially or entirely on the same chip. Whether to dispose of the devices independently on different chips or integrate them on one or more chips often depends on the needs of the product design. This application does not limit the specific implementation of the aforementioned devices.

[0221] Those skilled in the art will understand that Figure 17 The network operation and maintenance device 1700 shown is merely an example. The network operation and maintenance device 1700 may also include other components, which will not be listed here. Figure 17The network operation and maintenance device 1700 shown can obtain a request result matching the operation and maintenance request by executing all or part of the steps of the network operation and maintenance method provided in the above embodiments.

[0222] Based on the same inventive concept, embodiments of this application provide a network operation and maintenance system, which includes network operation and maintenance equipment and an AI server. The AI ​​server deploys a multimodal large language model, and the network operation and maintenance equipment includes, for example, […]. Figure 16 or Figure 17 The network operation and maintenance device shown is used to: obtain at least one operation and maintenance task based on the first content expressed in non-textual form included in the operation and maintenance request using the multimodal large language model, and obtain a request result matching the operation and maintenance request based on the at least one operation and maintenance task.

[0223] For example, the network operation and maintenance system is as follows: Figures 1 to 3 As shown.

[0224] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium storing a computer program. When the computer program is executed (e.g., executed by a network maintenance device, network maintenance equipment, server, one or more processors, etc.), it implements the following... Figure 4 , Figure 10 , Figures 13 to 15 All or part of the steps of the method provided in the illustrated embodiments.

[0225] Based on the same inventive concept, embodiments of this application provide a computer program product, which includes a program or code. When the program or code is executed (e.g., executed by a network maintenance device, network maintenance equipment, server, one or more processors, etc.), it implements the following: Figure 4 , Figure 10 , Figures 13 to 15 All or part of the steps of the method provided in the illustrated embodiments.

[0226] Based on the same inventive concept, this application provides a chip that includes programmable logic circuitry and / or program instructions. When the chip operates, it is used to implement, for example... Figure 4 , Figure 10 , Figures 13 to 15 All or part of the steps of the method provided in the illustrated embodiments.

[0227] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented entirely or partially as a computer program product, which includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium, or a semiconductor medium (e.g., solid-state drive), etc.

[0228] It should be understood that the term "at least one" in this application refers to one or more, and "multiple" refers to two or more. In this application, unless otherwise stated, the symbol " / " generally means "or," for example, A / B can mean A or B. The term "and / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, for clarity, this application uses terms such as "first," "second," and "third" to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first," "second," and "third" do not limit the quantity or order of execution.

[0229] The different types of embodiments, such as the method embodiments and device embodiments provided in this application, can be referenced to each other. The order of operations in the method embodiments can be adjusted appropriately, and the operations can be added or removed in response to the situation. Any variations that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application, and therefore will not be described in detail.

[0230] In the corresponding embodiments provided in this application, it should be understood that the disclosed devices, etc., can be implemented through other configurations. For example, the device embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed between devices or modules may be through some interfaces, or indirect coupling or communication connection between devices or modules, which may be electrical or other forms. Modules described as separate components may or may not be physically separate, and components described as modules may or may not be physical modules; they may be located in one place or distributed across multiple network nodes. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0231] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any equivalent modifications or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A network operation and maintenance method, characterized in that, The method includes: Obtain an operation and maintenance request, the operation and maintenance request including first content, the first content being content expressed in a non-textual form; Based on the first content, at least one operation and maintenance task is obtained using a multimodal large language model; Obtain the request result that matches the operation and maintenance request based on the at least one operation and maintenance task.

2. The method according to claim 1, characterized in that, The non-text format includes at least one of image or audio formats, and the acquisition of the maintenance request includes: Receive the first content entered by the user in the network operation and maintenance interface; or, The first content is obtained based on the region selection command triggered by the user in the network operation and maintenance interface. The first content is an image of the target region selected by the region selection command.

3. The method according to claim 1 or 2, characterized in that, The multimodal large language model is used for: The first content is encoded to obtain a non-text encoded vector; Convert the non-text encoded vector into a semantic vector; The at least one operation and maintenance task is obtained based on the semantic vector.

4. The method according to claim 3, characterized in that, The multimodal large language model includes a multimodal encoder, a modality converter, and a language sub-model. The multimodal encoder is used to encode the first content to obtain the non-text encoded vector. The modality converter is used to convert the non-text encoded vector into the semantic vector. The language sub-model is used to obtain the at least one operation and maintenance task based on the semantic vector. The method further includes: Acquire training data, wherein the training data is data related to network operation and maintenance, and the training data is data expressed in a non-textual form; The modality converter in the multimodal large language model is trained based on the training data.

5. The method according to any one of claims 1 to 4, characterized in that, The at least one operation and maintenance task includes a query task, and obtaining the request result matching the operation and maintenance request based on the at least one operation and maintenance task includes: The first content is encoded according to the query task to obtain a non-text encoded vector; Convert the non-text encoded vector into a semantic vector; Obtain a knowledge vector that matches the semantic vector from the operations and maintenance knowledge base; The knowledge corresponding to the knowledge vector is determined as the request result that matches the operation and maintenance request.

6. The method according to claim 1, characterized in that, The non-text format includes at least one of image or audio formats, and the maintenance request further includes second content, which is content expressed in text format. The step of obtaining at least one maintenance task using a multimodal large language model based on the first content includes: Based on the first content and the second content, the at least one operation and maintenance task is obtained using the multimodal large language model.

7. The method according to claim 6, characterized in that, The acquisition of maintenance requests includes: Receive the first and second content input by the user in the network operation and maintenance interface; or, The system receives the second content input by the user in the network operation and maintenance interface, and obtains the first content based on the preceding text of the second content. The preceding text of the second content refers to the content interacted between the user and the network operation and maintenance interface before the user inputs the second content, and the preceding text of the second content includes the first content; or, The first content is obtained based on the region selection command triggered by the user in the network operation and maintenance interface, and the second content input by the user in the network operation and maintenance interface is received. The first content is an image of the target region selected by the region selection command.

8. The method according to claim 6 or 7, characterized in that, The multimodal large language model is used for: The first content and the second content are encoded to obtain a multimodal encoding vector; Convert the multimodal encoded vector into a semantic vector; The at least one operation and maintenance task is obtained based on the semantic vector.

9. The method according to claim 8, characterized in that, The multimodal large language model includes a multimodal encoder, a modality converter, and a language sub-model. The multimodal encoder is used to encode the first content and the second content to obtain the multimodal encoded vector. The modality converter is used to convert the multimodal encoded vector into the semantic vector. The language sub-model is used to obtain the at least one operation and maintenance task based on the semantic vector. The method further includes: Acquire training data, which is data related to network operation and maintenance. The training data includes first data content and second data content. The first data content is expressed in non-text form, and the second data content is expressed in text form. The modality converter in the multimodal large language model is trained based on the training data.

10. The method according to any one of claims 6 to 9, characterized in that, The at least one operation and maintenance task includes a query task, and obtaining the request result matching the operation and maintenance request based on the at least one operation and maintenance task includes: The first content and the second content are encoded according to the query task to obtain a multimodal encoding vector; Convert the multimodal encoded vector into a semantic vector; Obtain a knowledge vector that matches the semantic vector from the operations and maintenance knowledge base; The knowledge corresponding to the knowledge vector is determined as the request result that matches the operation and maintenance request.

11. The method according to any one of claims 1 to 9, characterized in that, The maintenance request is for maintenance issues related to the target network, the first content is for content related to the target network, and obtaining a request result matching the maintenance request based on the at least one maintenance task includes: obtaining network information of the target network based on the at least one maintenance task; determining a request result matching the maintenance request based on the network information of the target network; or... The maintenance request is a configuration request for the target network, the first content is content for the target network, and obtaining the request result matching the maintenance request based on the at least one maintenance task includes: configuring the target network according to the at least one maintenance task, and determining the request result matching the maintenance request based on the configuration result of the target network.

12. The method according to any one of claims 1 to 11, characterized in that, The method further includes: The request result is validated using the aforementioned multimodal large language model; If the request result is successfully verified, the request result is output.

13. A network operation and maintenance device, characterized in that, The apparatus includes modules for implementing the functions corresponding to the steps included in any of the methods of claims 1 to 12.

Images