Communication scheduling large screen monitoring method and system based on large model agent

Abstract

The embodiment of the invention provides a communication scheduling large-screen monitoring method and system based on a large model agent. The method comprises the following steps: performing supervised training on a link prediction model by using a historical question data set; obtaining the current questioning content of the user to the communication optical cable, and obtaining a predicted questioning link of the current questioning content by using the trained link prediction model; generating prompt content based on the predicted question link, the current display data and a prompt template, and inputting the prompt content into a trained visual language model to obtain layout data generated by the visual language model; wherein the layout data comprises to-be-displayed data and display parameters; the to-be-displayed data is related to the predicted question link and comprises monitoring data, environment data, fault data and emergency data; and updating a display interface of the communication scheduling screen based on the layout data. A solution is provided before a user puts forward a problem, and the problem is prevented from worsening or expanding. Managing the layout of the communication scheduling screen using a visual language model may improve management efficiency.

Description

Technical Field

[0001] The present application relates to the field of screen display technology, and more specifically, to a communication scheduling large-screen monitoring method and system based on a large model intelligent agent. Background Art

[0002] With the development of communication technology, optical communication network is one of the most important communication networks in the current power industry. As the scale of optical communication network continues to expand, maintaining communication optical cables deployed in different geographical areas is a challenging and important task.

[0003] In the related art, the maintenance party of the communication optical cable can view various data such as the working status and fault information of the communication optical cable through the communication scheduling screen. However, the amount of data involved in the communication optical cable is huge and the types are numerous. It is difficult for users who are not familiar with the communication system to quickly retrieve the required data for display, and it is also difficult to quickly obtain the required data from the numerous data displayed on the communication scheduling screen. As a result, it is difficult for users to quickly and conveniently perform optical cable maintenance through the communication scheduling screen. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a communication scheduling large-screen monitoring method and system based on a large model intelligent agent to solve at least one of the technical problems mentioned above.

[0005] In a first aspect, an embodiment of the present application provides a communication scheduling large-screen monitoring method and system based on a large model agent, wherein the large model agent includes a visual language model, and the communication scheduling screen is used to display monitoring data, environmental data, fault data, and emergency data of a communication optical cable. The method includes:

[0006] The link prediction model is supervisedly trained using a historical question data set; wherein the historical question data set includes a plurality of historical question links carrying features, each of the historical question links includes historical question content and subsequent question content of a user on the display content of a communication scheduling screen; the features include text features and context features of the historical question links;

[0007] Acquire the current question content of the user to the communication optical cable, and use the trained link prediction model to acquire a predicted question link that predicts the current question content, wherein the predicted question link at least includes the next question content to the communication optical cable;

[0008] Generate prompt content based on the predicted question link, the current display data of the communication scheduling screen and a preset prompt template, and input the prompt content into a trained visual language model to obtain layout data generated by the visual language model; wherein the layout data includes data to be displayed and display parameters; the data to be displayed is related to the predicted question link and includes one or more of the monitoring data, the environmental data, the fault data, and the emergency data;

[0009] A display interface of the communication scheduling screen is updated based on the layout data.

[0010] A second aspect provides an electronic device, including:

[0011] processor;

[0012] a memory for storing processor-executable instructions;

[0013] Wherein, when the processor calls the executable instruction, the method described in the first aspect is implemented.

[0014] The third aspect provides a communication scheduling large-screen monitoring system based on a large model agent, wherein the large model agent includes a visual language model, and the communication scheduling screen is used to display monitoring data, environmental data, fault data, and emergency data of the communication optical cable. The system includes:

[0015] A training module, used for performing supervised training on a link prediction model using a historical question data set; wherein the historical question data set includes a plurality of historical question links carrying features, each of the historical question links includes historical question content and subsequent question content of a user on the display content of a communication scheduling screen; the features include text features and context features of the historical question links;

[0016] A prediction module, used to obtain the current question content of the user on the communication optical cable, and use the trained link prediction model to obtain a predicted question link of the current question content, wherein the predicted question link at least includes the next question content on the communication optical cable;

[0017] a layout module, configured to generate prompt content based on the predicted question link, the current display data of the communication scheduling screen and a preset prompt template, and input the prompt content into a trained visual language model to obtain layout data generated by the visual language model; wherein the layout data includes data to be displayed and display parameters; the data to be displayed is related to the predicted question link and includes one or more of the monitoring data, the environmental data, the fault data, and the emergency data;

[0018] An updating module is used to update the display interface of the communication scheduling screen based on the layout data.

[0019] A fourth aspect provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0020] This application understands and predicts the user's needs by predicting the user's question chain, and then uses the visual language model to adjust the layout of the communication scheduling screen based on the user's question chain, so that before the user asks the next question, the detailed data related to the user's question can be presented in time according to the predicted next question content, saving the user's time to interact with the system, making the entire process smoother and more efficient, and the display content of the communication scheduling screen can better meet user needs. Providing solutions before the user asks a question helps to avoid the deterioration or expansion of the problem. And using the visual language model to manage the layout of the communication scheduling screen can make the management of the communication scheduling screen more efficient and convenient. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 A flow chart of a communication scheduling large-screen monitoring method based on a large model intelligent agent provided in an embodiment of the present application;

[0023] Figure 2 A flow chart of another communication scheduling large-screen monitoring method based on a large model agent provided in an embodiment of the present application;

[0024] Figure 3 A hardware structure diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0026] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0027] In the related art, due to the huge amount of data involved in communication optical cables and the numerous types of data, it is difficult for users to quickly and conveniently perform optical cable maintenance through the communication scheduling screen. To this end, the present application provides a communication scheduling large-screen monitoring method based on a large model intelligent agent. The large model intelligent agent includes a visual language model, and the communication scheduling large screen is a communication scheduling screen, which is used to display one or more of the monitoring data, environmental data, fault data and emergency data of the communication optical cable.

[0028] As an example, the communication scheduling screen displays the monitoring data of the communication optical cable, and the display content specifically includes but is not limited to the length of the optical cable, the fiber core utilization rate, the maintenance level, the asset information, and the business carrying status. The display screen may include but is not limited to the real-time status and historical trend charts of the communication optical cable, and the detailed information of the important optical cables. The charts include but are not limited to line charts, bar charts, etc. The detailed information includes but is not limited to the fiber core utilization rate and the maintenance level, and the detailed information of the important optical cables can be highlighted. The monitoring data can also be displayed according to preset display rules, which include giving priority to displaying high-load optical cables according to the fiber core utilization rate and / or maintenance level, and using color coding to identify optical cables in different load states. For example, green indicates normal load and red indicates high load.

[0029] As an example, the communication scheduling screen displays the environmental data of the communication optical cable, and the environmental data includes the dynamic environmental data of the environment in which the communication optical cable is located, and the health data of the equipment related to the communication optical cable. Among them, the dynamic environmental data specifically includes but is not limited to ambient temperature, ambient humidity and voltage. The display screen may include but is not limited to real-time environmental parameter charts. The charts include, for example, but are not limited to temperature curves, humidity bar charts, voltage line charts, etc. Optionally, abnormal environmental parameters can be highlighted, and detailed abnormal causes and handling suggestions are provided. Dynamic environmental data can also be displayed according to preset display rules. The display rules, for example, include triggering an alarm according to a preset environmental data threshold, and giving priority to displaying dynamic environmental data that exceeds the environmental data threshold. The display rules also include, for example, using a thermal map to display the temperature distribution, and combining the map to display the location and status of the device.

[0030] In addition, health data specifically includes but is not limited to device health, the number of normal devices and the number of abnormal devices. The display screen may include but is not limited to detailed information and processing suggestions for abnormal devices, and a device health status chart. Among them, the device health status chart is used to show the ratio between normal devices and abnormal devices. Health data can also be displayed according to preset display rules. Display rules, for example, include giving priority to displaying devices with reduced health and using color coding to identify abnormal devices.

[0031] As an example, the communication dispatch screen displays the emergency data of the communication optical cable, and the emergency data includes emergency event monitoring data and emergency plan data. Among them, the emergency event monitoring data specifically includes but is not limited to security records, real-time statistical data, maintenance data and emergency dispatch personnel information. The display screen may include but is not limited to the real-time status and processing progress chart of the emergency event, and the location and task status of the emergency dispatch personnel displayed by maps and icons. The emergency event monitoring data can also be displayed according to preset display rules, which include priority display according to the severity and scope of impact of the emergency event, and the use of colors and / or icons to identify emergency events and personnel locations in different states.

[0032] In addition, the emergency plan data specifically includes but is not limited to the content of the emergency plan and the execution status of the plan. The display screen may include but is not limited to the detailed information of the emergency plan currently in effect, and the plan execution progress chart. Optionally, the detailed information of the emergency plan currently in effect can be displayed in the form of a card or a list. The plan execution progress chart can update the execution status in real time. The emergency plan data can also be displayed according to preset display rules, which include giving priority to displaying key plans and execution status, and using color coding to identify the plan status.

[0033] Of course, the communication scheduling screen is not limited to displaying the above-listed data. In practical applications, users can also schedule other data of the communication optical cable for display according to actual needs. This application does not limit the data type displayed on the communication scheduling screen.

[0034] Based on this, the monitoring method provided by the present application may include: Figure 1 Steps 110 to 130 are shown.

[0035] Step 110: Obtain the current question content of the user on the communication optical cable, and use the trained link prediction model to obtain the predicted question link of the current question content. The predicted question link at least includes the next question content on the communication optical cable.

[0036] Regarding the current question content of the user on the communication optical cable, as an example, the user can input the current question content through the human-computer interaction component. The human-computer interaction component includes, but is not limited to, a keyboard, a mouse, a microphone, etc. The current question content can be in text form or in voice form, and this application does not limit this.

[0037] After obtaining the current question content, the link prediction model can be used to predict the subsequent question content that the user may continue to ask. Among them, the training process of the link prediction model will be unfolded below. The subsequent question content may include one or more. In this way, a predicted question link including the current question content and at least the next question content can be obtained.

[0038] Step 120: Generate prompt content based on the predicted question link, the current display data of the communication scheduling screen and the preset prompt template, and input the prompt content into the trained visual language model to obtain layout data generated by the visual language model.

[0039] The layout data includes data to be displayed and display parameters; the data to be displayed is related to the prediction question link and includes one or more of the monitoring data, the environmental data, the fault data, and the emergency data.

[0040] Exemplarily, the visual language model can be a large model. The large model refers to a deep learning model trained based on massive text data, and the number of its parameters can be as high as millions or even hundreds of millions. Therefore, the large model can process more complex and comprehensive data, and learn more patterns and rules from it. In the large model, the role of prompt is to prompt the large model with the context of input information and parameter information of the input model. The prompt template consists of instructions, context, examples, input and output. In the prompt word project, prompt can use the RTF prompt word framework. The RTF prompt word framework covers three main elements: role, task and format to ensure that the model generates results that meet user needs.

[0041] In some embodiments, the visual language model used in this embodiment is, for example, a ScreenAI model. In this way, the predicted question link predicted in step 110 and the current display data of the communication scheduling screen can be input into the prompt template of the ScreenAI model to obtain the prompt content. Among them, the current display data can be an interface screenshot of the current display interface, and / or HTML data corresponding to the current display interface. Subsequently, the prompt content carrying the predicted question link is input into the trained ScreenAI model to obtain the layout data generated by the ScreenAI model. It can be seen that the layout data is generated based on the predicted question link prediction. The layout data includes at least data to be displayed. The data to be displayed includes one or more of the monitoring data, environmental data, fault data, and emergency data mentioned above. In addition, in addition to the data to be displayed, the layout data may also include display parameters. Display parameters include, for example, but are not limited to, the display position, size, color, brightness, contrast, animation parameters, and display time of the display content in the communication scheduling screen. By adjusting the display parameters, more important display content can be displayed differently from unimportant display content.

[0042] Exemplarily, the layout data specifically includes hierarchical information of a display interface in a communication scheduling screen. The hierarchical information is used to characterize the layout of the display interface, and the layout includes information such as element size, position, color, etc. For example, the hierarchical information may include, but is not limited to, DOM (Document Object Model) elements of the display interface.

[0043] In addition, the monitoring data, environmental data, fault data, and emergency data in the data to be displayed are underlying data stored in the communication system or generated in real time. The layout data can indicate the location and display form of one or more of the monitoring data, environmental data, fault data, and emergency data in the display interface. The form includes, but is not limited to, charts, maps, and texts, etc. The communication system can read the corresponding data from the storage space based on the instructions of the layout data to generate a corresponding display interface.

[0044] Optionally, the currently displayed data of the communication scheduling screen includes at least the statistical graph currently displayed in the communication scheduling screen. The statistical graph includes but is not limited to a column chart, a bar chart, a pie chart, and the like. As an example, after obtaining the current question content of the user in step 110, in addition to using the current question content to predict the question link, it is also possible to determine whether the currently displayed data of the communication scheduling screen includes the question data included in the current question content. The question data includes, for example, but is not limited to one or more of data about statistical graphs, data about tables, and data about text. If not included, the question data is also included in the data to be displayed determined by the visual language model.

[0045] Step 130: Update the display interface of the communication scheduling screen based on the layout data.

[0046] Finally, based on the visual language model, for example, the layout data output by the ScreenAI model updates the display interface of the communication scheduling screen in real time. In this way, for the display interface of the communication scheduling screen, this embodiment provides real-time update and dynamic interaction functions. Specifically, visualization tools (such as D3.js, ECharts, etc.) can be used to realize the dynamic display of data and the interaction with users. Users can click to view the detailed data of the content displayed on the screen, and can also dynamically adjust the display elements and order in the display interface according to user questions and needs.

[0047] It can be seen that this embodiment understands and predicts the user's needs by predicting the user's question link, and then uses the visual language model to adjust the layout of the communication scheduling screen based on the user's question link, so that before the user asks the next question, the detailed data related to the user's question can be presented in time according to the predicted next question content, saving the user's interaction time with the system, making the entire process smoother and more efficient, and the display interface of the communication scheduling screen can better meet user needs. Providing solutions before the user asks a question helps to avoid the deterioration or expansion of the problem. And using the visual language model to manage the layout of the communication scheduling screen can make the management of the communication scheduling screen more efficient and convenient.

[0048] The following is a detailed introduction to step 110 to step 130.

[0049] Regarding the prediction process of predicting the question link in step 110, in some embodiments, a trained link prediction model can be used for prediction. The link prediction model includes, for example, but is not limited to, a sequence model, and a Transformer model, etc. Among them, the sequence model includes, for example, an LSTM model and a GRU model, etc. The Transformer model includes, for example, a BERT (Bidirectional Encoder Representations from Transformers) model and a GPT (Generative Pre-Trained Transformer) model, etc.

[0050] The following is the training process of the link prediction model: Specifically, first, the user's historical question content and the corresponding subsequent question content are constructed into a historical question data set. Among them, a user's historical question content and its subsequent question content constitute a historical question link. The historical question data set includes multiple historical question links.

[0051] Optionally, the data in the historical question data set may be cleaned and preprocessed. The preprocessing includes but is not limited to noise removal, format standardization, and word segmentation. The data may be cleaned and preprocessed based on the historical question link. Alternatively, the data may be cleaned and preprocessed based on the question content in the historical question link.

[0052] Subsequently, feature extraction can be performed on the historical question link. The feature extraction includes text feature extraction and context feature extraction. Specifically, text feature extraction includes using natural language processing technology to extract features from the historical question link, and / or extract features from the question content in the historical question link. For example, one or more technologies in TF-IDF (Term Frequency–Inverse Document Frequency), word vector model and BERT model are used to generate a vector representation of the question link, and / or a vector representation of the question content in the historical question link. The word vector model includes, for example, the Word2Vec model and the GloVe (Global Vectors for Word Representation) model. In addition, the context features include but are not limited to features such as the historical question order and the time interval between questions.

[0053] Subsequently, the link prediction model can be supervisedly trained using the historical question data set. Specifically, the input of the link prediction model is part of the content of the historical question link, such as one or more previous question contents in the historical question link, and the model output is the next possible question content. By comparing the next question content output by the model with the actual next question content in the historical question link, a loss function is constructed, and the model parameters are optimized using the loss function, and finally a trained link prediction model is obtained. In addition, in the above training process, the historical question data set used can carry the extracted features to help improve the prediction accuracy and training efficiency of the link prediction model.

[0054] After obtaining the trained link prediction model, when executing step 110, the user's current question content can be input into the trained link prediction model, or feature extraction can be performed on the current question content first, including text feature extraction and context feature extraction. Then the current question content carrying text features and context features is input into the trained link prediction model. The link prediction model can predict the corresponding next question content based on the current question content, and obtain a predicted question link including the current question content and the next question content. In addition, the prediction result of the link prediction model can be used as part of the prompt for training the visual language model.

[0055] Optionally, the next question content predicted by the model may be further input into the link prediction model to obtain the next question content, and so on, so as to obtain a predicted question link including the current question content and multiple subsequent question contents.

[0056] In addition, in some embodiments, the output of the link prediction model may include multiple candidate question contents. Each candidate question content carries a probability and / or a relevance score, wherein the probability is used to characterize the possibility that the candidate question content is the next question content of the current question content, and the relevance score is used to characterize the degree of relevance between the candidate question content and the current question content.

[0057] As an example, the candidate question content with the highest probability and / or the highest relevance score among all the candidate question contents may be used as the next question content.

[0058] As an example, multiple target candidate question contents whose probabilities are greater than a preset probability threshold and / or whose relevance scores are greater than a preset score threshold can be determined, a predicted question link corresponding to each target candidate question content can be obtained, and subsequent layout data generation can be performed based on the multiple predicted question links.

[0059] In addition, optionally, multiple candidate question contents can also be displayed to the user through the communication scheduling screen. The user can select the question he wants to ask from the displayed multiple candidate question contents, so that the user can ask questions.

[0060] Regarding the fine-tuning process of the visual language model, in order to enable the visual language model to have the ability to generate layout data corresponding to the question content, in some embodiments, the training data of the visual language model may include historical question content carrying historical layout data, so that the visual language model can learn from the training data how to generate corresponding layout data based on the question content. The historical layout data may include screenshots of the historical display interface of the communication scheduling screen, and / or HTML data corresponding to the historical display interface. In addition, in some embodiments, the fine-tuning process of the visual language model may also include the following: Figure 2 Steps 210 to 230 are shown.

[0061] Step 210: Obtain the user's historical behavior data.

[0062] The historical behavior data includes multiple rounds of historical interaction data between the user and the communication scheduling screen, and the user's historical question link data. The historical interaction data may include each operation, click, input question and other interactive behavior data of the user when using the user interface (UI) of the communication scheduling screen, as well as the user's stay time data on different pages and functional modules. Optionally, the collected historical behavior data may be cleaned and preprocessed to remove invalid or noise data, and the historical behavior data may be converted into a format suitable for visual language model processing.

[0063] Step 220: Add a low-rank adaptive LoRA layer to the visual language model.

[0064] For example, adding a LoRA (Low-Rank Adaptation) layer typically involves adding an additional weight matrix to the linear layer and / or attention layer of the visual language model.

[0065] Step 230: Use the historical behavior data to train the LoRA layer, optimize the parameters of the LoRA layer based on a preset loss function and optimizer, and obtain a trained visual language model.

[0066] The loss function can be selected according to actual needs. The optimizer is used to train the parameters of the LoRA layer, for example including but not limited to the AdamW optimizer.

[0067] The following is an example of the ScreenAI model to expand the fine-tuning process of the visual language model. First, a data storage mechanism is established to record multiple rounds of user interactions with the communication scheduling screen and the data of the question link. Subsequently, the user's historical behavior data is collected based on the data storage mechanism, and the ScreenAI model is fine-tuned using the accumulated historical behavior data. Specifically, a suitable fine-tuning strategy can be determined based on the architecture of the ScreenAI model and the available tools, such as adjusting the parameters of certain layers of the model, adding new layers, or using specific optimization algorithms. In this embodiment, it is selected to add a LoRA layer to the ScreenAI model, which generally involves adding additional weight matrices to the linear layer and / or attention layer of the model. Define the loss function and optimizer, specifically, determine the evaluation indicators for predicting the user's next question, such as accuracy, recall rate, or F1 value (F1 Score), etc. And construct an objective function based on the evaluation indicators to optimize the model during the training process. The optimizer is used to train the parameters of the LoRA layer, for example, including but not limited to the AdamW optimizer.

[0068] Subsequently, the LoRA layer can be trained using historical behavior data to adjust the parameters of the LoRA layer. For example, standard back propagation and gradient descent can be used for training, so that the model can learn the underlying patterns and regularities between the question content. Experiment with different hyperparameters, such as learning rate, number of training rounds, regularization parameters, etc., to find the optimal model configuration. And record performance indicators during training, such as loss and accuracy on the validation set. The performance of the fine-tuned ScreenAI model can then be evaluated on the test set. If the performance is poor, the problem can be analyzed based on the evaluation results, such as insufficient data, inappropriate model architecture, or poor hyperparameter settings. And make corresponding improvements, such as adjusting hyperparameters, adjusting model structure, or collecting more user behavior data for re-fine-tuning until satisfactory results are achieved.

[0069] After completing the model fine-tuning, you can save the parameters of the LoRA layer, integrate the fine-tuned ScreenAI model into the communication system, and configure the initial UI layout and related settings. In this way, by continuously optimizing the model parameters, the ScreenAI model can learn the user's behavior patterns and preferences to improve prediction accuracy.

[0070] In addition, in some embodiments, the prompt template further includes association relationships between multiple data; wherein the association relationships are obtained by mining multiple rounds of interaction data between the user and the communication scheduling screen using association rules in advance; the association relationships include the association relationship between the monitoring data of the communication optical cable and the environmental data, the association relationship between the fault data and the emergency data, and the association relationship between the monitoring data of the communication optical cable and the fault data;

[0071] The generated prompt content carries the association relationship, so that the layout data generated by the visual language model includes first data to be displayed related to the predicted question link, and second data to be displayed related to the first data to be displayed based on the association relationship; wherein the association between the first data to be displayed and the second data to be displayed is represented in the form of a chart and / or a map.

[0072] Association rules are an important technology in data mining, which is used to discover the association relationship between various data. Association rules may include, but are not limited to, Apriori (association rule) algorithm and FP-Growth algorithm, etc.

[0073] When establishing the association relationship between the above-mentioned multiple data, user interaction data, i.e., multiple rounds of interaction data between the user and the communication scheduling screen, can be collected first. Optionally, the user interaction data can be cleaned to remove invalid and erroneous data. The user interaction data can also be converted to make it suitable for association rule mining.

[0074] Subsequently, association rules, such as the Apriori algorithm, can be used to mine the processed user interaction data. Association rules are generated by setting minimum support and minimum confidence thresholds. The generated association rules can then be evaluated to filter out association rules that meet the communication optical cable scheduling scenario. Optionally, the mined association rules can be displayed in a chart, text, or other form for easy understanding and application.

[0075] Exemplarily, based on the mining of association rules, the association relationships between the above-mentioned data can be obtained, such as but not limited to the association relationship between the monitoring data and environmental data of the communication optical cable, the association relationship between the fault data and the emergency data, and the association relationship between the monitoring data and the fault data of the communication optical cable. For example, when a user views the fault condition of the communication optical cable, in most cases the user may also view relevant information charts such as the relevant causes of the fault, the fault location, the common alarm types, the emergency plan, and the safeguard measures. Therefore, the mined association relationship has important guiding significance for how to display detailed content according to the user's interaction.

[0076] Since the prompt template includes the association relationship between multiple data, the generated prompt content also carries the association relationship. Inputting the prompt content into the visual language model can enable the visual language model to predict the first data to be displayed based on the predicted predicted question link, and predict the second data to be displayed that has an association relationship with the first data to be displayed. Among them, the association between the first data to be displayed and the second data to be displayed is represented in the form of a chart and / or a map. The chart is, for example, a line chart, a bar chart, a pie chart, etc. The map is, for example, a heat map, a map with superimposed status data, a map with superimposed fault data, a map with superimposed emergency data, etc., so that the content displayed on the final communication scheduling screen can better meet user needs.

[0077] The communication scheduling screen displays the correlation between the monitoring data and environmental data of the communication optical cable. Specifically, the communication scheduling screen displays the following data: ① A line graph of temperature and optical cable status, which is used to show the relationship between environmental temperature changes and the optical cable core utilization rate and maintenance level; ② A scatter plot of humidity and optical cable status, which is used to show the changes in the optical cable status under different environmental humidity levels; ③ A regional temperature heat map, which displays the temperature distribution of different areas on a map; ④ An optical cable status overlay map, which overlays the optical cable status identification on the heat map, for example, using color coding to indicate the fiber core utilization rate and maintenance level.

[0078] There is a close connection between fault information and emergency data. When an emergency occurs, timely processing of fault information and dispatch of emergency personnel are key. The communication dispatch screen displays the fault data of the communication optical cable in association with the emergency data. Specifically, the communication dispatch screen displays the following data: ① Fault processing progress bar, which is used to display the processing progress after the fault occurs, including the dispatch of emergency personnel; ② Emergency response time bar chart, which is used to display the average response time of different fault types; ③ Fault location and emergency personnel location map, that is, displaying the fault location and the location and movement trajectory of emergency personnel on the map.

[0079] The communication dispatching screen displays the monitoring data and fault data of the communication optical cable in association. Specifically, the communication dispatching screen displays the following data: ① A line graph of fiber core utilization and fault probability, which is used to display the relationship between fiber core utilization and fault probability; ② A graph of optical cable status and fault statistics, which is used to display fault statistics under different optical cable statuses; ③ A high-load warning, including when the fiber core utilization exceeds the preset utilization threshold, such as 80%, the high-load area on the map turns red, and the status of the relevant optical cable is marked. At the same time, the fault information of the optical cable is updated in real time, and the fault handling progress and warning information are displayed.

[0080] This embodiment can realize the association display between data, laying a foundation for the subsequent stratification and dynamic update of warning content.

[0081] In addition, based on any of the above embodiments, the prompt content further includes one or more prompt information, so that the layout data generated by the visual language model also includes third display data obtained based on the prompt information;

[0082] As an example, the prompt information includes prompt information for prompting data related to the current question content mined from a preset database. In this way, in addition to considering the predicted question link, the visual language model also uses data mining means to mine data related to the current question content from the preset database as the third content to be displayed and displayed together in the communication scheduling screen.

[0083] As an example, the prompt information includes prompt information for prompting to call a multimodal pre-training model to obtain fused data. Specifically, the text, image, video and other data displayed on the communication scheduling screen history can be collected, and the data can be pre-processed and labeled so that different types of data can be effectively associated. Subsequently, a multimodal pre-training model, such as a CLIP (Contrastive Language-Image Pre-training) model is used for training to map text, image, video and other data to the same vector space, so that the multimodal pre-training model can perform multimodal learning. After obtaining the multimodal pre-training model, in addition to predicting the question link, the visual language model also calls the multimodal pre-training model to fuse the multimodal data related to the current question content obtained from multiple data sources, for example, mapping them to the same vector space, and the obtained fused data is the third display data. In this way, different types of multimodal information related to the current question content can be integrated and displayed on the communication scheduling screen, realizing the integrated display of different data types (including text, images, videos, sensor data) on the large screen, ensuring that users can see the most relevant content when interacting, and improving the usability of the communication scheduling screen.

[0084] As an example, the prompt information includes prompt information for prompting to call the knowledge graph. Specifically, the relevant entities of the optical cable and the relationships between the entities can be collected, and the knowledge graph of the optical cable can be constructed using multiple entities and their relationships. The entities include, for example, but are not limited to, optical cable types, fault types, maintenance steps, etc. In this way, in addition to predicting the question link, the visual language model also uses an inference engine (such as an OWL inference engine) to process the current question content, and derives data related to the current question content through the relationship chain in the knowledge graph as the third content to be displayed in the communication scheduling screen. In this way, a question-and-answer system can be created that can understand user questions and provide accurate answers, ensuring that users can quickly obtain the required information. And support real-time updates of the knowledge graph to ensure that the displayed information is always the latest.

[0085] As another example, the prompt information includes prompt information for prompting to call the adaptive model. Specifically, professional question and answer data in the field of optical cables can be collected and annotated to ensure that the system can adapt to the needs of different scenarios. The scene adaptive model is then trained using a contrastive learning method so that it can better distinguish the subtle semantic differences in different scenarios. Subsequently, the trained scene adaptive model can be used to automatically adjust the query parsing logic according to the context of the current question content and the question scene, so that the displayed content is more relevant. In this way, the visual language model can obtain the third content to be displayed based on the context of the current question content and / or the question scene by calling the scene adaptive model, so that the displayed content is more relevant. In addition, the system can also use domain adaptive technology and contrastive learning to continuously adjust and optimize the scene adaptive model according to the user's interactive behavior (such as clicks, browsing time, etc.), so that the communication scheduling screen can understand and adapt to changes in user queries in specific optical cable scenarios.

[0086] Finally, the above-mentioned data mining methods, multimodal pre-trained models, knowledge graphs, and adaptive models can be integrated into a complete communication system in the form of software modules, so that the visual language model can generate layout data by calling corresponding auxiliary methods through the API (Application Programming Interface).

[0087] This embodiment is described by the following examples:

[0088] If the current question asked by the user is "See what high-priority alarms are currently available?", the visual language model will call out high-priority alarm information for display. As an example, if the prompt content includes prompt information for prompting data related to the current question content mined from a preset database, the visual language model will call data mining methods to quickly extract high-priority alarm information related to the user's query from the huge amount of alarm data. At the same time, the mining algorithm will evaluate the priority, history, and potential impact of all alarms, and display these data together on the communication scheduling screen.

[0089] As an example, if the prompt content includes prompt information for calling a multimodal pre-trained model to obtain fused data, the visual language model will call the multimodal pre-trained model to fuse multiple modal data such as alarm data, geographic information, and video surveillance to form a unified display. For example, the location of a high-priority alarm can be displayed through a combination of maps, real-time video, and sensor data.

[0090] As an example, if the prompt content includes prompt information for calling the knowledge graph, the visual language model calls the knowledge graph to understand the relationship between the alarm data and other relevant information (such as historical data, fault causes, treatment plans, etc.), and provides accurate display data based on these relationships.

[0091] As an example, if the prompt content includes prompt information for calling the adaptive model, the visual language model will automatically adjust the query parsing logic based on the alarms queried by the user, making the result display more relevant. For example, when the user queries "Where is the latest alarm?", the system will prioritize displaying the geographic location information.

[0092] It can be seen that by combining various prompt information, relevant detailed data can be presented in a timely manner according to the user's current question content, allowing users to understand comprehensive information related to the question in a short time through simple operations. In this way, a highly intelligent visual language model system is constructed, which can accurately understand and display user needs, making the management of the communication scheduling screen more efficient and convenient.

[0093] In addition, the communication scheduling screen provided by the present application also has a scene adaptation function. Specifically, the visual language model can adjust the display data corresponding to the scene on the communication scheduling screen in different scenes. Thus, based on any of the above embodiments, the prompt content also includes the current scene information input by the user; and the data to be displayed is related to the current scene information.

[0094] As an example, the current scene includes a daily communication scheduling scene. The prompt content includes information of the daily communication scheduling scene, and the visual language model adjusts the data to be displayed based on the daily communication scheduling scene. Specifically, in the daily communication scheduling scene, the data to be displayed includes monitoring data, environmental data, and fault data.

[0095] For example, in daily communication scheduling scenarios, the status of communication optical cables directly affects the communication quality and scheduling efficiency. Displaying the current status of each optical cable and its historical trend helps to identify potential maintenance needs and optimize communication scheduling. Based on this, the monitoring data displayed in the daily communication scheduling scenario includes visualization information of the communication route, which includes the length of the optical cable, asset level, maintenance level, fiber core utilization rate, business carrying status, etc. Optionally, based on the established association relationship, communication optical cables with high fiber core utilization rate and low maintenance level that may fail can also be monitored in detail.

[0096] For another example, in daily communication scheduling scenarios, dynamic environmental conditions will affect the performance of communication optical cables. By monitoring environmental data, environmental anomalies can be discovered and handled in a timely manner. Based on this, the environmental data displayed in daily communication scheduling scenarios include ambient temperature, ambient voltage, ambient humidity and their changing trends. Optionally, based on the constructed association relationship, it can be known that abnormal environmental parameters (such as high temperature and high humidity) will cause changes in the state of the optical cable, so it can be displayed in combination with the monitoring data of the communication optical cable.

[0097] For another example, in the daily communication scheduling scenario, fault data is closely related to the communication optical cable and environmental status. By monitoring the communication optical cable for faults, the problem can be quickly located and solved. Based on this, the fault data displayed in the daily communication scheduling scenario includes the fault title, location, cause, type, response time, processing progress, etc. Optionally, based on the established association relationship, the monitoring data and environmental data associated with the fault data of the faulty optical cable can also be associated and displayed.

[0098] As another example, the current scene includes an emergency command scene. The prompt content includes information about the emergency command scene, and the visual language model will adjust the data to be displayed based on the emergency command scene. Specifically, in the emergency command scene, the data to be displayed includes the optical cable topology data of the emergency area, the alarm data in the emergency area, and the security data of the emergency area.

[0099] For example, the optical cable topology data of the emergency area displayed in the emergency command scene can be an optical cable topology map. Optionally, based on the established association relationship, it can be known that the status of the optical cable in the emergency area is closely related to the emergency response, so it can be displayed in combination with the monitoring data of the communication optical cable and the emergency support data.

[0100] For another example, in an emergency command scenario, alarm information can help quickly identify the problem. Based on this, the alarm data in the emergency area displayed in the emergency command scenario can include the alarm level, location, time, etc. Optionally, based on the established association relationship, the alarm information is associated with the monitoring data of the communication optical cable and the emergency support data. Therefore, in the emergency response, the monitoring data of the communication optical cable in the alarm area and the location of the emergency personnel can be displayed in combination.

[0101] For another example, in an emergency command scenario, emergency support records can ensure the transparency and traceability of emergency response, which helps to summarize and optimize emergency plans afterwards. Based on this, the support data in the emergency area displayed in the emergency command scenario includes the location of emergency personnel, dispatch status, processing progress, etc. Optionally, based on the established association relationship, the emergency support data can be displayed in association with the fault data and the monitoring data of the communication optical cable to ensure the transparency and traceability of the emergency response.

[0102] As another example, the current scene includes a holiday theme scene. The prompt content includes information about the holiday theme scene, and the visual language model will adjust the data to be displayed based on the holiday theme scene. Specifically, in the holiday theme scene, the data to be displayed includes: monitoring data of the target communication optical cable in the holiday protection area, alarm information in the holiday protection area, and management information of the holiday protection area.

[0103] For example, in the holiday theme scenario, since holidays are peak communication periods, monitoring the operating status of communication optical cables can ensure that the communication system operates stably under high load. Based on this, in the holiday theme scenario, the monitoring data of the target communication optical cable in the holiday guarantee area displayed includes the cable length, fiber core utilization rate, business carrying status, etc. of the target communication optical cable. Optionally, based on the established association relationship, since the high load status during holidays needs to be monitored, the impact of the environment on the status of the communication optical cable can be displayed in combination with the display of environmental data.

[0104] For another example, in a holiday-themed scenario, smooth communication during holidays can be ensured by timely processing of alarm information. Based on this, in a holiday-themed scenario, the alarm information displayed in the holiday guarantee area includes the alarm level, location, time, etc. Optionally, based on the established association relationship, it can be seen that the alarm information is associated with the monitoring data and fault data of the communication optical cable, and the high load state of the communication optical cable during holidays may lead to frequent alarms, so the monitoring data, fault data and alarm data of the communication optical cable can be combined for key display.

[0105] For another example, in a holiday-themed scenario, the efficiency and pertinence of holiday support work can be improved through customized templates. Based on this, in a holiday-themed scenario, the management information of the holiday support area displayed can include a customized management template, which includes common alarm types, emergency plans, support measures, etc. Optionally, based on the constructed association relationship, the customized template needs to be combined with historical data and current monitoring data to ensure efficient and orderly support work during holidays.

[0106] As another example, the current scene includes a reporting scene. The prompt content includes prompt information of the reporting scene, and the visual language model adjusts the data to be displayed based on the reporting scene. Specifically, in the reporting scene, the data to be displayed includes: indicator statistics, indicator analysis data, and business guarantee status data of the communication optical cable.

[0107] For example, in the reporting scenario, the operation status of the communication system can be fully understood through summary display, which is convenient for reporting to the management and decision-making. Based on this, the indicator statistics of the communication optical cable displayed in the reporting scenario include optical cable status statistics, fault situation statistics, emergency response record statistics, etc. Optionally, the indicator statistics can also be associated with the data to be displayed in the daily communication dispatch scenario, emergency command scenario and holiday special scenario to fully display the operation status of the system.

[0108] For another example, in a reporting scenario, the efficiency and effectiveness of the communication system operation can be evaluated by analyzing the performance indicators, providing data support for subsequent improvements. Based on this, the indicator analysis data displayed in the reporting scenario refers to the statistical analysis results of various performance indicators, including fiber core utilization rate, fault handling efficiency, emergency response time, etc. Optionally, the performance indicator analysis data can also be displayed in combination with historical data and the current status of the communication optical cable to evaluate the efficiency and effectiveness of the system operation.

[0109] For another example, in a reporting scenario, the service assurance situation can directly reflect the stability and reliability of the communication system, and timely discover and solve problems to ensure the normal operation of the business. Based on this, the service assurance situation data displayed in the reporting scenario includes business volume, service quality, failure rate, etc. In addition, the service assurance situation data can also be associated with the monitoring data, failure data and environmental data of the communication optical cable to ensure the stable operation of the business.

[0110] In addition, when facing the joint display of multiple types of data, the layout of multiple types of data in the communication scheduling screen can also be determined according to the following rules. The rules include:

[0111] 1) Prioritize display of important information: Important information (such as high-priority alarms, high-load optical cables, and emergency response data) is displayed in an important display area of ​​the communication scheduling screen. The important display area is, for example, the center or top of the screen, and its area ratio on the screen is not less than a preset ratio threshold, so that the user's attention is focused on the important display area.

[0112] 2) Tight display of related data: Various types of data with related relationships are displayed adjacent to each other in the communication scheduling screen, which is convenient for comparison and analysis.

[0113] 3) Display data in chronological order. For example, data with chronological order such as fault handling progress and emergency response records can be displayed in the form of a timeline or carousel.

[0114] 4) Display data in order of priority. For example, data such as alarm information and fault information can be prioritized and sorted using color coding and lists, with high-priority data displayed first.

[0115] Through the above display logic and display method, input support can be provided for the visual language model, and dynamic adjustment and personalized display can be performed to ensure that the system can flexibly respond to different needs and warning scenarios.

[0116] In addition, in addition to making different layouts of the communication scheduling screen according to different scenarios, the visual language model can also perform screen information understanding, question answering, UI navigation and content summary based on the current display content of the communication scheduling screen in different scenarios.

[0117] Taking the ScreenAI model as an example, screen information understanding means that the ScreenAI model can identify and understand the elements of the currently displayed content and the content of infographics in the communication scheduling screen, including the type, location and relationship between each other. Question answering means that the ScreenAI model can understand the visual information obtained from the currently displayed content of the communication scheduling screen and answer questions about the displayed content, such as the content of infographics. UI navigation means that the ScreenAI model can interpret the navigation instructions entered by the user (such as the "return" instruction) and identify the appropriate UI elements from the currently displayed content for interaction, understand the user's intentions and be able to navigate accurately in the interface. Content summary means that the ScreenAI model can concisely summarize the currently displayed content, as well as refine and summarize the core points of the currently displayed content.

[0118] In order to enable the visual language model to realize screen information understanding, question answering, UI navigation and content summary functions, the visual language model can be trained using screenshots carrying target information. The target information includes: question and answer pairs about the screenshot, logical relationship information between elements in the screenshot, navigation operation information, and summary information. The specific training process is as follows:

[0119] First, we collect training data. Specifically, we collect screenshots of various UI elements, infographics, etc. displayed on the communication scheduling screen, and mark the location, type, and relationship of each element. At the same time, we prepare corresponding question-answer pairs for each screenshot, which cover the content of the UI elements and infographics in the screenshot.

[0120] Subsequently, computer vision technology is used to train the visual language model to identify UI elements and infographics on the screen. Specifically, semantic segmentation, object detection and other methods can be used to identify element types and locations. The logical relationship information carried by the screenshot can be used to train the visual language model to understand the logical relationship between elements (such as the relationship between a button and a text box), so that the visual language model can realize the screen information understanding function.

[0121] In addition, natural language processing technology can be used to train visual language models to understand questions, and by combining screen content and question pairs, the visual language model can be trained to generate answers to achieve question-answering functions.

[0122] In addition, the navigation operation information carried in the screenshot can train the visual language model to understand natural language instructions, such as the "return" instruction, the "open settings" instruction, etc., so that the visual language model can identify UI elements related to user instructions and realize UI navigation function.

[0123] Furthermore, natural language generation technology can be used to train a visual language model based on the summary information carried by the screenshot to extract key information from the screen content, so that the visual language model can generate concise and clear summaries, thereby realizing the content summarization function.

[0124] Finally, the trained visual model can be encapsulated as an API interface, and a user interface can be designed to allow users to enter questions or instructions in the user interface. By calling the trained visual language model through the API, screen information understanding, question answering, UI navigation and content summary can be performed based on the current display content of the communication scheduling screen in different scenarios.

[0125] Based on this, the user can also input user instructions, which include one or more of screen information understanding instructions, question and answer instructions, navigation instructions, and summary instructions. In this way, since the prompt content of the visual language model includes the current scene information input by the user and the user instruction, the visual language model can execute the user instruction on the data to be displayed corresponding to the current scene information, including one or more of screen information understanding, answering questions, navigation operations, and generating summaries.

[0126] Take the daily communication scheduling scenario as an example. As an example, in the daily communication scheduling scenario, if the communication scheduling screen currently displays the monitoring data of the communication optical cable, the trained visual language model can be called to understand the screen information. Specifically, the visual language model can recognize and understand the visual information of the currently displayed communication optical cable, including the length of the optical cable, asset level, maintenance level, fiber core utilization rate, business carrying status, etc. At the same time, the visual language model can also understand logical relationships, specifically the status of different communication optical cables and the correlation between them. For example, optical cables with high utilization rates may require more maintenance.

[0127] In addition, the trained visual language model can be called to answer questions. For example, if the user's question is "Which optical cable has the highest fiber core utilization rate?", the visual language model can answer based on the monitoring data of the currently displayed communication optical cable, such as "According to the current data, the fiber core utilization rate of cable A is 85%, which is the highest among all optical cables."

[0128] In addition, the trained visual language model can be called for content summarization. For example, based on the monitoring data of the currently displayed communication optical cable, a content summary can be generated: "Today, the fiber core utilization rate of two optical cables exceeds 80%, and one of them is optical cable B with a lower maintenance level, which requires special attention."

[0129] In addition, the trained visual language model can be called for UI navigation. For example, if the user enters the navigation command "show optical cables with a low maintenance level in the last month", the visual language model can parse the navigation command and display the optical cables with a low maintenance level in the last month based on the monitoring data of the currently displayed communication optical cables that meet the navigation command content, that is, a list of optical cables with a low maintenance level in the last month.

[0130] Take the emergency command scenario as an example. In the emergency command scenario, if the communication dispatch screen currently displays the optical cable topology data of the emergency area, the trained visual language model can be called to understand the screen information. Specifically, the visual language model can automatically identify the various nodes, connection lines, and possible fault points in the communication optical cable topology diagram displayed on the communication dispatch screen. It can also understand the visual elements in the topology diagram and convert them into structured data, such as the location information of the starting point, end point, and intermediate connection point of the optical cable.

[0131] In addition, the trained visual language model can be used to answer questions about the status of optical cables based on the optical cable topology data of the currently displayed emergency area, such as answering questions such as "Which optical cables are in normal operation?" or "Where is the nearest fault point?"

[0132] In addition, the trained visual language model can be used for content summarization. For example, a concise summary of a communication optical cable topology map can be generated, highlighting important information such as the status of key nodes, potential risk points, etc.

[0133] Furthermore, the trained visual language model can be used for UI navigation. For example, the topology map on the screen can be navigated according to the user's navigation instructions in voice or text form, such as zooming in on a specific area or displaying the status of a specific line.

[0134] Taking the holiday theme scene as an example, in the holiday theme scene, if the communication scheduling screen currently displays the alarm information in the holiday guarantee area, the trained visual language model can be called to understand the screen information. Specifically, the visual language model can understand the alarm information displayed on the screen, including the alarm level, location, time, etc., and can parse the color coding and map markings to quickly locate the problem.

[0135] In addition, the trained visual language model can be used to answer questions about alarms based on the alarm information in the currently displayed holiday protection area. For example, answer questions such as "What are the most frequent alarm types during holidays?" or "Where does the latest alarm appear?"

[0136] In addition, the trained visual language model can be used for content summarization. For example, a concise summary report can be generated to outline the key points of alarm information during holidays, such as the number of alarms, type distribution, and processing progress.

[0137] Furthermore, the trained visual language model can be used for UI navigation. For example, the user can navigate to the indicated alarm details according to the user's navigation instructions, such as viewing the processing progress of a specific alarm or displaying the alarm records of the same location.

[0138] Taking the reporting scenario as an example, in the reporting scenario, if the communication scheduling screen currently displays the indicator statistics of the communication optical cable, the trained visual language model can be called to understand the screen information. Specifically, the visual language model can identify the various indicator statistics displayed on the screen, including the optical cable status, fault conditions, emergency response records, etc., and understand the logical relationship between these data.

[0139] In addition, the trained visual language model can be called to answer questions about statistics based on the currently displayed communication optical cable indicator statistics, such as "How many optical cable failures were there this month?" or "What was the average emergency response time last quarter?"

[0140] In addition, the trained visual language model can be used for content summarization. For example, a concise summary report can be generated to outline the key statistics of various indicators, such as the stability of the optical cable status, the trend of fault conditions, and the effectiveness of emergency response.

[0141] In addition, the trained visual language model can be used for UI navigation. For example, it can navigate to the indicated statistical data view according to the user's navigation instructions, such as switching to view statistical data of different time periods or adjusting the chart type to display the data more clearly.

[0142] It can be seen that this embodiment uses the visual language model to implement screen information understanding, question answering, UI navigation and content summary functions for different display data in different scenarios, which greatly facilitates users to use and manage the communication scheduling screen.

[0143] In addition, in some cases, the user's questions may be mixed with colloquial factors, resulting in the visual language model failing to understand the user's intentions well. To solve this problem, this embodiment proposes that the prompt content of the visual language model can also include the current question content and question structuring task information, and the visual language model converts the current question content into a structured question based on the question structuring task information and generates the answer content.

[0144] This embodiment defines a question structuring task, which is used to perform structured analysis on the user's current question content, so that the visual language model can understand the user's intention from the converted structured question and generate answer content.

[0145] Specifically, the question structuring tasks include determining the scope of the question topic, determining the type of data involved in the question, constructing the question framework, refining the question elements, and setting the answer format.

[0146] Determining the scope of the question topic refers to determining whether the current question content is centered around the communication optical cable related data displayed in the communication scheduling screen.

[0147] Determining the data type involved in the question refers to classifying and organizing the data type corresponding to the current question content, which includes optical cable core utilization rate, environmental parameters and their impact on optical cable performance, computer room temperature thresholds and response measures, faulty optical cable information, and unresolved optical cable faults, etc.

[0148] The so-called question framework construction means starting from different data categories, transforming the current question content into one or more of general questions (such as "Please explain the specific situation of [data category]"), specific numerical questions (such as "What is the numerical value of [data category]"), comparative questions (such as "Compared with [time period], how has [data category] changed"), cause analysis questions (such as "What is the reason for the change of [data category]"), impact assessment questions (such as "What impact has [data category] had on [related aspects]"), prediction questions (such as "What may happen to [data category] in the future [time period]"), and solution questions (such as "Are measures taken for [problem]")? The content in brackets [] is extracted from the current question content.

[0149] The so-called refinement of problem elements refers to the refinement of time, space, objects, conditions, causes and other elements involved in the problem. Specifically, refinement of time elements refers to the refinement of specific time periods, such as accurate to years, months, days, hours, minutes and seconds, and the determination of time ranges in the past, present or future. Refinement of spatial elements refers to the refinement of specific geographical locations or regional scopes, and the determination of specific spatial objects such as places and buildings involved. Refinement of object elements refers to clarifying the specific objects targeted by the problem, whether they are people, objects, events or concepts, and a more detailed description of the characteristics and attributes of the objects. Refinement of conditional elements refers to determining the preconditions, restrictions or assumptions contained in the problem, such as resource availability, environmental factors, policy regulations, etc.

[0150] Setting the answer format refers to specifying the format of the answer, for example, specifying that the answer content of the question be output in the form of a list, a chart, or a detailed text description.

[0151] For example, here is an example of a prompt that defines a structured task with a question:

[0152] “Question topic scope: Optical cable related data in the communication monitoring large screen system.

[0153] Data type classification: 1. Optical cable core utilization rate, 2. Environmental parameters and their impact on optical cable performance, 3. Equipment room temperature thresholds and countermeasures, 4. Faulty optical cable information, 5. Unresolved optical cable faults.

[0154] Example: Problem framework: 1. Optical cable core utilization rate

[0155] -General question: Please explain the specific situation of fiber optic cable core utilization rate.

[0156] -Specific numerical question: What is the current value of optical cable core utilization rate?

[0157] -Comparison question: How has the fiber optic cable core utilization rate changed compared to last week?

[0158] -Cause analysis question: What is the reason for the reduced utilization rate of optical cable cores?

[0159] - Impact assessment question: What impact does excessive fiber core utilization have on communication quality?

[0160] -Forecast question: What is the likely rate of fiber optic cable core utilization in the next week?

[0161] -Solution measures: Have measures been taken to address the low utilization rate of optical cable cores?

[0162] Refine the problem elements: 1. Time element:

[0163] - Past: For example, "What was the fiber core utilization rate of optical cables in August last year?"

[0164] -Now: "What is the temperature in the computer room at this moment?"

[0165] -Future: "Can the faulty optical cable be repaired within the next three days?"

[0166] 2. Spatial elements:

[0167] - Geographically specific: "What is the humidity in the equipment room in area A?"

[0168] -Regional scope: "What is the number of fiber optic cable failures in the entire campus?"

[0169] 3. Object elements:

[0170] -Specific: "What is the fiber core utilization of a particular fiber optic cable?"

[0171] -Object characteristic attribute: "What is the failure frequency of optical cables with high maintenance level?"

[0172] 4.Conditional elements:

[0173] -Prerequisite: "How to improve the efficiency of optical cable maintenance when resources are limited?"

[0174] - Constraints: "How will the cable perform if the temperature does not exceed 30 degrees?"

[0175] 5.Cause elements:

[0176] - Direct cause: "What is the direct cause of this fiber optic cable failure?"

[0177] -Root cause: "What is the root cause of the long-term low utilization rate of optical cable cores?"

[0178] Set format requirements:

[0179] The responses should be presented in a list format, with each answer including the following:

[0180] 1. Description of the problem, 2. Specific answer, 3. Relevant data or charts"

[0181] By converting the current provided content input by the user into a structured question through a question structuring task, it can be ensured that the question focuses on the display content of the communication scheduling screen and avoids deviation from the topic. At the same time, the question framework covers different types of inquiries, and the comprehensiveness of the question can be ensured by refining the question elements. And by constructing different types of question frameworks, the visual language model can better understand the user's question intention. By clarifying elements such as time, space, and objects, it helps the visual language model to give more specific operational suggestions or solutions. By refining the question elements, misunderstandings in communication can be reduced and more effective information exchange can be promoted. In summary, through this embodiment, not only the accuracy and effectiveness of questions can be improved, but also the respondents can be encouraged to provide more targeted and practical answers, thereby better serving the management and decision-making process of the communication monitoring large screen system.

[0182] In addition, the visual language model can also layout multiple communication scheduling screens at the same time. And each communication scheduling screen can be located in a different geographical location. For example, the visual language model can layout the communication scheduling screen located in City A and the communication scheduling screen located in City B at the same time. In some possible scenarios, multiple regions may need to jointly control communication optical cables. At this time, the personnel in each region will jointly control the personnel in other regions on the communication scheduling screen in their own region. However, the communication scheduling screens in different regions may display different content, which will cause the messages of the personnel in the two regions to be unable to be synchronized, which brings great inconvenience to the joint control. The following three scenarios are listed to illustrate:

[0183] Scenario 1: When two communication scheduling screens assist in processing early warnings, one communication scheduling screen may display the optical cable fault situation, for example, the communication scheduling screen is displayed centered on a chart of the optical cable fault situation. The other communication scheduling screen displays the current alarm situation of the optical cable. At this time, one person is concerned about the fault details, and the other is concerned about the alarm status. If the display content between the two communication scheduling screens is not synchronized in time, it will cause information deviation when the two people discuss the impact of the fault and alarm processing. And when two people discuss based on different display content, if they do not actively mention the display content of the other party's screen during communication, some key information may be missed, thereby affecting the accuracy of decision-making. In addition, due to the different display content of the two communication scheduling screens, it may also lead to misjudgment of the urgency of a certain fault when evaluating the processing priority.

[0184] Scenario 2: When two communication scheduling screens assist in handling warnings, communication scheduling screen A focuses on the transmission network topology logic diagram and network element alarm information, while communication scheduling screen B focuses on displaying alarm type pie charts and the number of unprocessed alarms in different regions. In this case, due to the different regions and alarm types focused on by the two communication scheduling screens, it may be difficult for users to synchronously understand the overall warning situation during communication, such as being unable to quickly determine the severity and specific impact of an alarm in a certain region. In addition, since communication scheduling screen A displays the network topology and specific node locations of alarms, while communication scheduling screen B displays alarm information classified by region, the alarm types and numbers in different regions may be displayed as statistical pie charts in screen B, but they are not easy to intuitively perceive in communication scheduling screen A, which can easily cause regional information errors. In addition, communication scheduling screen A focuses on displaying network element alarm information in the network topology, but the severity of the alarm and the number of unprocessed alarms are reflected in communication scheduling screen B. This separation may cause users to deviate when evaluating the severity of alarms, especially when decisions need to be made quickly.

[0185] In order to solve the problem of inconsistent display of multiple communication scheduling screens, this embodiment proposes that a multi-screen display content monitoring task can be input in the prompt template, and the prompt content includes the multi-screen display content monitoring task, so that the visual language model performs difference detection on the display content of at least two communication scheduling screens based on the multi-screen display content monitoring task; wherein the difference detection includes time synchronization difference detection, monitoring value difference detection, and image content difference detection. If the difference detection indicates that the display content difference of at least two communication scheduling screens is greater than a preset difference threshold, a synchronization instruction is output to at least two communication scheduling screens, so that the communication scheduling screens consume and read data from the same message queue and display it.

[0186] In order to enable the visual language model to detect the difference between the display contents of multiple communication scheduling screens and complete the data update, the visual language model may be trained first. The training process of the visual language model is introduced below.

[0187] First, a large number of difference data sets of screen display content can be collected, and the difference data sets include time lag type data, numerical difference type data, image difference type data, etc. And each type of difference data includes screen data of two communication scheduling screens and processing operations on the two communication scheduling screens. Among them, the screen data includes timestamps, alarm numbers, and image snapshots, and the processing operations include synchronization operations and data update operations. Subsequently, a training data set can be generated using the collected difference data sets, and the optimal processing method corresponding to each difference type can be marked. Subsequently, the training data set can be used to train the visual language model. In this way, the visual language model can not only understand the differences between the screen display contents during training, but also make optimal processing based on the differences.

[0188] Here are three prompt template examples for training visual language models:

[0189] Example 1: "It is detected that there is a significant image difference between screen A and screen B in the image display of optical cable t: screen A displays the service carrying bar graph and fiber core utilization rate graph of optical cable t. Screen B displays the global topology and alarm location distribution graph of optical cable t.

[0190] The synchronization strategy used for processing and the optimal processing method:

[0191] Q1: Do I need to obtain the HTML format of screens A and B and parse the HTML? A1: The visual language model obtains the HTML content of screens A and B through the API, uses JavaScript to parse the HTML content, and extracts the data display area that needs to be synchronized.

[0192] Q2: When the image difference detection SSIM is lower than the set threshold of 0.8, is it necessary to output the HTML format of screen AB and synchronize the image information of screen A and screen B? A2: Yes. The visual language model calls the API of the background server to obtain the data that needs to be synchronized. For example, a GET request is sent to the API to obtain the service carrying and alarm data of cable t, and the data of screen B (such as alarm distribution map, global topology map, etc.) is obtained from the API, and the data is injected into the corresponding HTML elements of screen A and screen B.

[0193] Q3: Is it necessary to synchronize the information of screen A and screen B based on real-time prediction technology? A3: Yes. Based on the operation modes of operator A and operator B, it is predicted that operator A may further check the alarm location, and operator B may pay attention to the change of fiber core utilization rate. Therefore, the information of the two screens is synchronized in advance, and the prediction data is injected into the corresponding HTML elements of screen A and screen B so that screen A and screen B can display the alarm location and fiber core utilization rate at the same time.

[0194] Q4: Is it necessary to use multimodal data fusion technology to synchronously display the fiber core utilization rate, alarm information and dynamic environment monitoring data? A4: Yes. The system should fuse the fiber core utilization rate, alarm information and dynamic environment monitoring data, and inject the data into the corresponding HTML elements of screen A and screen B for simultaneous display. In this way, the operator can see the high-load operation status of the optical cable, the alarm location and the impact of the external environment on both screens at the same time to avoid misunderstanding.

[0195] Q5: Is it necessary to analyze the correlation between historical alarms and the current status through knowledge graphs and display the cause of the alarm? A5: Yes. The system should automatically analyze the correlation between the historical alarms of the optical cable and the current status, infer the cause of the alarm, and inject the alarm background and related network element status into the corresponding HTML elements of screen A and screen B to help operators understand the detailed background of the current alarm.

[0196] Through the above training process, the trained visual language model has the ability to handle various types of screen display content differences, which can improve its adaptability to different scenarios. This enables the visual language model to perform well in a variety of application scenarios. In addition, by designing specific training prompts, the visual language model can more accurately identify and process the differences in content displayed between screens, reduce the possibility of misunderstanding and error processing, and thus improve the overall accuracy of the system.

[0197] Prompts designed for different types of screens can help visual language models respond more quickly, such as synchronizing data in a timely manner to ensure consistency of information, thereby reducing operational problems caused by delays or errors.

[0198] Using the trained visual language model, a multi-screen display content monitoring task can be performed. Exemplarily, the multi-screen display content monitoring task is actually a dynamic error detection task. Based on the multi-screen display content monitoring task, the visual language model can continuously monitor the content displayed on at least two communication scheduling screens and continuously detect the differences between their displayed contents.

[0199] In order to realize the multi-screen display content monitoring task (dynamic error detection task), first, multi-level error thresholds can be set, such as time lag threshold, numerical difference threshold and image difference threshold, and these detection thresholds can be initialized. Among them, the time lag threshold is defined as the time synchronization error between the two communication scheduling screens, in milliseconds. The numerical difference threshold is used to monitor the difference in numerical information in the display content, such as the number of alarms, the number of fault statistics, etc. The numerical difference threshold can be a percentage error or an absolute numerical error. The image difference threshold can be defined by the structural similarity index (SSIM) or the mean square error (MSE) between the two images.

[0200] After completing the initialization and configuration of the detection threshold, the visual language model can perform the above-mentioned multi-screen display content monitoring task, that is, perform time synchronization difference detection, monitoring value difference detection, and image content difference detection on the display content of at least two communication scheduling screens.

[0201] For example, deep learning real-time monitoring algorithms, such as convolutional neural networks (CNN) and recurrent neural networks (RNN), can be used to continuously monitor the display content of multiple communication scheduling screens. In addition, optical character recognition (OCR) technology and image comparison algorithms can be used to regularly capture and analyze display content to detect differences in numerical, text or image displays.

[0202] Specifically, when performing time synchronization difference detection, the update time of each piece of data on multiple communication scheduling screens can be marked by a timestamp, and the time difference Δt is periodically calculated based on the timestamp of each communication scheduling screen. The time difference is compared with a preset time lag threshold. If the time difference exceeds the time lag threshold, the data synchronization mechanism is triggered.

[0203] When performing monitoring numerical difference detection, difference calculation or percentage deviation calculation can be used to characterize the numerical difference. The difference calculation is, for example, the absolute value of the difference between two values ​​of the same parameter in two communication scheduling screens, denoted as |AB|, where A and B are the values ​​of the same parameter on the two communication scheduling screens. The percentage deviation calculation is, for example, the absolute value of the difference between two values ​​of the same parameter in two communication scheduling screens and the percentage of one of the values, denoted as |(AB) / B|*100%. Subsequently, the numerical difference can be compared with a preset numerical difference threshold. If the numerical difference exceeds the numerical difference threshold, the data synchronization mechanism is triggered.

[0204] When performing image content difference detection, snapshots of the display contents of the two communication scheduling screens can be captured, and the two snapshots can be compared using an image processing algorithm (such as SSIM or MSE). For example, the output value of SSIM is between [0,1], and the output value is negatively correlated with the image content difference, that is, the closer the output value is to 1, the smaller the image content difference between the two snapshots is, and the more similar the display content is. In this way, the SSIM value can be compared with the preset image difference threshold. If the SSIM value is lower than the image difference threshold, it is marked as an error and the data synchronization mechanism is triggered.

[0205] Subsequently, once it is detected that the display content difference between the two communication scheduling screens exceeds the difference threshold, the communication system will trigger an automatic correction process based on the data synchronization mechanism. For example, based on a preset priority, such as the timeliness of the alarm information, the key data may be synchronized and updated first. The automatic correction process is triggered based on the data synchronization mechanism, for example, including sending a synchronization instruction to the two communication scheduling screens whose display content difference is greater than the difference threshold.

[0206] Error capture can be integrated into the prompt template and call the visual language model to synchronize the displayed content.

[0207] As an implementation method, the data synchronization mechanism can be triggered through a rule-based engine. That is, a rule-based engine is designed to dynamically select and apply the appropriate data synchronization mechanism, so as to match the error size with the data synchronization mechanism.

[0208] Based on this, the communication system can use the visual language model to periodically or in real time perform difference detection on multiple communication scheduling screens, and when the difference detection indicates that the display content difference between at least two communication scheduling screens is greater than a difference threshold, the difference value is matched with the predefined rules and the corresponding data synchronization mechanism is executed, such as one or more of data retransmission, image re-rendering, regeneration of layout data, and issuing an alarm.

[0209] For example, in scenario 1, the corresponding prompt content may be set as:

[0210] "Suppose you are a dynamic error detection and content synchronization system. The tasks performed include: detecting that screen A displays optical cable fault data, screen B displays alarm status information, and there are differences in the content of the two screens. Specifically, screen A displays a chart of the optical cable fault situation; screen B displays the current alarm status of the optical cable. The current error detection is as follows: Time lag detection: The timestamp T1 of screen A and the timestamp T2 of screen B differ by several hours, and the calculated time difference Δt is T1-T2. Numerical difference detection: The fiber core utilization rate on screen A is A, the number of alarms on screen B is B, and the numerical difference is |AB|. Image difference detection: The snapshot SSIM value of screen A and screen B is 0.7 (the ideal value is close to 1), and the displayed content is significantly different.

[0211] The output format is: Please generate synchronization instructions to ensure that the information on screen A and screen B is synchronized. The specific synchronization strategies are as follows: 1. Based on real-time prediction technology, predict the user's next operation: By analyzing the historical operation modes of operator A and operator B, predict their possible next operation needs. For example, it is predicted that operator A may further analyze the impact area of ​​the optical cable fault, while operator B may view the historical trend of the alarm event. Please synchronize the contents of the two screens in advance to ensure that the information on the two large screens is consistent before the next operation to reduce misunderstandings caused by inconsistent data. 2. Based on data association and synchronization technology: The communication system needs to ensure the data association between screen A and screen B. If the time difference Δt exceeds the set threshold, the data synchronization mechanism is triggered immediately; if the difference between the number of faults A and the number of alarms B exceeds the threshold, the system automatically synchronizes and updates the alarm status and fault information to ensure that the data on the two screens is consistent. 3. Question and answer and analysis based on knowledge graph: When the operator encounters the problem of inconsistent fault and alarm data, the knowledge graph technology is used to automatically analyze the association between the two and provide suggestions for the association between alarms and faults. For example, the system will infer the specific cause of the alarm caused by the optical cable fault based on the knowledge graph, and display the relevant information on screen A and screen B simultaneously. 4. Based on multimodal data fusion technology: The system performs multimodal data fusion on fault charts and alarm status information, integrates charts and alarm status into a unified comprehensive view, and displays them on screen A and screen B at the same time, so that operators can obtain complete warning status from both screens at the same time. 5. Synchronization strategy: Time synchronization: When the time difference Δt exceeds the set threshold, the system immediately synchronizes the alarm status of screen B with the fault data of screen A to make the timestamps consistent. Data synchronization: When the numerical difference between the number of faults A and the number of alarms B exceeds the threshold, the system automatically updates the alarm information synchronously to ensure that the data on screen B is consistent with that on screen A. Image synchronization: When the image difference SSIM value is lower than the set threshold, the system automatically updates the charts and alarm status of screens A and B to ensure real-time synchronization of image content. "

[0212] In addition, in some embodiments, in order to prevent errors or "hallucinations" encountered when the visual language model updates the display interface, that is, the model generation results do not match the actual situation, the communication system can support determining the user's attention information based on the user's interaction behavior with the communication scheduling screen. Among them, the interactive behavior includes operation interaction and dialogue interaction. Exemplarily, the operation interaction refers to monitoring the user's operations on the communication scheduling screen, including actions such as mouse hovering, clicking, dragging, and zooming. For example, if a user clicks or views a specific alarm area, the communication system will regard it as the user's current attention information. The dialogue interaction refers to a dialogue between a user and the communication scheduling screen (communication system). For example, if a user mentions "the progress of the processing of a certain optical cable alarm" many times in the conversation, the communication system will identify it as the user's attention information.

[0213] For the two users corresponding to the two communication scheduling screens indicated by the multi-screen display content monitoring task, if the attention information of the two users is different, and the attention information of at least one user is not displayed on the communication scheduling screen corresponding to the other user, the synchronization mechanism is triggered to synchronously display the attention information of the two users on the communication scheduling screen corresponding to the other user, thereby ensuring that the display of the two communication scheduling screens is consistent and improving the accuracy of the layout.

[0214] For example, for a scenario where at least two communication scheduling screens are used to assist in processing tasks, if the users of the two communication scheduling screens are user A and user B, and it is monitored that user A repeatedly checks the historical fault records of the optical cable, and the topic of the conversation between user B and the communication system is regional alarm processing, it can be known that the information of interest to user A and user B is not the same. Based on this, the display content on the communication scheduling screen corresponding to user A can be further matched with the information of interest to user B to obtain a first matching result, and the display content on the communication scheduling screen corresponding to user B can be matched with the information of interest to user A to obtain a second matching result. If the first matching result indicates a mismatch, it means that the information concerned by user B is not displayed in the communication scheduling screen corresponding to user A. Similarly, if the second matching result indicates a mismatch, it means that the information concerned by user A is not displayed in the communication scheduling screen corresponding to user B. This is obviously not conducive to task assistance processing. Therefore, the HTML content of the communication scheduling screen corresponding to the two users can be obtained through the visual language model, the data display area that needs to be synchronized and the content difference data on the communication scheduling screen corresponding to the two users can be extracted, and the content difference data can be injected into the corresponding HTML content of the two communication scheduling screens, so that the display of the two communication scheduling screens is consistent.

[0215] In addition, after receiving the synchronization instruction, the communication scheduling screen can consume, read and display data from the same message queue. For example, a distributed database or message queue can be used to achieve real-time synchronization of data between screens. Exemplarily, a distributed message queue system such as Kafka or RabbitMQ can be used. Specifically, a message queue can be set between two communication scheduling screens, and whenever warning data is generated, the warning data will be pushed to the message queue. As a consumer of the message queue, the communication scheduling screen can read the warning data from the message queue and display it synchronously. After the consumer confirms the message, the message queue will mark the data as processed to ensure that the data will not be lost, thereby achieving data synchronization.

[0216] In addition, the communication scheduling screen can also be monitored synchronously. Here, the synchronous monitoring refers to the monitoring performed when the communication scheduling screen synchronizes the display content in response to the synchronization instruction. Exemplarily, Prometheus can be used in combination with Grafana for monitoring.

[0217] In addition, in order to improve the synchronization efficiency of the displayed content, a low-latency communication protocol may be used for delay control, such as WebSocket or gRPC (high-performance remote procedure call protocol).

[0218] Continuing with scenario 1 as an example: when the optical cable fault information or alarm information is updated, the communication system can push the updated data to the message queue through tools such as Kafka. The two communication scheduling screens to be synchronized act as consumers, respectively, to obtain the latest data from the queue and display it. After each communication scheduling screen receives and displays the data, a confirmation message can be sent to the message queue to ensure that the data is processed correctly and prevent data loss. At the same time, the data synchronization between the two communication scheduling screens can be monitored based on Prometheus and Grafana tools, and the delay and data consistency indicators can be collected and displayed in real time. If data delay or synchronization inconsistency is detected between the two communication scheduling screens, an alarm is immediately issued to notify the operation and maintenance personnel to handle it. In addition, a two-way communication link between the two communication scheduling screens can be established through WebSocket to ensure that data can be transmitted in real time. When one of the communication scheduling screens updates the fault chart, the alarm information of the other communication scheduling screen will also be updated synchronously to ensure that users can obtain consistent information on different screens.

[0219] It can be seen that this embodiment couples the multi-screen information synchronization technology with the prompt word engineering of the visual language model. When a significant error is found between the two screens, a decision is made automatically to reduce human intervention. At the same time, it can also ensure that the information between the two communication scheduling screens remains consistent and complementary. The two screens can accurately and synchronously process and display warning information, and can dynamically adjust the display content to ensure that the information on the two screens complement each other, which helps team members better understand the overall situation and avoid communication barriers caused by inconsistent information. At the same time, when the information on one screen changes, the other screen can automatically make corresponding adjustments to guide the user's attention through intelligent prompts or highlights. At the same time, it also ensures that the data between the two screens is synchronized in real time to ensure consistency and accuracy during information transmission, so that team members can see updates instantly, effectively prevent display errors caused by communication delays or errors, and improve the efficiency of collaborative work.

[0220] Based on the large-screen monitoring method for communication scheduling based on a large model agent described in any of the above embodiments, the present application also provides a computer program product, which includes one or more computer programs or instructions. The computer program or instruction can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. When the computer program is executed by the processor, the large-screen monitoring method for communication scheduling based on a large model agent described in any of the above embodiments is implemented.

[0221] Based on the communication scheduling large-screen monitoring method based on a large model agent described in any of the above embodiments, the present application also provides a communication scheduling large-screen monitoring system based on a large model agent, wherein the large model agent includes a visual language model, and the communication scheduling screen is used to display monitoring data, environmental data, fault data, and emergency data of the communication optical cable, and the system includes:

[0222] A training module, used for performing supervised training on a link prediction model using a historical question data set; wherein the historical question data set includes a plurality of historical question links carrying features, each of the historical question links includes historical question content and subsequent question content of a user on the display content of a communication scheduling screen; the features include text features and context features of the historical question links;

[0223] A prediction module, used to obtain the current question content of the user on the communication optical cable, and use the trained link prediction model to obtain a predicted question link of the current question content, wherein the predicted question link at least includes the next question content on the communication optical cable;

[0224] a layout module, configured to generate prompt content based on the predicted question link, the current display data of the communication scheduling screen and a preset prompt template, and input the prompt content into a trained visual language model to obtain layout data generated by the visual language model; wherein the layout data includes data to be displayed and display parameters; the data to be displayed is related to the predicted question link and includes one or more of the monitoring data, the environmental data, the fault data, and the emergency data;

[0225] An updating module is used to update the display interface of the communication scheduling screen based on the layout data.

[0226] Based on the communication scheduling large-screen monitoring method based on a large model intelligent agent described in any of the above embodiments, the present application also provides: Figure 3 A schematic diagram of the structure of an electronic device is shown in FIG. Figure 3 At the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement a communication scheduling large-screen monitoring method based on a large model agent as described in any of the above embodiments.

[0227] The present application also provides a computer storage medium, which stores a computer program. When the computer program is executed by a processor, it can be used to execute a communication scheduling screen layout method based on a visual language model described in any of the above embodiments.

[0228] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0229] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0230] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0231] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0232] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0233] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A communication scheduling large-screen monitoring method based on a large model agent, characterized in that: The large model agent includes a visual language model, and the communication scheduling screen is used to display monitoring data, environmental data, fault data, and emergency data of the communication optical cable. The method includes: The link prediction model is supervisedly trained using a historical question data set; wherein the historical question data set includes a plurality of historical question links carrying features, each of the historical question links includes historical question content and subsequent question content of a user on the display content of a communication scheduling screen; the features include text features and context features of the historical question links; Acquire the current question content of the user on the communication optical cable, and use the trained link prediction model to acquire a predicted question link of the current question content, wherein the predicted question link at least includes the next question content on the communication optical cable; Generate prompt content based on the predicted question link, the current display data of the communication scheduling screen and a preset prompt template, and input the prompt content into a trained visual language model to obtain layout data generated by the visual language model; wherein the layout data includes data to be displayed and display parameters; the data to be displayed is related to the predicted question link and includes one or more of the monitoring data, the environmental data, the fault data, and the emergency data; A display interface of the communication scheduling screen is updated based on the layout data.

2. The method according to claim 1, characterized in that The prompt template also includes associations between multiple data; wherein the associations are obtained by mining multiple rounds of interaction data between the user and the communication scheduling screen using association rules in advance; the associations include the association between the monitoring data of the communication optical cable and the environmental data, the association between the fault data and the emergency data, and the association between the monitoring data of the communication optical cable and the fault data; The generated prompt content carries the association relationship, so that the layout data generated by the visual language model includes first data to be displayed related to the predicted question link, and second data to be displayed related to the first data to be displayed based on the association relationship; wherein the association between the first data to be displayed and the second data to be displayed is represented in the form of a chart and / or a map.

3. The method according to claim 1, characterized in that The prompt content further includes one or more prompt information, so that the layout data generated by the visual language model also includes third display data obtained based on the prompt information; the prompt information includes: Prompt information for prompting data related to the current question content mined from a preset database, wherein the third display data includes the data related to the current question content obtained by mining; Prompt information for prompting to call a multimodal pre-trained model to obtain fused data, wherein the third display data includes fused data obtained by mapping data related to the current question content obtained from multiple data sources to the same vector space using the multimodal pre-trained model; wherein the multimodal pre-trained model is trained using text, image, and video data historically displayed on the communication scheduling screen; Prompt information for prompting to call the knowledge graph, the third display data including knowledge data in the knowledge graph related to the current question content; wherein the knowledge graph is constructed using the relationships between multiple entities of the optical cable; The prompt information is used to prompt the calling of the adaptive model, and the third display data includes display data obtained based on the context and / or scene of the current question content by using the adaptive model.

4. The method according to claim 1, characterized in that: The prompt content also includes the current scene information input by the user; the data to be displayed is related to the current scene information; wherein, If the current scene information is a daily communication scheduling scene, the data to be displayed includes: the monitoring data, the environmental data, and the fault data; If the current scene information is an emergency command scene, the data to be displayed includes: optical cable topology data of the emergency area, alarm data in the emergency area, and security data of the emergency area; If the current scene information is a holiday-themed scene, the data to be displayed includes: monitoring data of the target communication optical cable in the holiday protection area, alarm information in the holiday protection area, and management information of the holiday protection area; If the current scene information is a reporting scene, the data to be displayed includes: indicator statistical data, indicator analysis data, and business assurance status data of the communication optical cable.

5. The method according to claim 4, characterized in that The visual language model is trained using screenshots carrying target information, wherein the target information includes: question-answer pairs about the screenshots, logical relationship information between elements in the screenshots, navigation operation information, and summary information; The prompt content includes the current scene information and user instructions input by the user, so that the visual language model executes the user instructions on the data to be displayed corresponding to the current scene information; the user instructions include one or more of screen information understanding instructions, question and answer instructions, navigation instructions and summary instructions.

6. The method according to claim 1, characterized in that The prompt content also includes the current question content and question structured task information, and the method further includes: Converting the current question content into a structured question and generating answer content based on the question structured task information through the visual language model; The problem structuring task includes determining the scope of the problem topic, determining the type of data involved in the problem, constructing the problem framework, refining the problem elements, and setting the answer format; Determining the scope of the question topic includes determining that the current question content is centered around the communication optical cable related data displayed in the communication scheduling screen; The determining of the data types involved in the question includes classifying and arranging the data types corresponding to the current question content, wherein the data types include optical cable core utilization rate, environmental parameters, impact of environmental parameters on optical cable performance, computer room temperature thresholds and countermeasures, faulty optical cable information, and unresolved optical cable faults; The constructing of the question framework includes converting the current question content into one or more of general questions, specific numerical questions, comparison questions, cause analysis questions, impact assessment questions, prediction questions, and solution questions based on different data categories; The refinement of question elements includes refining the time, space, object, condition, and cause elements involved in the current question content; The setting of the answer format includes specifying the answer format, and the answer format includes a list, a chart, and text.

7. The method according to claim 1, characterized in that The visual language model is used to simultaneously layout a plurality of the communication scheduling screens, the prompt content also includes a multi-screen display content monitoring task, and the method further includes: Performing difference detection on display contents of at least two communication scheduling screens based on the multi-screen display content monitoring task through the visual language model; wherein the difference detection includes time synchronization difference detection, monitoring value difference detection, and image content difference detection; If the difference detection indicates that the display content difference of the at least two communication scheduling screens is greater than a preset difference threshold, a synchronization instruction is output to the at least two communication scheduling screens so that the communication scheduling screens consume, read and display data from the same message queue.

8. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, when the processor calls the executable instruction, the method described in any one of claims 1-7 is implemented.

9. A communication scheduling large-screen monitoring system based on a large model agent, characterized in that: The large model agent includes a visual language model, and the communication scheduling screen is used to display monitoring data, environmental data, fault data, and emergency data of the communication optical cable. The system includes: A training module, used for performing supervised training on a link prediction model using a historical question data set; wherein the historical question data set includes a plurality of historical question links carrying features, each of the historical question links includes historical question content and subsequent question content of a user on the display content of a communication scheduling screen; the features include text features and context features of the historical question links; A prediction module, used to obtain the current question content of the user on the communication optical cable, and use the trained link prediction model to obtain a predicted question link of the current question content, wherein the predicted question link at least includes the next question content on the communication optical cable; a layout module, configured to generate prompt content based on the predicted question link, the current display data of the communication scheduling screen and a preset prompt template, and input the prompt content into a trained visual language model to obtain layout data generated by the visual language model; wherein the layout data includes data to be displayed and display parameters; the data to be displayed is related to the predicted question link and includes one or more of the monitoring data, the environmental data, the fault data, and the emergency data; An updating module is used to update the display interface of the communication scheduling screen based on the layout data.

10. A computer-readable storage medium, characterized in that: Computer instructions are stored thereon, and when the computer instructions are executed by a processor, the steps of any method described in claims 1-7 are implemented.

Images