Intelligent communication network real-time monitoring method and system based on multiple modes

Abstract

The invention provides a multi-mode-based intelligent communication network real-time monitoring method and system, and the method comprises the steps: responding to a plurality of intelligent communication network tasks, and determining the correlation information between the intelligent communication network tasks through a task incidence relation; generating prompt content based on the correlation information, the current display data and a prompt template, and inputting the prompt content into a multi-modal visual language model to obtain a region division result; the region division result indicates a target region and display data of the target region; if the correlation information indicates that the intelligent communication network task is not correlated, the target area comprises a task area corresponding to the intelligent communication network task; if the correlation information indicates that the intelligent communication network task is related, the target area comprises a task area and a sharing area; the display data of the task area comprises real-time monitoring data of an intelligent communication network task; the display data of the shared area comprises associated data, and displaying real-time monitoring data of the intelligent communication network based on an area division result.

Description

Technical Field

[0001] The present application relates to the field of screen display technology, and in particular to a multi-modal intelligent communication network real-time monitoring method and system. Background Art

[0002] With the increasing development of communication technology, intelligent communication network is one of the most important communication networks in the current power industry. As the scale of intelligent communication network continues to expand, maintaining the intelligent communication network deployed in different geographical areas is a challenging and quite important task.

[0003] Taking the optical communication network as an example, in the relevant technology, the communication optical cable maintainer can view various data such as the working status and fault information of the communication optical cable through the communication scheduling screen. However, the data involved in the communication optical cable is numerous and complicated, so a large amount of data information related to the communication optical cable is often displayed through a large-size communication scheduling screen. At this time, multiple maintenance personnel are often required to view and process the display information in different display areas of the screen at the same time. Based on this, how to improve the user experience of multiple users using the same communication scheduling screen at the same time is a technical problem that needs to be solved urgently in this field. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a multi-modal intelligent communication network real-time monitoring method and system to improve the user experience of multiple users using the same communication scheduling screen at the same time.

[0005] A first aspect of an embodiment of the present application provides a multi-modal based intelligent communication network real-time monitoring method, the method being used to realize displaying real-time monitoring data of the intelligent communication network on a communication scheduling screen; the method comprising:

[0006] In response to a plurality of intelligent communication network tasks, determining correlation information between the plurality of intelligent communication network tasks using a pre-established task association relationship;

[0007] Generate prompt content based on the correlation information, the current display data of the display area and a preset prompt template, and input the prompt content into a trained multimodal visual language model to obtain a region division result generated by the multimodal visual language model for the display area of ​​the communication scheduling screen; the region division result is used to indicate a plurality of target regions of the display area division, and display data of each of the target regions;

[0008] Wherein, if the correlation information indicates that the multiple intelligent communication network tasks are unrelated, the multiple target areas include task areas corresponding to the intelligent communication network tasks one by one; if the correlation information indicates that the multiple intelligent communication network tasks are related, the multiple target areas include multiple task areas and shared areas; the display data of each task area includes the real-time monitoring data of the corresponding intelligent communication network task; the display data of the shared area includes the association data between the multiple related intelligent communication network tasks;

[0009] Based on the area division result generated by the multimodal visual language model, real-time monitoring data of the intelligent communication network is displayed on the communication scheduling screen.

[0010] A second aspect of the embodiment of the present application provides a multi-modal intelligent communication network real-time monitoring system, the system is used to realize the display of real-time monitoring data of the intelligent communication network on the communication scheduling screen; the system includes:

[0011] A correlation module, configured to determine correlation information between the plurality of intelligent communication network tasks in response to the plurality of intelligent communication network tasks by using a pre-established task association relationship;

[0012] A division module, used to generate prompt content based on the correlation information, the current display data of the display area and a preset prompt template, and input the prompt content into a trained multimodal visual language model to obtain a region division result generated by the multimodal visual language model for the display area of ​​the communication scheduling screen; the region division result is used to indicate a plurality of target regions of the display area division, and display data of each of the target regions;

[0013] Wherein, if the correlation information indicates that the multiple intelligent communication network tasks are unrelated, the multiple target areas include task areas corresponding to the intelligent communication network tasks one by one; if the correlation information indicates that the multiple intelligent communication network tasks are related, the multiple target areas include multiple task areas and shared areas; the display data of each task area includes the real-time monitoring data of the corresponding intelligent communication network task; the display data of the shared area includes the association data between the multiple related intelligent communication network tasks;

[0014] A display module is used to display the real-time monitoring data of the intelligent communication network on the communication scheduling screen based on the area division result generated by the multimodal visual language model.

[0015] According to a third aspect of the present application, an electronic device is provided, the electronic device comprising:

[0016] processor;

[0017] a memory for storing processor-executable instructions;

[0018] Wherein, when the processor calls the executable instruction, it implements the operation of any method described in the first aspect.

[0019] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of any method described in the first aspect.

[0020] In the above implementation process, the correlation between multiple intelligent communication network tasks is determined by the task association relationship. When the tasks are unrelated, the display area is divided into multiple independent task areas using a multimodal visual language model. Users can perform tasks in their respective task areas without interfering with each other. When the tasks are related, the display area is divided into multiple task areas and a shared area, and the associated data between the tasks is displayed in the shared area, so that multiple users can collaboratively process tasks through the shared area, thereby improving the user experience and task execution efficiency of multiple users when using the same communication scheduling screen at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 A flowchart of a multi-modal intelligent communication network real-time monitoring method provided in an embodiment of the present application;

[0023] Figure 2-Figure 4 A flowchart of another multi-modal intelligent communication network real-time monitoring method provided in an embodiment of the present application;

[0024] Figure 5 A hardware structure diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0026] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0027] In order to improve the user experience of multiple users using the same communication scheduling screen at the same time, the first aspect of the present application provides a multi-modal real-time monitoring method for an intelligent communication network, which is used to realize the display of real-time monitoring data of the intelligent communication network on the communication scheduling screen. The intelligent communication network is, for example, an optical communication network. In an optical communication network, communication optical cables are an important component. The communication scheduling screen can be a large-sized screen formed in one piece, or it can be a large screen composed of multiple small-sized screens. There is no limitation on the size of the communication scheduling screen. For example Figure 1 As shown, the method includes steps 110 to 130.

[0028] Step 110: In response to the plurality of intelligent communication network tasks, determining correlation information between the plurality of intelligent communication network tasks using a pre-established task association relationship;

[0029] Step 120: Generate prompt content based on correlation information, current display data of the display area and a preset prompt template, and input the prompt content into a trained multimodal visual language model to obtain a region division result generated by the multimodal visual language model for the display area of ​​the communication scheduling screen; the region division result is used to indicate a plurality of target regions of the display area division, and display data of each of the target regions;

[0030] Wherein, if the correlation information indicates that the multiple intelligent communication network tasks are unrelated, the multiple target areas include task areas corresponding to the intelligent communication network tasks one by one; if the correlation information indicates that the multiple intelligent communication network tasks are related, the multiple target areas include multiple task areas and shared areas; the display data of each task area includes the real-time monitoring data of the corresponding intelligent communication network task; the display data of the shared area includes the association data between the multiple related intelligent communication network tasks;

[0031] Step 130: Based on the area division result generated by the multimodal visual language model, the real-time monitoring data of the intelligent communication network is displayed on the communication scheduling screen.

[0032] Regarding the intelligent communication network task, illustratively, the intelligent communication network task may include, but is not limited to, any one or more of a communication equipment monitoring task, a communication fault analysis task, and a communication resource scheduling task. Among them, the intelligent communication network task may be triggered by a user input or according to a preset rule event. For example, a user may input a monitoring task for an intelligent communication network in a certain area in the communication system. For another example, when a communication optical cable in a certain area fails, the communication system automatically triggers a communication fault analysis task. For another example, when the intelligent communication network in a certain area fails or communication resources are in short supply and support is required, the communication resource scheduling task may be triggered by a user or a communication system, and so on. In addition, each intelligent communication network task is handled by one or more users.

[0033] When there are multiple intelligent communication network tasks in the communication system, the communication scheduling screen needs to simultaneously display the data required to process each intelligent communication network task, so that the users corresponding to each intelligent communication network task can simultaneously execute tasks on the same communication scheduling screen. Therefore, the correlation information between the multiple intelligent communication network tasks can be determined by using the pre-established task association relationship. The process of establishing the task association relationship will be described below. The correlation information can represent whether the multiple intelligent communication network tasks are related or not.

[0034] Subsequently, prompt content is generated based on the correlation information, the current display data of the display area and the preset prompt (prompt word) template. Among them, the current display data of the display area can be the current screenshot of the display area, and / or the current HTML data of the display area. The prompt template is a prompt word template of the multimodal visual language model, and the composition of the prompt template includes instructions, context, examples, input and output. Prompt content can be generated by inputting correlation information into the prompt template. In the large model, the role of the prompt content is to prompt the multimodal visual language model with the context of the input information and the parameter information of the input model. The multimodal visual language model can be a large model. The large model refers to a deep learning model trained based on massive text data, and the number of its parameters can be as high as millions or even hundreds of millions. Therefore, the large model can handle more complex and comprehensive data, and learn more patterns and laws from it. In some embodiments, the multimodal visual language model used in this embodiment is, for example, a ScreenAI model.

[0035] The generated prompt content can be input into the trained multimodal visual language model. In this way, the multimodal visual language model can divide the display area of ​​the communication scheduling screen into multiple target areas based on the correlation information, and can determine the display data of each target area to obtain the area division result output by the multimodal visual language model.

[0036] Specifically, if multiple intelligent communication network tasks are unrelated, the display area of ​​the communication scheduling screen is divided into task areas corresponding to the intelligent communication network tasks one by one. That is, the multiple target areas include multiple task areas. The number of task areas is consistent with the number of intelligent communication network tasks, and the display data corresponding to each task area includes the real-time monitoring data of the intelligent communication network task corresponding to the task area, so that each user who performs the intelligent communication network task can perform the task in the corresponding task area based on the relevant real-time monitoring data.

[0037] If multiple intelligent communication network tasks are related, the display area of ​​the communication scheduling screen is divided into task areas corresponding to the intelligent communication network tasks one by one, and a shared area. That is, the multiple target areas include multiple task areas and shared areas. Similarly, the number of task areas is consistent with the number of intelligent communication network tasks, and the display data corresponding to each task area includes the real-time monitoring data of the intelligent communication network task corresponding to the task area. In addition, the display data corresponding to the shared area includes the association data between multiple related intelligent communication network tasks, so that users who perform associated tasks can also view the association data between tasks in the shared area.

[0038] Among them, the position and size of the shared area can be determined based on the position data of the user performing the associated task relative to the communication scheduling screen and / or the distance data between the user and the communication scheduling screen. Among them, the process of obtaining the position data and distance data will be unfolded below. For example, the position data and / or distance data corresponding to multiple users performing associated tasks can be input into the prompt template, so that the generated prompt content includes the position data and / or distance data corresponding to multiple users performing associated tasks. In this way, the multimodal visual language model can determine the shared area in a suitable position in the display area based on the position data and / or distance data, and the associated data displayed in the shared area includes chart data, such as bar charts, bar graphs, pie charts, tables, etc.

[0039] Finally, after obtaining the area division results output by the multimodal visual language model, the communication system can divide the display area to obtain multiple target areas, and display the real-time monitoring data of the intelligent communication network on the communication scheduling screen to achieve real-time monitoring of the intelligent communication network.

[0040] The following is an illustrative example:

[0041] In a communication system, multiple users simultaneously handle different intelligent communication network tasks. Assume that user A is responsible for the communication equipment monitoring task, mainly monitoring the operating status and abnormal conditions of the communication; user B is responsible for the communication fault analysis task, mainly analyzing the causes of communication accidents, and collecting relevant fault data for analysis. If it is determined that the communication equipment monitoring task and the communication fault analysis task are not related according to the pre-established task association relationship, the display area of ​​the communication scheduling screen is divided into two independent task areas and operation isolation is enabled. The operation isolation means that user A has no operation authority in the task area corresponding to user B, and the operation authority includes read authority and write authority, and vice versa. Operation isolation can ensure that each user's task area is independent to prevent misoperation or information confusion. In this way, user A and user B respectively perform communication equipment monitoring tasks and communication fault analysis tasks in their respective task areas without interfering with each other.

[0042] Subsequently, when user A discovers that the communication equipment is faulty, the communication resource scheduling task is triggered, and user A continues to perform the communication equipment monitoring task, while user B performs the communication resource scheduling task. According to the pre-established task association relationship, it is determined that the communication equipment monitoring task is related to the communication resource scheduling task, and the display area of ​​the communication scheduling screen is divided into two task areas and a shared area. Among them, the task area corresponding to the communication equipment monitoring task gives priority to displaying the status information of the communication equipment, while the task area corresponding to the communication resource scheduling task gives priority to displaying the resource scheduling information. Since the user needs to pay attention to the dynamics of the faulty equipment in both the communication equipment monitoring task and the resource scheduling task, the shared area focuses on displaying the specific status information of the faulty equipment. At the same time, both user A and user B have operation permissions in the shared area.

[0043] It can be seen that this embodiment determines the correlation between multiple intelligent communication network tasks through task association relationships. When the tasks are unrelated, the multimodal visual language model is used to divide the display area into multiple independent task areas, and users can perform tasks in their respective task areas without interfering with each other. When the tasks are related, the display area is divided into multiple task areas and a shared area, and the associated data between the tasks is displayed in the shared area, so that multiple users can collaboratively process tasks through the shared area, so as to improve the user experience and task execution efficiency of multiple users when using the same communication scheduling screen at the same time.

[0044] Regarding the process of establishing the task association relationship in step 110, in some embodiments, multiple rounds of historical interaction data between the user and the communication scheduling screen may be collected, and then an association rule mining algorithm may be used to mine association rules between intelligent communication network tasks from the multiple rounds of historical interaction data. The intelligent communication network tasks are related to user interaction behaviors.

[0045] Association rules are an important technology in data mining, which is used to discover the association relationship between various data. Association rules can be, for example, the Apriori (association rule) algorithm, which uses an iterative method of layer-by-layer search to generate association rules by calculating the support and confidence of item sets. Association rules can also be, for example, the FP-Growth algorithm, which can find high-frequency association rules.

[0046] Since the intelligent communication network tasks are related to the interaction behaviors between the user and the communication scheduling screen, by mining the historical interaction data between the user and the communication scheduling screen, it is possible to learn whether various intelligent communication network tasks are related.

[0047] Specifically, first, the historical interaction data of multiple rounds between the user and the communication scheduling screen can be collected. Since the communication system records the operation trajectory data of each user, the operation trajectory data includes user operations, operation sequence and operation time. The operation trajectory data is stored in the user's behavior log, so by reading the user's behavior log, the historical interaction data can be obtained. For example, in a user's behavior log, it can be determined that after viewing the fault information of the communication optical cable, the user further viewed the status of other optical cables (including temperature, fiber core utilization rate, maintenance level, etc.).

[0048] Optionally, the historical interaction data can be cleaned to remove invalid and erroneous data, and the user interaction data can be transformed to make it suitable for association rule mining.

[0049] Subsequently, an association rule mining algorithm, such as the Apriori algorithm, can be used to mine the processed historical interaction data. Association rules are generated by setting the minimum support and minimum confidence thresholds. The generated association rules can then be evaluated to screen out association rules that meet the communication optical cable scheduling scenario as task association relationships. Optionally, the mined task association relationships can be displayed in a chart, text or other form for easy understanding and application. The generated task association relationships between each two tasks can be stored in a database. As an example, a task association relationship can be expressed as "TaskRelation (task ID1, task ID2, association type)", where the association relationship includes related and unrelated.

[0050] The Apriori algorithm can be used to mine task associations. For example, when a user performs a task and checks the current optical cable fault status, in most cases the user may also need to view relevant information charts such as the cause of the fault, fault location, common alarm types, emergency plans, and safeguards. These rules are of great guiding significance for determining whether tasks between users are associated based on user interactions.

[0051] Regarding the storage method of the data related to the intelligent communication network in the database, based on any of the above embodiments, when executing the above Figure 1 Prior to the illustrated embodiment, the following steps may be performed:

[0052] A first relational model of each relevant entity of the intelligent communication network and a second relational model for characterizing the relationship between a plurality of the relevant entities are stored in advance in a query database.

[0053] The so-called related entities of the intelligent communication network include, for example, but are not limited to, one or more of the optical cable routing monitoring entity, dynamic environment monitoring entity, optical cable fault entity, emergency support entity, intelligent communication network task entity, screen layout entity, user station entity, and user entity. The related entities are actually the data of the intelligent communication network at different levels. The first relationship model corresponding to all related entities of the intelligent communication network and the second relationship model representing the relationship between multiple related entities constitute the relationship model of the intelligent communication network. After converting the data at different levels of the intelligent communication network into a relationship model, data storage, query and management can be realized.

[0054] The first relationship model of each related entity includes entity attributes of the related entity. One or more entity attributes are entity codes of the related entity. The entity code includes a primary code and an external code. The primary code is used to uniquely identify the related entity. The external code is used to establish a relationship with other related entities, and usually points to the primary code of another related entity.

[0055] For example, the first relational model of the Cable Monitoring entity can be expressed as: Cable Monitoring (cable ID, cable name, cable length, fiber core utilization rate, maintenance level, service carrying capacity, status). Among them, "cable ID, cable name, cable length, fiber core utilization rate, maintenance level, service carrying capacity, status" are entity attributes, and "cable ID" is the main code in the entity code.

[0056] For another example, the first relational model of the dynamic environmental monitoring (EnvironmentalMonitoring) entity can be expressed as: EnvironmentalMonitoring (monitoring ID, device ID, temperature, humidity, voltage, monitoring time), where "monitoring ID" is the main code in the entity code, and "device ID" is the external code in the entity code.

[0057] For another example, the first relational model of the optical cable fault (Faults) entity can be expressed as: Faults (fault ID, optical cable ID, fault time, fault type, fault level, processing status), where "fault ID" is the main code in the entity code, and "optical cable ID" is the external code in the entity code.

[0058] For another example, the first relational model of the Emergency Response entity can be expressed as: EmergencyResponse (Emergency ID, Task ID, Emergency Type, Personnel Location, Scheduling Status, Processing Progress), where "Emergency ID" is the primary code in the entity code, and "Task ID" is the external code in the entity code.

[0059] For another example, the first relational model of the intelligent communication network task entity can be expressed as: Task (task ID, task name, task status, creation time), where "task ID" is the main code in the entity code.

[0060] For another example, the first relational schema of the screen layout (ScreenRegion) entity can be expressed as: ScreenRegion (region ID, screen partition ID, task ID, partition status, layout information), where "region ID" is the main code in the entity code, and "task ID" is the external code in the entity code.

[0061] For another example, the first relational schema of the UserPositionAndAngleCorrection entity can be expressed as: UserPositionAndAngleCorrection (user ID, task ID, user position, distance, posture angle, correction angle), where "user ID" is the primary code in the entity code, and "task ID" is the external code in the entity code.

[0062] For another example, the first relational model of the User entity can be expressed as: User (user ID, task ID, user name, role, authority), where "user ID" is the primary key in the entity code, and "task ID" is the external key in the entity code.

[0063] In addition to storing entity attributes in the above-mentioned independent relational schemas, the relational schemas of multiple related entities can also be merged. For example, the emergency support entity is associated with the intelligent communication network task entity through the task ID, so the merged relational schema can be expressed as: Task (task ID, task name, task status, creation time, emergency ID, emergency type, personnel location, scheduling status, processing progress).

[0064] In addition, the query database also stores a second relationship model for characterizing the relationship between multiple related entities. The second relationship model records the entity codes of multiple related entities with which the relationship exists. As mentioned above, the external code is used to establish a relationship with other related entities, usually pointing to the primary code of another related entity. Therefore, the second relationship model between multiple related entities can be established through the external code.

[0065] For example, when a communication optical cable fails, the communication system needs to retrieve specific information about the communication optical cable from the first relationship model corresponding to the optical cable route monitoring, including the fiber core utilization rate and maintenance level. Therefore, there is a relationship between the optical cable route monitoring entity and the optical cable fault entity, and the second relationship model can be expressed as: CableMonitoring (optical cable ID) - 1:n - Faults (optical cable ID). Among them, "optical cable ID" is the main code in the optical cable fault entity and the external code in the optical cable fault entity. "1:n" means that a communication optical cable may have multiple fault records.

[0066] For another example, when a communication optical cable fault occurs, the communication system will trigger an emergency support task, including dispatching relevant personnel to handle the fault. The dispatch and execution of the emergency response need to be displayed in real time. Therefore, there is a relationship between the optical cable fault entity and the emergency support entity, and the second relationship model can be expressed as: Faults (fault ID) - 1:1 - EmergencyResponse (task ID). "1:1" means that one communication optical cable fault corresponds to one emergency support dispatch.

[0067] For another example, when executing an emergency task, the dispatched personnel information, location, and task processing progress need to be associated with the first relationship model corresponding to the user. Therefore, there is a relationship between the emergency guarantee entity and the user entity, and the second relationship model can be expressed as: EmergencyResponse (task ID) - n:1 - User (user ID). "n:1" means that multiple emergency guarantees can be monitored and managed by the same user.

[0068] When determining which related entities have relationships, an association rule mining algorithm can also be used to extract the relationships between related entities from multiple rounds of historical interaction data. Exemplarily, frequently occurring operation sequences can be identified through user historical operation transactions. The historical operation transaction refers to the user's historical operation in the query database. For example, each time a user views the entity attributes in some first relationship patterns, it can be regarded as a historical operation transaction. Using the association rule mining algorithm, user behavior records can be analyzed to find operation sequences such as queries or accesses performed by multiple users. For example, after viewing the communication cable fault information, the user often views the communication cable status information. This operation sequence can be identified as a frequent item set.

[0069] It can be seen that each relevant entity of the intelligent communication network is stored in the database in the form of a relationship model (including the first relationship model and the second relationship model), which can realize data storage, query and management.

[0070] On this basis, regarding the determination process of display data, in this embodiment, the target entity code corresponding to each intelligent communication network task can also be input into the prompt template, so that the generated prompt content includes not only the correlation information between multiple intelligent communication network tasks, but also the target entity code corresponding to each intelligent communication network task. In this way, the multimodal visual language model can determine the target entity that has a relationship with each intelligent communication network task based on the target entity code and the second relationship model, and determine the entity attribute recorded in the first relationship model of the target entity as the display data.

[0071] Since the second relational schema records the entity codes of multiple related entities with which there is a relationship, by querying the target entity code corresponding to the intelligent communication network task in all second relational schemas, it is possible to know which related entities the intelligent communication network task has a relationship with. In this way, the multimodal visual language model can determine the entity attributes of the target entity as display data. Based on the output of the multimodal visual language model, the communication system can read the indicated entity data from the query database for display.

[0072] It can be seen that this embodiment stores the data related to the intelligent communication network in the form of a relational model in the query database to achieve data storage, query and management. At the same time, by using the relational model and the target entity code corresponding to the intelligent communication network task, the multimodal visual language model can determine the target entity that has a relationship with the intelligent communication network task, and determine the entity attribute of the target entity as display data, thereby realizing data reading and display.

[0073] In addition, in some embodiments, the first relational model and the second relational model can also be stored in the form of vectors in a vector database. Exemplarily, the various relational models constructed above can be cleaned and preprocessed to make them more suitable for vectorized representation. The preprocessing includes one or more of text standardization processing, field merging processing and data block processing. Among them, text standardization processing refers to removing irrelevant characters, correcting time format, etc., so as to ensure the consistency of text format. Field merging processing refers to merging all key information into a complete text. For example, the first relational model after the optical cable fault entity is merged can be expressed as "fault ID: 12345, optical cable ID: 987, fault time: 2024-01-0110:00, fault type: optical cable break, fault level: high, processing status: processing". Data block processing refers to dividing the text into multiple text blocks while maintaining the text continuity between multiple text blocks.

[0074] After cleaning and preprocessing, you can use a Transformer-based model, such as the BERT model, to convert the preprocessed text into a vector. Each relationship pattern needs to be vectorized to generate a corresponding vector representation. Then use a vector database (such as the FAISS library) to store the generated vectors and index them. After the index is established, you can quickly retrieve it through the vector database using vectorized query statements.

[0075] After the vector database is built, the corresponding answer data can be retrieved based on the RAG (Retrieval-Augmented Generation) technology for the user's question, and the display data can be generated based on the answer data. The implementation process includes the following: Figure 2 Steps 210 to 220 are shown.

[0076] Step 210: Obtain a user question input by a user, and generate a target vector related to the user question.

[0077] The target vector includes a question vector mapped from the user question and / or a keyword vector mapped from question keywords; the question keywords are extracted from the user question.

[0078] As an example, after obtaining the user question, the user question may be mapped into a question vector, and the obtained question vector is the target vector described in step 210 .

[0079] As an example, after obtaining the user's question, question keywords can be extracted from the user's question and mapped into a keyword vector. The obtained keyword vector is the target vector described in step 210.

[0080] For example, when a user initiates a query and inputs a user question, the communication system can first convert the user question into a question vector through a vectorization model (such as a BERT model), and then perform answer retrieval based on the following step 220. The communication system supports users to provide feedback on the retrieval results, such as providing "satisfied" or "unsatisfied" options. If feedback information is obtained that the user is dissatisfied with the retrieval results, the question keywords are extracted from the user question. For example, the question keywords may be "optical cable break", "emergency failure", or "maintenance task", etc. The question keywords are converted into keyword vectors again through the vectorization model. Finally, the keyword vector is used to retrieve the answer through the following step 220. In this way, the accuracy of the retrieval results can be improved through a keyword-based re-retrieval mechanism.

[0081] Step 220: Acquire initial data matching the target vector from the vector database based on vector similarity, and filter out answer data from the initial data through one or more filtering processes.

[0082] Exemplarily, the search may be performed based on the RAG technique, for example, the initial data most similar to the target vector may be retrieved from the vector database based on the L2 distance or the cosine similarity. Then, the answer data may be filtered out from the initial data through one or more screening processes. The screening process may include:

[0083] 1) Based on the limiting conditions carried by the user question, answer data that meets the limiting conditions are screened from the initial data. The limiting conditions include time conditions and / or entity attribute conditions.

[0084] User questions may carry time conditions and / or entity attribute conditions, so answer data that meets the carried limiting conditions may be filtered out from the initial data.

[0085] For example, a communication optical cable break in a certain area triggers an emergency support warning, and the on-site maintenance team is performing fault repair tasks. At this time, if the user wants to view the latest fault information of a certain section of optical cable, the user will enter "Please show the latest fault information of optical cable A". However, the search results obtained based only on vector similarity will include some past fault records, resulting in the search results not meeting user expectations.

[0086] In this embodiment, by analyzing the time condition carried by the user's question, that is, "latest", the data that meets the "latest" condition can be screened out from the initial data obtained based on vector similarity. For example, if the difference between some fault times in the initial data and the current time exceeds a preset time threshold, a time window filter is added. The time window filter is used to screen out data from the initial data whose time difference with the current time does not exceed the time threshold as answer data. For example, only the optical cable fault records within the past 48 hours are screened out as answer data.

[0087] For another example, if a user wants to view the fault information of a communication optical cable exceeding a certain length, the user will enter "Please show the fault information of optical cables exceeding length B". However, the search results obtained based only on vector similarity will include some fault information of optical cables that do not exceed the specified length, resulting in the search results not meeting the user's expectations.

[0088] In this embodiment, by analyzing the entity attribute condition carried by the user question, that is, "length exceeds B", the answer data that meets the entity attribute condition can be screened out according to the entity attributes of each initial data, thereby filtering out irrelevant results.

[0089] In this way, through time window and entity attribute screening, the communication system can effectively filter out answer data that meets user expectations, improve retrieval accuracy, and help users quickly locate events they want to pay attention to.

[0090] 2) Performing semantic analysis on the user question to obtain semantic information of the user question, and filtering out answer data that conforms to the semantic information from the initial data.

[0091] By analyzing the semantics of user questions, answer data that better matches the semantics of the questions is filtered out from the initial data based on the semantic information.

[0092] For example, in the optical cable fault monitoring scenario, users need to check the status of the monitored optical cable, detect the faulty optical cable, and obtain environmental data of the fault location to display on the communication scheduling screen. Users can analyze the fault and dispatch the maintenance team to the site to handle it. At this time, if the user wants to query the "major" fault information related to a certain optical cable, he will enter "view the major fault information of optical cable A". However, the search results obtained based only on vector similarity will contain a large number of minor or irrelevant fault records, which fails to meet the user's query needs.

[0093] In this embodiment, the "major fault" in the user's question is understood as an event with "high impact and unclosed processing status" through semantic analysis. Then, the semantic analysis algorithm can be applied to the fault description field in the initial data to interpret "major" as a high fault level (such as "broken" or "affecting business"), and combined with the unresolved status, major faults are screened out. In this way, users can directly see historical records with a high impact, reducing manual screening time, providing accurate fault record display, and accelerating the decision-making process.

[0094] Subsequently, the obtained answer data can be input into the prompt template together with the relevance information. For example, the answer data can be input into a predefined placeholder position in the prompt template so that the generated prompt content includes the answer data. The multimodal visual language model then determines the display data based on the answer data.

[0095] It can be seen that this embodiment stores the relationship model in a vector database, and then retrieves the answer data matching the user question based on vector similarity, analyzes the limiting conditions in the user question, and performs semantic analysis on the user question, and finally generates prompt content based on the answer data, so that the multimodal visual language model generates corresponding display data based on the answer to the user question to meet the user's needs.

[0096] In addition, Figure 2 Based on the illustrated embodiment, after the answer data is obtained, the following steps 230 to 240 may be performed.

[0097] Step 230: Determine the user's focus content based on the user's operation behavior in the display area of ​​the communication scheduling screen.

[0098] The communication system supports judging the user's attention content by monitoring the user's operation behavior in the display area of ​​the communication scheduling screen. The operation behavior can be mouse movement, clicking, scrolling and other operations. For example, if the user repeatedly clicks on the same optical cable node or frequently checks the fault information of a certain device, the system will think that the user pays more attention to the optical cable node or the fault information of the device. For another example, if the user checks the ambient temperature data of the optical cable many times, the communication system will increase the weight of the optical cable and / or the ambient temperature data of the optical cable, and determine it as the user's attention content.

[0099] Step 240: When the answer data does not match the content of interest, a content of interest vector is generated based on the content of interest, and re-retrieved answer data matching the content of interest is obtained from the vector database based on vector similarity.

[0100] If the answer data obtained through steps 210-220 does not match the user's content of concern, a content of concern vector can be generated. Exemplarily, the focus keywords can be extracted from the content of concern, and the focus keywords can be mapped to the content of concern vector. Then, based on the vector similarity, the re-retrieval answer data matching the content of concern vector is retrieved from the vector database. For example, if the user's content of concern is "optical cable break location", focus keywords such as "optical cable failure" and "optical cable break" can be extracted, and based on this, the re-retrieval answer data is retrieved to improve the accuracy of the retrieval results.

[0101] As an example, after receiving a user question, the user question may be first mapped into a question vector, and after obtaining answer data through step 220, the answer data finally outputted may be adjusted by executing steps 230 to 240.

[0102] As an example, after receiving the user's question, the question keywords can be extracted from the user's question and mapped into keyword vectors. After the answer data is obtained through step 220, the answer data finally outputted is adjusted in steps 230-240.

[0103] As an example, the answer search may be performed based on the question vector and step 220 first, and when feedback information is received from the user that the user is dissatisfied with the answer data, the answer search may be performed based on the keyword vector and step 220 again, and / or the answer data may be regenerated and retrieved by executing steps 230-240.

[0104] It can be seen that the re-search mechanism provided in this embodiment includes two parts. The first part is to extract question keywords from user questions and perform re-search based on the question keywords to improve the retrieval accuracy. The second part is to determine the user's focus content based on the user's operating behavior in the display area, and perform re-search based on the focus content, so that the retrieval results are more relevant to the user's current focus content, thereby improving the retrieval accuracy. Among them, the three retrieval methods of "retrieval based on user questions", "retrieval based on question keywords" and "retrieval based on focus content" can be used in combination with each other, and the combination form is not limited to the three examples given above.

[0105] In addition, in some embodiments, in order to prevent the multimodal visual language model from encountering errors or "hallucinations" when updating the page layout, that is, the model generation result is inconsistent with the actual situation, this embodiment prompts the multimodal visual language model to first generate and update the page content according to the user input or preset requirements, which may include the update of text, numerical values, charts or other interface elements. After the data is updated, the generated page content can be saved in the cache. Subsequently, the page content in the cache is verified field by field using an external data source through the fact verification module in the communication system. Specifically, structured data comparison and text verification based on the knowledge base can be performed. The structured data comparison refers to comparing the numerical values ​​or other structured data in the page content with the real data item by item, and marking inconsistent data. Among them, the real data is obtained from any of the above databases. The text verification based on the knowledge base means that if the page content is updated with text content, the fact verification module can find facts related to the text content in the knowledge base based on the RAG retrieval mechanism, compare them, and mark inconsistent content. If the fact verification module verifies the data and text and obtains the marked data and / or marked content, the multimodal visual language model is triggered to update the generated page content and ensure that the regenerated page content passes the inspection of the fact verification module.

[0106] In this way, through multiple rounds of checks by the fact verification module, the multimodal visual language model can be gradually guided to generate an updated and accurate layout. Users can provide feedback and supplementary information in each round of checks to help the model gradually correct and improve its layout. This approach not only improves the accuracy of the layout, but also enhances the user's trust in the model output.

[0107] In addition, based on any of the above embodiments, the prompt content further includes one or more prompt information, so that the visual language model obtains the display data based on the prompt information.

[0108] As an example, the prompt information includes prompt information for prompting to mine data related to the user question from a preset database. In this way, the visual language model mines data related to the user question from the preset database using data mining means and displays it as data to be displayed in the communication scheduling screen.

[0109] As an example, the prompt information includes prompt information for prompting to call a multimodal pre-training model to obtain fusion data. Specifically, the text, image, video and other data displayed historically on the communication scheduling screen can be collected, and the data can be pre-processed and annotated so that different types of data can be effectively associated. Then a multimodal pre-training model, such as a CLIP (Contrastive Language-Image Pre-training, contrast language-image pre-training) model is used for training, and text, image, video and other data are mapped to the same vector space, so that the multimodal pre-training model can perform multimodal learning. After obtaining the multimodal pre-training model, the visual language model can call the multimodal pre-training model to fuse the multimodal data related to the user's problem obtained from multiple data sources, for example, mapped to the same vector space, and the obtained fusion data is the display data. In this way, different types of multimodal information related to user problems can be integrated and displayed in the communication scheduling screen, realizing the fusion display of different data types (including text, images, videos, sensor data) on the large screen, ensuring that users can see the most relevant content when interacting, and improving the ease of use of the communication scheduling screen.

[0110] As an example, the prompt information includes prompt information for prompting to call the knowledge graph. Specifically, the relevant entities of the optical cable and the relationships between the entities can be collected, and the knowledge graph of the optical cable can be constructed using multiple entities and their relationships. The entities include, for example, but are not limited to, optical cable types, fault types, maintenance steps, etc. In this way, the visual language model can use an inference engine (such as an OWL inference engine) to process user questions, and derive data related to the user questions as display data through the relationship chain in the knowledge graph. In this way, a question-and-answer system can be created that can understand user questions and provide accurate answers, ensuring that users can quickly obtain the required information. And support real-time updates of the knowledge graph to ensure that the displayed information is always the latest.

[0111] As another example, the prompt information includes prompt information for prompting the call of the adaptive model. Specifically, professional question and answer data in the field of optical cables can be collected and annotated to ensure that the system can adapt to the needs of different scenarios. The scene adaptive model is then trained using a contrastive learning method so that it can better distinguish the subtle semantic differences in different scenarios. Subsequently, the trained scene adaptive model can be used to automatically adjust the query parsing logic according to the context of the user's question and the question scenario, so that the displayed data is more relevant. In addition, the system can also use domain adaptation technology and contrastive learning to continuously adjust and optimize the scene adaptive model according to the user's interactive behavior (such as clicks, browsing time, etc.), so that the communication scheduling screen can understand and adapt to changes in user queries in specific optical cable scenarios.

[0112] Finally, the above-mentioned data mining methods, multimodal pre-trained models, knowledge graphs, and adaptive models can be integrated into a complete communication system in the form of software modules, so that the visual language model can generate layout data by calling corresponding auxiliary methods through the API (Application Programming Interface).

[0113] As described above, when multiple intelligent communication network tasks are unrelated, the display area of ​​the communication scheduling screen will be divided into independent task areas corresponding to multiple intelligent communication network tasks. At the same time, the communication system can start operation isolation to ensure that each user's operation does not interfere with other users. Exemplarily, by using data flow isolation technology, it can be ensured that each user's task area is independent. For example, virtualization technology can be used at the operating system level to achieve data and operation isolation to avoid cross-influence between tasks.

[0114] At the same time, when the intelligent communication network tasks change, the communication system can re-determine whether the multiple intelligent communication network tasks after the change are related. If they are related, the display area is re-divided, the operation isolation is lifted, the user's operation permissions in the task area and the shared area are adjusted, and the user is allowed to interact in the shared area. For example, multiple users have information viewing permissions and resource scheduling permissions in the shared area.

[0115] In addition, based on any of the above embodiments, the communication system may also be deployed with an RBAC (Role-Based Access Control) model. It is understandable that different users have different identities and roles in the communication system, and users with different identities have different user permissions and different data of interest. Therefore, the identity and user permissions of each user can be managed through the RBAC model. In this way, when a user logs into the communication system, the communication system can determine the user permission information based on the RBAC model. At the same time, the user permission information is input into a preset prompt template so that the generated prompt content includes the user permission information. In this way, the multimodal visual language model can determine the display data that matches the user permission information. For example, the core commander has the highest user permissions and has priority to see the global control information. The operator or technician only sees the operation details related to his task.

[0116] In addition, each target area includes multiple display data, and each display data is displayed in the target area in the form of an interface element. It can be understood that the communication scheduling screen displays content in the form of an interface, and the interface includes multiple elements, and different elements have the same or different attributes. Therefore, the display data is displayed in the target area in the form of an interface element. The interface elements are, for example, controls, components, charts, lists, and the like. Based on this, the area division result output by the multimodal visual language model also includes initial layout data for indicating the display position of each interface element in the target area. After obtaining the initial layout data, the multimodal visual language model will further optimize the initial layout data. Specifically, the multimodal visual language model can adjust the initial layout data based on the priority scores of multiple interface elements to obtain optimized layout data. As an example, the priority score of an interface element can be negatively correlated with the distance from the interface element to the center of the field of view of the target area. That is, the higher the priority score of the interface element, the closer the interface element is to the center of the field of view of the target area. Among them, the determination process of the center of the field of view will be unfolded below.

[0117] Exemplarily, the priority score of the interface element can be determined by following steps S1-S2:

[0118] S1: For each of the interface elements in each of the target areas, evaluate a first correlation score between the interface element and the corresponding intelligent communication network task, a second correlation score between the interface element and the user, and an urgency score of the intelligent communication network task; wherein the user is a user who executes the intelligent communication network task.

[0119] Exemplarily, the communication system can assign a task relevance model to each user. For each target area, after obtaining each interface element in the target area, the task relevance model can calculate the first relevance score between each interface element and the corresponding intelligent communication network task based on the intelligent communication network task corresponding to the user. The higher the first relevance score, the more relevant the interface element is to the intelligent communication network task. Optionally, the value range of the first relevance score is 0-1. For example, if the user is monitoring the device status, the first relevance score of the interface elements related to the device status is higher, while the first relevance score of other interface elements is lower. For another example, if the user is scheduling resources, the first relevance score of the interface elements related to resource scheduling is higher, while the first relevance scores corresponding to other tasks are lower.

[0120] The second relevance score is used to characterize the relevance between the interface element and the user. As mentioned above, users of different identities pay attention to different data, so the relevance between the interface element and the user can be characterized by the second relevance score. For example, the commander has a higher second relevance score for the interface element corresponding to the global information, while the technician has a higher second relevance score for the interface element corresponding to the operation detail data.

[0121] The urgency score of the intelligent communication network task is used to characterize the urgency of the task. For example, the urgency of the fault alarm task is high.

[0122] Step S2: determining the priority score of the interface element based on a preset task relevance weight, the first relevance score, a preset user relevance weight, the second relevance score, a preset task urgency weight, and the urgency score.

[0123] The communication system presets three weights corresponding to the scores, namely task relevance weight, user relevance weight and task urgency weight. After obtaining the above three scores, the scores are weighted using the corresponding weights, and the sum of the weighted scores is determined as the priority score of the interface element.

[0124] For example, the priority score S can be expressed as: S = W 任务 *R 任务 +W 用户 *R 用户 +W 紧急 *R 紧急 Among them, W 任务 is the preset task relevance weight, R 任务 is the first correlation score, W 用户 is the preset user relevance weight, R 用户 Score the second relevance, W 紧急 is the preset task urgency weight, R紧急 Score for urgency.

[0125] Optionally, after obtaining the priority score of each interface element, the interface elements can be divided into multiple priority levels based on the priority score. For example, the communication system can divide the interface elements into three levels of data: high priority layer, medium priority layer and low priority layer. Among them, the high priority layer includes critical task data, such as equipment failure, alarm information, etc. These data should be displayed to the user immediately. The medium priority layer includes routine data related to the task, such as the real-time status of the equipment, the progress of resource scheduling, etc., which can be placed in the central field of view but not conspicuous. The low priority layer includes information that is less relevant to the task, such as the status of other equipment, expired historical data, etc. These data can be faded or hidden.

[0126] It can be seen that this embodiment evaluates the priority of each interface element from these three dimensions by evaluating the relevance between each interface element and the intelligent communication network task, the relevance between each interface element and the user, and the urgency of the intelligent communication network task, and then displays the interface elements with high priority near the center of the field of vision, while relatively less important interface elements are placed away from the center of the field of vision, or even faded or hidden, so that users can focus on high-priority display data, avoid information overload, and improve task execution efficiency and response efficiency.

[0127] The visual field center in the target area refers to the area where the user's sight is relatively concentrated, and is the area that the user will focus on. Regarding the process of determining the visual field center, in some embodiments, it includes the following steps: Figure 3 Steps 310 to 320 are shown.

[0128] Step 310: Detect in real time the posture data of the user relative to the communication scheduling screen; the posture data includes the position data of the user relative to the communication scheduling screen, the distance data between the user and the communication scheduling screen, and the head posture data of the user.

[0129] Exemplarily, the position data such as the user's position relative to the communication scheduling screen, the distance from the screen, and the head posture can be detected in real time by a camera and / or an infrared sensor.

[0130] Step 320: Determine the user's field of view information based on the posture data; the field of view information includes a field of view range and a line of sight direction.

[0131] For example, a fusion algorithm can be used to map the posture data to the posture data in a three-dimensional coordinate system, and the user's head posture data can be combined to determine the user's field of view. In addition, a facial detection algorithm can be used to accurately identify the user and track his head movement in real time to determine the user's field of view direction, thereby obtaining field of view information including the field of view range and field of view direction.

[0132] The field of view information can then be input into the prompt template, and the generated prompt content includes the field of view information, so that the multimodal visual language model determines the position and size of the target area based on the field of view range in the field of view information, and determines the field of view center of the target area based on the line of sight direction in the field of view information.

[0133] In addition, projection transformation technology can also be used to adjust the angle of the displayed content. For example, the communication system uses an affine transformation algorithm to adjust the angle and direction of the interface elements in real time to make them consistent with the user's viewing angle, ensuring that the screen content can be clearly displayed at any angle. For example, multiple target areas can be rendered by a multi-view rendering engine.

[0134] For example, user A is located at the center of the screen, and users B and C are located on both sides of the screen. The communication system automatically adjusts the layout of the display area according to their positions, so that the global information is located in the center of user A, and the task data related to other users is located in the field of view of the corresponding user. And the communication system adjusts the interface display of the task area corresponding to each user through affine transformation, so that each user can clearly see the display data of his or her task area from his or her own perspective.

[0135] For another example, if it is assumed that user A is responsible for the communication equipment monitoring task, and user B performs the communication resource scheduling task, it can be seen from the above examples that the two tasks are related, so the display area of ​​the communication scheduling screen will be divided into two task areas and a shared area. If user A stands on the left side of the screen and user B stands on the right side of the screen, the communication system can dynamically adjust the position and size of the two task areas and the position of the shared area based on the positions of the two users through this embodiment to ensure that both users can see the equipment and resource scheduling information. Among them, since user A needs to know the detailed equipment status information, user B needs to know the resource scheduling data. Therefore, the detailed information of the equipment status can be displayed in the task area corresponding to user A, and the detailed data of the resource scheduling can be displayed in the task area corresponding to user B, to ensure that user A can see the equipment details more clearly, and user B can see the resource scheduling information more clearly, without interfering with each other.

[0136] It can be seen that this embodiment determines the position, size and visual field center of the target area through the user's posture data, so that when multiple users use the communication scheduling screen at the same time, the communication system can synchronize the viewing angle requirements of each user. The communication scheduling screen is controlled to generate different layout versions according to the different standing positions of each user, and the angle is adjusted in real time to meet the use needs of multiple users.

[0137] In addition, based on any of the above embodiments, the method may further include: Figure 4 Steps 410 - 430 are shown.

[0138] Step 410: receiving a zoom instruction input by a user; the zoom instruction is used to instruct to perform a zoom-in operation or a zoom-out operation on a target element in a target area.

[0139] Exemplarily, the zoom command can be input by gesture, touch, or voice. As an example, the user's gesture image can be captured by a camera, and the gesture image can be recognized by image recognition technology, such as a deep learning gesture recognition model, to obtain gesture data, and the gesture data can be preprocessed using a posture estimation algorithm (such as OpenPose or MediaPipe), including extracting key points, analyzing the direction, amplitude and speed of the gesture, and obtaining the zoom command. As an example, the zoom command input by the user can be directly obtained through a touch device. As another example, a speech recognition module (such as Google Speech-to-Text or Microsoft Azure speech recognition service) can be used to recognize the user's voice. After speech recognition, the user's intention is analyzed by a natural language processing model to identify the zoom command in the user's voice.

[0140] Step 420: After controlling the target element to adjust its size based on the zoom instruction, determine the element sizes of all interface elements included in the target area.

[0141] The zoom instruction carries the zoom ratio of the target element, which is used to instruct the target element to be zoomed according to the zoom ratio. Exemplarily, if the zoom instruction is a voice instruction, the user can directly indicate the size of the zoom ratio in the voice. Exemplarily, if the zoom instruction is a gesture instruction, the ratio of the current distance between the two touch points of the gesture instruction and the initial distance can be calculated based on the scaling algorithm as the zoom ratio.

[0142] In this way, the size of the target element after resizing can be calculated based on the scaling algorithm, and then the size information of each interface element in the target area can be obtained. The size information of the interface element includes the size information of the target element after scaling and the size information of other interface elements other than the target element that have not been scaled.

[0143] Step 430: Based on the element sizes of all the interface elements, adjust the position of the interface element in the target area; wherein the element size of the interface element is negatively correlated with the distance from the interface element to the center of the field of view of the target area.

[0144] Subsequently, the position of the interface element in the target area can be adjusted based on the element size of all interface elements, so that the element size is negatively correlated with the distance from the interface element to the center of the field of view. That is, the arrangement of the interface elements on the screen is adjusted according to the zoom ratio. For the enlarged interface element, it is considered to have increased its priority, so it is moved closer to the center of the field of view, and the positions of other low-priority interface elements are automatically adjusted to ensure that the display of important information is not affected.

[0145] At the same time, in order to ensure the readability and integrity of information and avoid distortion of graphics or text when zooming, the Canvas API can be used to handle graphic display to ensure that the displayed data remains clear at any zoom ratio. And an adaptive container can be defined for each target element, and the container dynamically adjusts its size according to the zoom ratio. For example, CSS Flexbox or Grid layout can be used, so that the communication system can intelligently adjust the arrangement of content to ensure that the interface remains neat and orderly even at different zoom levels.

[0146] In addition, in order to reduce the delay between the zoom command and the interface update, technologies such as WebSocket or FirebaseRealtime Database are also used to achieve low-latency data transmission. When the user performs a zoom operation, the system sends the operation and the zoom ratio to other related users in real time. That is, a certain display data is displayed in multiple target areas at the same time, and the intelligent communication network tasks of two users are related, then the zoom command entered by a user for the interface element of the display data also acts on the interface elements in other target areas. For example, user A finds a fault in equipment monitoring and enlarges the fault status through the zoom function. The task area of ​​user B also synchronously displays the enlarged equipment fault status to better schedule resources.

[0147] Regarding the fine-tuning process of the multimodal visual language model, the ScreenAI model is used for expansion. In order to enable the ScreenAI model to have the layout capability of dividing the display area based on the correlation between multiple tasks, in some embodiments, the training data of the ScreenAI model may include multiple independent intelligent communication network tasks and corresponding historical layout data, as well as multiple related intelligent communication network tasks and corresponding historical layout data. The ScreenAI model can learn from the training data how to divide the area based on the correlation between multiple tasks. Among them, the historical layout data may include historical screenshots of the display area of ​​the communication scheduling screen, and / or historical HTML data corresponding to the display area. In addition, in some embodiments, the fine-tuning process of the ScreenAI model may also include the following process:

[0148] First, a data storage mechanism is established to record the data of multiple rounds of user interactions with the communication scheduling screen and the question link. Then, based on the data storage mechanism, the user's historical behavior data is collected, including each user's operation, click, input question and other interactive behaviors in the interface, and the final result of the correlation between each user's current task and the task is recorded. The collected data can then be cleaned and preprocessed to remove invalid or noise data. And the data is converted into a format suitable for ScreenAI model processing. The ScreenAI model is then fine-tuned using the accumulated historical behavior data. Specifically, a suitable fine-tuning strategy can be determined based on the architecture of the ScreenAI model and the available tools, such as adjusting the parameters of certain layers of the model, adding new layers, or using specific optimization algorithms. In this embodiment, it is selected to add a LoRA layer to the ScreenAI model, which generally involves adding additional weight matrices to the linear layer and / or attention layer of the model. Define the loss function and optimizer, specifically, determine the evaluation indicators for predicting the user's next question, such as accuracy, recall rate, or F1 value (F1Score), etc. And construct an objective function based on the evaluation indicators to optimize the model during the training process. The optimizer is used to train the parameters of the LoRA layer, including but not limited to the AdamW optimizer.

[0149] Subsequently, the LoRA layer can be trained using historical behavior data to adjust the parameters of the LoRA layer. For example, standard back propagation and gradient descent can be used for training, so that the model can learn the underlying patterns and regularities between the question content. Experiment with different hyperparameters, such as learning rate, number of training rounds, regularization parameters, etc., to find the optimal model configuration. And record performance indicators during training, such as loss and accuracy on the validation set. The performance of the fine-tuned ScreenAI model can then be evaluated on the test set. If the performance is poor, the problem can be analyzed based on the evaluation results, such as insufficient data, inappropriate model architecture, or poor hyperparameter settings. And make corresponding improvements, such as adjusting hyperparameters, adjusting model structure, or collecting more user behavior data for re-fine-tuning until satisfactory results are achieved.

[0150] After completing the model fine-tuning, you can save the parameters of the LoRA layer, integrate the fine-tuned ScreenAI model into the communication system, and configure the initial UI layout and related settings. In this way, by continuously optimizing the model parameters, the ScreenAI model can learn the user's behavior patterns and preferences to improve prediction accuracy.

[0151] The following two examples illustrate this:

[0152] Example 1: In the optical cable fault monitoring scenario, the user will monitor the status of the communication optical cable and detect the faulty communication optical cable. At this time, it is necessary to obtain the environmental data of the fault location and display it on the communication scheduling screen, so that the user can analyze the fault and dispatch the maintenance team to the site for processing. Based on this, the communication system can obtain the task details (including task ID, task status, etc.) in the first relationship mode of the intelligent communication network task entity, associate the optical cable fault entity through the task ID, and extract the fault information of the faulty optical cable (including fault ID, optical cable ID, fault time, fault type, etc.) from the first relationship mode of the optical cable fault entity. Subsequently, according to the optical cable ID, the communication system can further associate the optical cable routing monitoring entity with the dynamic environment monitoring entity, and obtain the detailed information of the faulty optical cable from the first relationship mode of the optical cable routing monitoring entity, and obtain the environmental data of the faulty optical cable from the first relationship mode of the dynamic environment monitoring entity. Next, the communication system can associate the emergency guarantee entity through the task ID, and obtain the emergency type, scheduling situation, processing progress, etc. from the first relationship mode of the emergency guarantee entity. Then, the user entity is associated with the task ID, and the user role and authority are obtained in the first relationship mode of the user entity, and the display data matching the user authority information is determined. At the same time, the user station entity is associated with the user ID, and the task area and the center of vision are determined according to the user position, distance, posture angle and correction angle recorded in the first relational model of the user station entity, so as to complete the division of the display area and ensure that the displayed data conforms to the user's perspective. Finally, through the above association, the "view optical cable fault monitoring and processing task logic table" can be obtained, which is expressed as: Faults (fault ID, optical cable ID, fault time, fault type, fault level, processing status, temperature, humidity, voltage, monitoring time, optical cable name, optical cable length, fiber core utilization rate, maintenance level, business carrying capacity, status, emergency type, personnel location, scheduling status, processing progress, user ID, user name, role, authority, user location, distance, posture angle, correction angle, task ID, task status, screen partition ID, partition status)

[0153] Example 2: Based on any of the above embodiments, the prompt content may be as follows:

[0154] “prompt: Role: UI designer; Task: Please design a new screen layout based on the following scenario. To meet the needs of multiple users on the same large screen, the displayed content should be automatically adjusted according to the user's role, task and position. The layout needs to be dynamically adjusted according to the user's task relevance, role authority, position and perspective to ensure that each user can see the optimal information display related to their task and authority. The specific requirements are as follows:

[0155] Get task-related data: Previously, we continuously recorded the relationship patterns of many user task scenarios. These relationship patterns contain task ID, task layout information, user permissions, user location, optical cable fault information, fault level and other related data, which are stored in the RAG vector library in this structure. Here, we need to retrieve these relationship patterns in the RAG library to obtain the task layout information, user permissions and other data related to the user task according to the task content input by each user or voice prompt.

[0156] Layout design: Based on the task content of each user input or voice prompt, the previous layout information of these tasks is retrieved from the RAG vector library. You can make further optimizations based on this layout information or use this layout directly and return this layout information to me in HTML format. Then the agent completes the page layout update based on the task-related data retrieved above and the returned HTML format.

[0157] If the updated layout is "unsatisfactory" according to user feedback, then the search will be repeated again based on the "FAIss task keyword re-search mechanism".

[0158] Finally, the user's position may change. By retrieving the {rules for field of view optimization and adjustment} in the RAG vector database, the field of view is visualized and adjusted in real time according to the retrieved rules, ensuring that each user can see the information related to their task from their own perspective. "

[0159] For example, multiple users perform their respective tasks on the same screen. Task 1: User A is responsible for monitoring optical cable faults, mainly monitoring the status of optical cables in various areas, promptly discovering any optical cable faults, and marking the specific location of the problem. Task 2: User B is responsible for fault cause analysis and location. When an abnormal fault occurs, User B needs to immediately analyze the cause of the fault (such as external interference, equipment damage, etc.), determine the scope of impact, and assist in determining the optimal repair plan. Task 3: User C needs to check the optical cable warning.

[0160] For Task 1 and Task 2, the database can be searched and queried through RAG retrieval technology to determine the task ID, task layout information, partition status, user authority, user location, optical cable fault information, fault level and other related data corresponding to Task 1 and Task 2 respectively. When it is determined that Task 1 is related to Task 2, the system can automatically allocate task areas and shared areas for users A and B, and output the current layout in HTML (HyperText Markup Language) format, and optimize the layout according to the retrieved display data and its priority, and update to achieve real-time collaborative work between the two users. At the same time, the field of view optimization, field of view adjustment, and dynamic zooming are completed in real time according to the position and perspective of users A and B, so that they can clearly view the content related to their tasks.

[0161] For Task 1 and Task 3, the database is searched and queried through RAG retrieval technology to obtain the task ID, task layout information, user authority, user location, optical cable fault information, fault level and other related data corresponding to Task 1 and Task 3. When it is determined that Task 1 and Task 3 are not related, the system can automatically assign independent task areas to User A and User B and enable the operation isolation function. Each user's task area is completely independent to ensure that there will be no interference due to the operations of other users. Finally, the layout optimization is completed based on the retrieved display data and its priority.

[0162] Among them, for Task 2, when user B asks a question, the answer data corresponding to the user's question can be retrieved based on the RAG technology and the pre-built vector database, and the display data can be generated based on the answer data. Specifically, the FAISS index can be used to quickly find the most relevant data as the answer data through vector similarity. At the same time, the re-retrieval mechanism can be triggered at an appropriate time, including re-retrieval based on question keywords and / or re-retrieval based on the content of interest, to further optimize the retrieval accuracy. At the same time, supplementary logic such as semantic analysis, time limitation, and entity attribute limitation can be introduced to further optimize the retrieval results of the answer data.

[0163] Based on any of the above embodiments, the present application also provides a multi-modal intelligent communication network real-time monitoring system, the system is used to realize the display of real-time monitoring data of the intelligent communication network on the communication scheduling screen; the system includes:

[0164] A correlation module, configured to determine correlation information between the plurality of intelligent communication network tasks in response to the plurality of intelligent communication network tasks by using a pre-established task association relationship;

[0165] A division module, used to generate prompt content based on the correlation information, the current display data of the display area and a preset prompt template, and input the prompt content into a trained multimodal visual language model to obtain a region division result generated by the multimodal visual language model for the display area of ​​the communication scheduling screen; the region division result is used to indicate a plurality of target regions of the display area division, and display data of each of the target regions;

[0166] Wherein, if the correlation information indicates that the multiple intelligent communication network tasks are unrelated, the multiple target areas include task areas corresponding to the intelligent communication network tasks one by one; if the correlation information indicates that the multiple intelligent communication network tasks are related, the multiple target areas include multiple task areas and shared areas; the display data of each task area includes the real-time monitoring data of the corresponding intelligent communication network task; the display data of the shared area includes the association data between the multiple related intelligent communication network tasks;

[0167] A display module is used to display the real-time monitoring data of the intelligent communication network on the communication scheduling screen based on the area division result generated by the multimodal visual language model.

[0168] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0169] Based on the multi-modal intelligent communication network real-time monitoring method described in any of the above embodiments, the present application also provides a computer program product, which includes one or more computer programs or instructions. The computer program or instruction can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. When the computer program is executed by a processor, the multi-modal intelligent communication network real-time monitoring method described in any of the above embodiments is implemented.

[0170] Based on the multi-modal intelligent communication network real-time monitoring method described in any of the above embodiments, the present application also provides: Figure 5 A schematic diagram of the structure of an electronic device is shown in FIG. Figure 5 At the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement a multi-modal intelligent communication network real-time monitoring method described in any of the above embodiments.

[0171] The present application also provides a computer storage medium, which stores a computer program. When the computer program is executed by a processor, it can be used to execute a multi-modal intelligent communication network real-time monitoring method described in any of the above embodiments.

[0172] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0173] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0174] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0175] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0176] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0177] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A multi-modal intelligent communication network real-time monitoring method, characterized in that: The method is used to realize displaying the real-time monitoring data of the intelligent communication network on the communication scheduling screen; the method comprises: In response to a plurality of intelligent communication network tasks, determining correlation information between the plurality of intelligent communication network tasks using a pre-established task association relationship; Generate prompt content based on the correlation information, the current display data of the display area and a preset prompt template, and input the prompt content into a trained multimodal visual language model to obtain a region division result generated by the multimodal visual language model for the display area of ​​the communication scheduling screen; the region division result is used to indicate a plurality of target regions of the display area division, and display data of each of the target regions; Wherein, if the correlation information indicates that the multiple intelligent communication network tasks are unrelated, the multiple target areas include task areas corresponding to the intelligent communication network tasks one by one; if the correlation information indicates that the multiple intelligent communication network tasks are related, the multiple target areas include multiple task areas and shared areas; the display data of each task area includes the real-time monitoring data of the corresponding intelligent communication network task; the display data of the shared area includes the association data between the multiple related intelligent communication network tasks; Based on the area division result generated by the multimodal visual language model, real-time monitoring data of the intelligent communication network is displayed on the communication scheduling screen.

2. The method according to claim 1, characterized in that The method further comprises: Pre-storing a first relational model of each relevant entity of the intelligent communication network and a second relational model for characterizing the relationship between a plurality of the relevant entities in a query database; The related entities include one or more of an optical cable routing monitoring entity, a dynamic environment monitoring entity, an optical cable fault entity, an emergency support entity, an intelligent communication network task entity, a screen layout entity, a user station entity, and a user entity; the first relationship model of each of the related entities includes entity attributes of the related entity, and one or more of the entity attributes are entity codes of the related entity; the second relationship model records entity codes of multiple related entities that have relationships; The prompt content also includes a target entity code corresponding to each of the intelligent communication network tasks, so that the multimodal visual language model determines the target entity that has a relationship with the intelligent communication network task entity based on the target entity code and the second relationship model, and determines the entity attributes recorded in the first relationship model of the target entity as the display data.

3. The method according to claim 2, characterized in that The first relationship model and the second relationship model are stored in a vector database in the form of vectors; the method further includes: Acquire a user question input by a user, and generate a target vector related to the user question; wherein the target vector includes a question vector mapped from the user question and / or a keyword vector mapped from question keywords; the question keywords are extracted from the user question; Acquiring initial data matching the target vector from the vector database based on vector similarity, and filtering out answer data from the initial data through one or more filtering processes; The screening process includes: based on the limiting conditions carried by the user question, screening the answer data that meets the limiting conditions from the initial data, wherein the limiting conditions include time conditions and / or entity attribute conditions; or performing semantic analysis on the user question to obtain semantic information of the user question, and screening the answer data that meets the semantic information from the initial data; The prompt content includes the answer data, so that the multimodal visual language model determines the display data based on the answer data.

4. The method according to claim 3, characterized in that The prompt content also includes one or more prompt information, so that the visual language model obtains the display data based on the prompt information; the prompt information includes: Prompt information for prompting to mine data related to the user question from a preset database, wherein the displayed data includes the data related to the user question obtained by mining; Prompt information for prompting to call a multimodal pre-trained model to obtain fused data, wherein the displayed data includes fused data obtained by mapping data related to the user question obtained from multiple data sources to the same vector space using the multimodal pre-trained model; wherein the multimodal pre-trained model is trained using text, image, and video data historically displayed on the communication scheduling screen; Prompt information for prompting to call a knowledge graph, wherein the displayed data includes knowledge data in the knowledge graph related to the user question; wherein the knowledge graph is constructed using relationships between multiple entities of an optical cable; The prompt information is used to prompt the calling of the adaptive model, and the display data includes display data obtained based on the context and / or scenario of the user question by using the adaptive model.

5. The method according to any one of claims 1 to 4, characterized in that: The prompt content also includes user authority information, so that the multimodal visual language model determines display data matching the user authority information; There are multiple display data for each target area, and each display data is displayed in the corresponding target area in the form of an interface element; the area division result also includes initial layout data for indicating the display position of each interface element in the target area, so that the multimodal visual language model adjusts the initial layout data based on the priority scores of the multiple interface elements to obtain optimized layout data; wherein the priority score of the interface element is negatively correlated with the distance from the interface element to the visual field center of the target area; The priority score of the interface element is determined by the following steps: For each of the interface elements in each of the target areas, a first correlation score between the interface element and the corresponding intelligent communication network task, a second correlation score between the interface element and the user, and an urgency score of the intelligent communication network task are evaluated; wherein the user is a user who executes the intelligent communication network task; The priority score of the interface element is determined based on a preset task relevance weight, the first relevance score, a preset user relevance weight, the second relevance score, a preset task urgency weight, and the urgency score.

6. The method according to claim 5, characterized in that The method further comprises: Real-time detection of the user's posture data relative to the communication scheduling screen; the posture data includes the user's position data relative to the communication scheduling screen, the distance data between the user and the communication scheduling screen, and the user's head posture data; Determine the user's field of view information based on the posture data; the field of view information includes a field of view range and a line of sight direction; The prompt content also includes the field of view information, so that the multimodal visual language model determines the position and size of the target area based on the field of view range, and determines the field of view center of the target area based on the line of sight direction.

7. The method according to claim 1, characterized in that The method further comprises: Receiving a zoom instruction input by a user; the zoom instruction is used to instruct to perform a zoom-in operation or a zoom-out operation on a target element in a target area; After controlling the target element to adjust its size based on the scaling instruction, determining the element sizes of all interface elements included in the target area; Based on the element sizes of all the interface elements, the position of the interface element in the target area is adjusted; wherein the element size of the interface element is negatively correlated with the distance from the interface element to the center of the field of view of the target area.

8. A multi-modal intelligent communication network real-time monitoring system, characterized in that: The system is used to realize the display of real-time monitoring data of the intelligent communication network on the communication scheduling screen; the system includes: A correlation module, configured to determine correlation information between the plurality of intelligent communication network tasks in response to the plurality of intelligent communication network tasks by using a pre-established task association relationship; A division module, used to generate prompt content based on the correlation information, the current display data of the display area and a preset prompt template, and input the prompt content into a trained multimodal visual language model to obtain a region division result generated by the multimodal visual language model for the display area of ​​the communication scheduling screen; the region division result is used to indicate a plurality of target regions of the display area division, and display data of each of the target regions; Wherein, if the correlation information indicates that the multiple intelligent communication network tasks are unrelated, the multiple target areas include task areas corresponding to the intelligent communication network tasks one by one; if the correlation information indicates that the multiple intelligent communication network tasks are related, the multiple target areas include multiple task areas and shared areas; the display data of each task area includes the real-time monitoring data of the corresponding intelligent communication network task; the display data of the shared area includes the association data between the multiple related intelligent communication network tasks; A display module is used to display the real-time monitoring data of the intelligent communication network on the communication scheduling screen based on the area division result generated by the multimodal visual language model.

9. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing processor-executable instructions; Wherein, when the processor calls the executable instructions, the operation of any method described in claims 1-7 is implemented.

10. A computer-readable storage medium, characterized in that: Computer instructions are stored thereon, and when the computer instructions are executed by a processor, the steps of any method described in claims 1-7 are implemented.

Images