Multi-modal intention-driven intelligent chart dynamic generation method and system

Abstract

The invention discloses a multi-modal intention-driven intelligent chart dynamic generation method and system. The system comprises a user input module, a semantic analysis module, a data matching and extracting module, a chart generation module, a chart visual arrangement module and a document linkage module. The method comprises the steps that a user inputs a query requirement through a text or voice, and after a system receives the input, semantic analysis is conducted on an input natural language instruction; the system extracts data related to an entity from a knowledge graph and a related data source in the field of power dispatching; the system automatically selects a suitable chart type according to the extracted data and generates an interactive chart. The problem that a traditional method depends on data source selection and chart style adjustment in the chart generation process is solved, a user can directly obtain the needed chart through natural language input, and the efficiency and flexibility of chart generation are improved.

Description

Technical Field

[0001] This invention relates to the field of power dispatching, and in particular to a method and system for dynamic generation of intelligent charts driven by multimodal intent. Background Technology

[0002] With advancements in data analysis and processing technologies, traditional chart generation techniques largely rely on manual selection of data sources and chart templates, lacking flexible interactivity and the ability to adjust data display formats in real time. This is particularly problematic in complex fields such as power dispatching, which involve large amounts of heterogeneous data. Traditional chart generation methods cannot quickly respond to user needs and update in real time, leading to a disconnect between data and business semantics.

[0003] Chinese Patent Publication No. CN113961597A discloses a method and apparatus for implementing power dispatching services based on business semantics. The method includes: traversing a business process library and constructing a microservice knowledge base and a business semantic knowledge base according to a microservice identification algorithm and a business semantic knowledge extraction algorithm, respectively; obtaining a second business semantic model in the business semantic knowledge base that does not match the first business semantic model according to a business semantic model similarity algorithm; obtaining a first business process corresponding to the second business semantic model and obtaining a first microservice component that has a mapping relationship with the first business process; identifying the first microservice node in the first microservice component; if the IPO information in the first microservice node is different from the preset IPO information, then using a similarity algorithm to identify a second microservice node in the microservice knowledge base for replacement, thereby generating a new power dispatching platform business process. Chinese patent CN119298027A discloses a power intelligent dispatching system and method, belonging to the field of data processing technology. The method includes: Step S1: Obtaining the location information and electricity consumption category of all distribution rooms in the target area, as well as historical power supply data and total power supply data; Step S2: Dividing the target area into multiple sub-areas, and integrating distribution rooms with the same electricity consumption category into three group levels within each sub-area; Step S3: Inputting each mean sequence into the prediction model, and setting the prediction result as the electricity demand of each distribution room in each group level at the prediction time; Step S4: Calculating the first total electricity demand of each electricity consumption category distribution room in each sub-area at the prediction time and the second total electricity demand of the target area, and performing power dispatching. Both methods suffer from the problem that chart generation cannot quickly respond to user needs and update in real time, leading to a disconnect between data and business semantics. Therefore, there is an urgent need for a dynamic chart generation technology based on deep semantic understanding and data association. Summary of the Invention

[0004] Purpose of the invention: The purpose of this invention is to provide a method and system for dynamic generation of intelligent charts driven by multimodal intent.

[0005] Technical solution: The multimodal intent-driven intelligent chart dynamic generation system of the present invention includes a user input module, a semantic parsing module, a data matching and extraction module, a chart generation module, a chart visualization and arrangement module, and a document linkage module;

[0006] The user input module: obtains the user's query intent through text or voice input;

[0007] The semantic parsing module: based on a preset semantic model, it parses the natural language commands input by the user and identifies relevant entities and query requirements in the power dispatching field;

[0008] The data matching and extraction module combines multi-source heterogeneous data stored in the knowledge graph, automatically matches the corresponding data source by parsing entity relationships, and extracts key indicators.

[0009] The chart generation module optimizes the selection of chart types through a generation adversarial network, generating different forms of interactive dynamic charts.

[0010] The chart visualization and arrangement module provides a drag-and-drop visualization interface through which users can adjust chart elements and update chart data logic and display effects in real time.

[0011] The document linkage module supports the linkage between charts and document content. When a user clicks on a block in a chart, the system automatically locates and displays document fragments related to that data, making it easier for the user to view detailed background information.

[0012] The multimodal intent-driven intelligent chart dynamic generation method of the present invention includes the following steps:

[0013] (1) Users input their query requirements via text or voice. After receiving the input, the system performs semantic parsing on the input natural language instructions.

[0014] (2) In the data matching and extraction stage, the system extracts entity-related data from the knowledge graph and related data sources in the field of power dispatching;

[0015] (3) The system automatically selects the appropriate chart type based on the extracted data and generates interactive charts; combined with BERT’s powerful contextual understanding capabilities, chart generation not only relies on traditional rules and templates, but can also be intelligently optimized based on the actual meaning of the data;

[0016] (4) After the chart is generated, the system needs to link the chart with the document content so that users can obtain relevant document or report information when viewing the chart, so as to gain a deeper understanding of the background and business logic of the data.

[0017] Furthermore, step (1) includes user input text preprocessing, entity recognition, dependency parsing, and output of comprehensive results.

[0018] Furthermore, the entity recognition includes: BERT uses its self-attention mechanism to generate a context-dependent representation for each word and performs entity classification based on this representation; the segmented text is input into the BERT model, and BERT processes the entire text sequence to generate a representation of each word in the context;

[0019] The output generated by BERT is a contextual representation of each word:

[0020]

[0021] The output of the BERT model is further processed through a classification layer to predict the entity label for each word; assuming the output of BERT is ht, entity classification can be performed using Softmax:

[0022]

[0023] Among them, y t It is word x t The corresponding labels are W, which is the weight of the classification layer, and b is the bias term.

[0024] Furthermore, step (2) includes determining the data source, data matching, data extraction and transformation, and data quality control.

[0025] Furthermore, step (3) includes chart type selection, chart optimization and style adjustment, dynamic updating of data and chart association, interactive chart generation, and combination of chart generation and data optimization.

[0026] Furthermore, the chart type selection includes using BERT to perform deep semantic analysis on the input text to understand the user's intent and the relationship between the data, expressed by the following formula:

[0027]

[0028] in, Let y represent the optimal chart type, y be the candidate set of chart types, and X be the context-sensitive text representation output by BERT. This represents the probability distribution of selecting chart type y given context information.

[0029] Furthermore, the dynamic updates of the data and charts depend on the chart's data update formula:

[0030]

[0031] Among them, y new For the updated chart data, y old The old data state is represented by u_a, where user operation is represented, and n_data is the new data; the system uses function f u The data is processed to generate an updated chart.

[0032] Furthermore, the combination of chart generation and data optimization is described by the following mathematical model:

[0033]

[0034] Among them, f v (y,data) represents the degree of match between the chart's visualization and the data, f s (y, X) represents the degree of match between the chart and the user's query intent.

[0035] Furthermore, step (4) includes chart click event processing, BERT-based document content retrieval, and dynamic document display.

[0036] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: The present invention provides a multimodal intent-driven intelligent chart dynamic generation method and system, which can automatically parse user query intent through natural language understanding and generate interactive charts in real time based on multi-source heterogeneous data; it solves the problem of dependence on data source selection and chart style adjustment in the chart generation process of traditional methods, enabling users to directly obtain the required charts through natural language input, and dynamically adjust the chart content and style through visual arrangement tools, thereby improving the efficiency and flexibility of chart generation. Attached Figure Description

[0037] Figure 1 This is a flowchart of the multimodal intent-driven intelligent chart dynamic generation method described in this invention. Detailed Implementation

[0038] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0039] This invention is primarily applied in the field of power dispatching to dynamically generate optimal visualization charts. The system combines multiple input methods such as natural language, gestures, and voice, and utilizes Large Language Modeling (LLM) and machine vision technology to automatically parse user requirements and generate charts adapted to dispatching scenarios in real time, providing intelligent visualization support for power dispatchers. The implementation steps of this invention are as follows: Figure 1 As shown:

[0040] (1) User input and semantic parsing; Users input their query requirements via text or voice. After receiving the input, the system first needs to perform semantic parsing on the input natural language instructions. The purpose of semantic parsing is to extract key information from the user's input, such as the query object, query indicators, and query time range.

[0041] (1.1) User input text preprocessing

[0042] Users express their query needs through text input, and the input text undergoes preprocessing. This includes: segmenting the input text into individual words or sub-tokens and mapping each word or sub-token to a corresponding word vector.

[0043] (1.2) Entity Recognition (NER)

[0044] The BERT model, through its powerful context modeling capabilities, can simultaneously consider the surrounding context information of each word. In the Entity Recognition (NER) task, BERT uses its self-attention mechanism to generate a context-sensitive representation for each word and performs entity classification based on this representation.

[0045] The segmented text is input into the BERT model, which processes the entire text sequence and generates a representation of each word in its context.

[0046] The output generated by BERT is a contextual representation of each word.

[0047]

[0048] The output of the BERT model is further processed through a classification layer (usually a softmax classifier) ​​to predict the entity label for each word. Assuming the output of BERT is `ht`, entity classification can be performed using softmax:

[0049]

[0050] Among them, y t It is word x t The corresponding labels are W, which represents the weight of the classification layer, and b, which is the bias term. In this way, BERT can accurately identify various entities in the text.

[0051] (1.3) Dependency parsing

[0052] In addition to entity recognition, BERT can also assist in dependency parsing by further analyzing the grammatical relationships between words to clarify the relationships between entities. Although BERT itself is not specifically optimized for dependency parsing tasks, its powerful contextual understanding capabilities enable it to effectively participate in dependency parsing.

[0053] (1.4) Output of comprehensive results

[0054] After BERT's entity recognition and dependency parsing, the system can comprehensively extract various types of information from the user input, including entity recognition and entity relationships.

[0055] (2) Data matching and extraction

[0056] During the user input and semantic parsing phase, the system has successfully identified and extracted entity information and their relationships. Next, in the data matching and extraction phase, the system needs to extract data related to these entities from the knowledge graph and relevant data sources in the power dispatching domain.

[0057] (2.1) Determine the data source

[0058] The system constructs data query conditions based on entity information extracted from the BERT model. Based on the semantic parsing results, the system queries relevant knowledge graphs or databases to obtain relevant data sources. These data sources include: equipment operating status data, power dispatch reports, system alarm logs, and historical operating parameters.

[0059] (2.2) Data matching

[0060] Once the query criteria are identified, the system will match these criteria with nodes and relationships in the knowledge graph through semantic mapping. Utilizing the definitions of device entities in the graph, as well as time and state attributes, the system automatically selects relevant datasets.

[0061] (2.3) Data extraction and transformation

[0062] The extracted raw data is usually unstructured or semi-structured, and the system needs to transform it into structured data suitable for chart display. This process includes:

[0063] Extract data from text, report, or log files and convert it into tabular or time series formats, extracting specific metrics or data points based on user query requirements.

[0064] (2.4) Data quality control

[0065] After data matching and extraction, the system also needs to perform quality control to ensure the accuracy and consistency of the data. By verifying the integrity of the data and removing duplicates, the system can ensure that the core data required to generate charts is reliable and of high quality.

[0066] (3) Chart generation and optimization

[0067] After the data matching and extraction phase is completed, the system enters the chart generation and optimization phase. In this phase, the system automatically selects the appropriate chart type based on the extracted data and generates interactive charts. Leveraging BERT's powerful contextual understanding capabilities, chart generation not only relies on traditional rules and templates but also performs intelligent optimization based on the actual meaning of the data.

[0068] (3.1) Chart type selection

[0069] Choosing the right chart type is a core step in chart generation. Selecting the correct chart type is crucial for effectively conveying information. Traditional chart selection methods often rely on the basic attributes of the data, but by combining BERT's capabilities, the system can understand the semantics of the data based on context, thus making a more accurate selection.

[0070] (3.2) Chart optimization and style adjustment

[0071] Once the chart type is determined, the next step for the system is to optimize the chart's style and design its interactivity. This process typically includes adjustments in the following areas:

[0072] Axis adjustment: The system needs to adjust the scale and range of the coordinate axes according to the range and units of the data.

[0073] Color mapping: Different data dimensions may require different colors to highlight them, especially in heatmaps and other multi-dimensional data visualizations. BERT can intelligently select the appropriate color mapping scheme for the data by understanding the context.

[0074] Data filtering: Users may want to focus on specific time periods, devices, or states. The system can allow users to adjust the displayed content through an interactive interface. In this case, the chart data needs to be updated in real time.

[0075] The goal of optimization is to make data easier to understand through visualization styles, while enhancing interactivity so that users can flexibly adjust the content displayed in the charts.

[0076] Chart style optimization can be modeled using optimization problems:

[0077]

[0078] in, An optimization function representing chart clarity. An optimization function representing the interactivity of a chart. This is an optimized chart style; the goal is to maximize chart clarity and interactivity to improve user readability and experience.

[0079] (3.3) Dynamic updates of data and chart associations

[0080] As users interact with and adjust the charts, the content and structure of the charts need to be updated dynamically. To update the data and charts in real time, the system combines the data processing module and the front-end interaction design to update the charts via real-time data streaming.

[0081] When data is updated, the system recalculates the data and updates the chart content based on user actions. This process relies on the chart's data update formula:

[0082]

[0083] Among them, y new For the updated chart data, y old The old data state is represented by u_a, where user operation is represented, and n_data is the new data; the system uses function f u The data is processed to generate an updated chart.

[0084] (3.4) Interactive chart generation

[0085] The system not only generates static charts but also enhances the user experience through interactive chart design. Interactive charts allow users to dynamically adjust the displayed content through dragging, zooming, and other operations.

[0086] Interactive charts can be generated using front-end technologies combined with back-end data processing. When a user interacts with the chart, it immediately responds and displays relevant data; this process is typically implemented through event-driven programming.

[0087] (3.5) Combination of chart generation and data optimization

[0088] Throughout the chart generation process, the BERT model plays a crucial role, not only enhancing the intelligence of chart generation but also helping to optimize chart selection and style adjustments. Through its understanding of context, BERT can more closely integrate data with charts and optimize chart display, making charts not merely graphical representations of data but intelligent tools that better meet user needs.

[0089] The combination of chart generation and data optimization is described by the following mathematical model:

[0090]

[0091] Among them, f v (y,data) represents the degree of match between the chart's visualization and the data, f s (y, X) represents the degree of matching between the chart and the user's query intent. Through this combination, the system can generate charts that are both beautiful and accurate, thereby improving the user experience.

[0092] (4) Linking charts and documents

[0093] After the chart is generated, the system also needs to link the chart with the document content, so that users can obtain relevant document or report information when viewing the chart, so as to gain a deeper understanding of the background and business logic of the data.

[0094] (4.1) Chart click event handling

[0095] Users can trigger the display of document content related to a specific data point or area by clicking on it in the chart.

[0096] (4.2) BERT-based document content retrieval

[0097] The key to linking charts and documents lies in document retrieval. The BERT model helps the system understand the semantic relationship between chart data and document content. When a user clicks on a data point in a chart, the system uses BERT to perform semantic matching, comparing that data point with related paragraphs or information in the document.

[0098] (4.3) Dynamic document display

[0099] Once document content related to the chart data is identified, the system will display the corresponding document snippets to the user through the document linking module. Document display can be achieved by highlighting relevant paragraphs or directly popping up a document view. Users can then further read and analyze the background information related to the data, improving the effectiveness of decision support.

[0100] (5) System Architecture

[0101] The system described in this invention includes a user input module, a semantic parsing module, a data matching and extraction module, a chart generation module, a chart visualization and layout module, and a document linkage module.

[0102] User input module: Obtains the user's query intent through text or voice input.

[0103] Semantic parsing module: Based on a preset semantic model, it parses the natural language commands input by the user and identifies relevant entities and query requirements in the power dispatching field.

[0104] Data matching and extraction module: Combining multi-source heterogeneous data stored in the knowledge graph, it automatically matches the corresponding data source by parsing entity relationships and extracts key indicators.

[0105] Chart generation module: By generating adversarial networks to optimize the selection of chart types, it generates interactive dynamic charts in different forms.

[0106] Chart visualization and arrangement module: Provides a drag-and-drop visualization interface, through which users can adjust chart elements and update chart data logic and display effects in real time.

[0107] Document Linkage Module: Supports linking charts with document content. When a user clicks on a section of a chart, the system can automatically locate and display document snippets related to that data, allowing the user to view detailed background information.

[0108] The system uses a deep neural network model to parse the natural language input from users, identifying key entities and relationships in the query. Then, based on predefined rules and structured data sources, it employs an automated data filtering and matching algorithm based on the user query to select a suitable chart type and generate the chart via an automated script. Furthermore, after the chart is generated, users can adjust chart elements using interactive functions on the chart interface.

Claims

1. A multimodal intent-driven intelligent chart dynamic generation system, characterized in that, It includes a user input module, a semantic parsing module, a data matching and extraction module, a chart generation module, a chart visualization and layout module, and a document linkage module; The user input module: obtains the user's query intent through text or voice input; The semantic parsing module: based on a preset semantic model, it parses the natural language commands input by the user and identifies relevant entities and query requirements in the power dispatching field; The data matching and extraction module combines multi-source heterogeneous data stored in the knowledge graph, automatically matches the corresponding data source by parsing entity relationships, and extracts key indicators. The chart generation module optimizes the selection of chart types through a generation adversarial network, generating different forms of interactive dynamic charts. The chart visualization and arrangement module provides a drag-and-drop visualization interface through which users can adjust chart elements and update chart data logic and display effects in real time. The document linkage module supports the linkage between charts and document content. When a user clicks on a block in a chart, the system automatically locates and displays document fragments related to that data, making it easier for the user to view detailed background information.

2. A method for dynamically generating intelligent charts driven by multimodal intent, characterized in that, Includes the following steps: (1) Users input their query requirements via text or voice. After receiving the input, the system performs semantic parsing on the input natural language instructions. (2) In the data matching and extraction stage, the system extracts entity-related data from the knowledge graph and related data sources in the field of power dispatching; (3) The system automatically selects the appropriate chart type based on the extracted data and generates interactive charts; combined with BERT’s powerful contextual understanding capabilities, chart generation not only relies on traditional rules and templates, but can also be intelligently optimized based on the actual meaning of the data; (4) After the chart is generated, the system needs to link the chart with the document content so that users can obtain relevant document or report information when viewing the chart, so as to gain a deeper understanding of the background and business logic of the data.

3. The multimodal intent-driven intelligent chart dynamic generation method according to claim 2, characterized in that, Step (1) includes user input text preprocessing, entity recognition, dependency parsing, and output of comprehensive results.

4. The multimodal intent-driven intelligent chart dynamic generation method according to claim 3, characterized in that, The entity recognition includes: BERT uses its self-attention mechanism to generate a context-dependent representation for each word and performs entity classification based on this representation; the segmented text is input into the BERT model, and BERT processes the entire text sequence to generate a representation of each word in the context; The output generated by BERT is a contextual representation of each word: ， The output of the BERT model is further processed through a classification layer to predict the entity label for each word; assuming the output of BERT is ht, entity classification can be performed using Softmax: ， Among them, y t It is word x t The corresponding labels are W, which is the weight of the classification layer, and b is the bias term.

5. The multimodal intent-driven intelligent chart dynamic generation method according to claim 2, characterized in that, Step (2) includes determining the data source, data matching, data extraction and transformation, and data quality control.

6. The multimodal intent-driven intelligent chart dynamic generation method according to claim 2, characterized in that, Step (3) includes chart type selection, chart optimization and style adjustment, dynamic updating of data and chart association, interactive chart generation, and the combination of chart generation and data optimization.

7. The multimodal intent-driven intelligent chart dynamic generation method according to claim 6, characterized in that, The selection of the chart type includes using BERT to perform deep semantic analysis on the input text to understand the relationship between user intent and data, expressed by the following formula: ， in, Let y represent the optimal chart type, y be the candidate set of chart types, and X be the context-sensitive text representation output by BERT. This represents the probability distribution of selecting chart type y given context information.

8. The multimodal intent-driven intelligent chart dynamic generation method according to claim 6, characterized in that, The dynamic updates of the data and charts depend on the chart's data update formula: ， Among them, y new For the updated chart data, y old The old data state is represented by u_c, where u_c represents the user operation, and n_data represents the new data; the system uses the function f u The data is processed to generate an updated chart.

9. The multimodal intent-driven intelligent chart dynamic generation method according to claim 6, characterized in that, The combination of chart generation and data optimization is described by the following mathematical model: ， Among them, f v (y,data) represents the degree of match between the chart's visualization and the data, f s (y, X) represents the degree of match between the chart and the user's query intent.

10. The multimodal intent-driven intelligent chart dynamic generation method according to claim 2, characterized in that, Step (4) includes chart click event processing, BERT-based document content retrieval, and dynamic document display.

Images