Semantic understanding-based multi-modal information visualization content generation method

Abstract

The invention relates to the technical field of semantic processing, and discloses a multi-modal information visualization content generation method and system based on semantic understanding. The method comprises the steps of obtaining a natural language instruction, structured data and context information; respectively carrying out semantic analysis, data annotation and context coding; generating a unified representation through a multi-modal semantic fusion encoder; constructing a visual semantic decision tree, and providing an interactive adjustment interface for a user to dynamically adjust semantic node parameters; and updating the semantic representation in response to the adjustment operation, and driving a rendering engine to generate final visual content. The system comprises a multi-modal input acquisition module, a semantic analysis module, a fusion coding module, a decision tree construction module, an interaction adjustment module, a content generation module and the like. According to the system, interpretability and real-time controllability of a semantic level are realized, and the consistency of a visualization result and a user intention is remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of semantic processing technology, specifically relating to a method for generating multimodal information visualization content based on semantic understanding. Background Technology

[0002] With the widespread application of multimodal data in scenarios such as intelligent interaction, digital content creation, and human-computer collaborative analysis, information visualization technology based on semantic understanding has become a key bridge connecting complex data with user cognition. This technology aims to fuse multiple modalities of information, including text, images, and audio, and then intuitively present their inherent semantic structure through graphical means, thereby improving users' efficiency in understanding high-dimensional data and their decision-making capabilities. In this process, the semantic understanding module is responsible for parsing the deeper meaning of the original input, while the visualization engine generates visual expressions that conform to cognitive logic based on the parsing results. Together, they constitute the basic architecture of multimodal information visualization.

[0003] Multimodal information visualization content generation methods based on semantic understanding focus on mapping natural language instructions or contextual semantics into executable visual element configurations. The core of this approach lies in establishing a precise mapping relationship from abstract semantics to specific visual attributes (such as layout, color, shape, and dynamic effects). Ideally, this method should be able to dynamically adjust semantic weights according to user intent, ensuring that the generated visualization content is both faithful to the essence of the data and aligns with the user's personalized cognitive preferences or task objectives.

[0004] Existing technologies typically employ pre-trained end-to-end models to achieve semantic-to-visual conversion, with semantic parsing and visualization generation processes highly coupled and parameters fixed. While such methods can achieve good results on standard test sets, they lack real-time intervention mechanisms for semantic parameters in real-world interactive scenarios: users cannot dynamically adjust the importance of keywords, the strength of association between concepts, or the emphasis ratio of modal fusion through intuitive operations (such as gestures, dragging, or voice commands) during the visualization generation process.

[0005] This deficiency means that if the generated result deviates from the user's expectations, the complete command must be re-entered or the underlying code adjusted, which greatly limits the system's interactive flexibility and efficiency. Especially in applications that emphasize immediate feedback and iterative optimization, such as exploratory data analysis, creative design assistance, or educational demonstrations, the static semantic mapping mechanism is unable to support the need for fine-grained semantic tuning, forming a critical breakpoint in the "understanding-expression-feedback" closed loop. Summary of the Invention

[0006] This invention provides a method for generating multimodal information visualization content based on semantic understanding, aiming to solve the technical problem that users cannot fine-tune semantic parameters in real time during the visualization generation process, resulting in deviations between the generated content and actual needs. This method constructs a semantically adjustable multimodal fusion representation space, uniformly encoding natural language instructions, structured data, and contextual information, and introduces an interactive adjustment mechanism with controllable semantic granularity. This allows users to precisely intervene in key semantic dimensions at different stages of visualization generation, thereby generating visualization content that highly matches the user's intent.

[0007] This invention provides a method for generating multimodal information visualization content based on semantic understanding, comprising: Acquire the user's natural language input commands, the structured dataset to be visualized, and contextual information; The natural language input command is semantically parsed to extract the topic keywords, intent categories, focus of attention and modifiers to form an initial set of semantic elements. Field identification and semantic annotation are performed on the structured dataset to establish a mapping relationship between field names, data types, numerical distribution features and semantic roles, and to generate data semantic description vectors; The context environment information is coded with context awareness, including user history interaction records, current task scene identifier, device display capability parameters, and time or geographical location context, to generate a context semantic embedding vector. The initial set of semantic elements, the data semantic description vector, and the context semantic embedding vector are input into the multimodal semantic fusion encoder. Feature alignment and semantic coupling are performed through a cross-modal attention mechanism to generate a unified multimodal semantic fusion representation. Based on the multimodal semantic fusion representation, a visual semantic decision tree is constructed. The visual semantic decision tree contains multiple levels of semantic nodes, and each semantic node corresponds to a visual design decision, including chart type selection, coordinate axis mapping, color scheme configuration, label density control, and interactive animation strategy. Provide users with an interactive adjustment interface for a visual semantic decision tree. The interactive adjustment interface presents each semantic node and its current value in a graphical manner, and allows users to directly modify the parameter value or adjust the weight of any semantic node. In response to the user's adjustment operation, the corresponding semantic components in the multimodal semantic fusion representation are updated, and the subsequent visualization generation process is re-executed; Based on the updated multimodal semantic fusion representation, the visualization rendering engine is invoked to generate the final visualization content, which is then output to the user's terminal device for display.

[0008] As one embodiment of the present invention, the semantic parsing of natural language input instructions specifically includes: using a pre-trained language model to segment the input instructions, identifying noun phrases as topic keywords, verb phrases as intent categories, and adjectives or adverbs as indicators of focus intensity, and determining the logical relationships between each semantic element through dependency parsing to form a set of structured semantic element triples.

[0009] As one embodiment of the present invention, the field identification and semantic annotation of the structured dataset specifically includes: traversing all fields of the dataset, matching the character pattern of the field name with a predefined semantic dictionary to identify geographic information fields, time fields, categorical fields, or numerical fields; calculating the mean, variance, skewness, and kurtosis of the numerical fields, and counting the number of unique values ​​and distribution entropy of the categorical fields, and encoding the above statistical features and field types together into a fixed-dimensional data semantic description vector.

[0010] As one embodiment of the present invention, the user history interaction records in the context environment information include the types of visualization content generated by the user in the past thirty days, commonly used chart configuration parameters, and the adjustment frequency of specific semantic dimensions; the current task scenario identifier is automatically inferred by the system based on the application or workflow stage currently opened by the user; the device display capability parameters include screen resolution, color depth, and whether touch interaction is supported; the time or geographical location context is provided by the system clock or positioning module.

[0011] As one embodiment of the present invention, the multimodal semantic fusion encoder adopts a three-layer stacked cross-modal Transformer structure, each layer containing a self-attention sublayer and a cross-modal cross-attention sublayer; in the self-attention sublayer, semantic units within each modality are associated and modeled; in the cross-modal cross-attention sublayer, natural language semantic units are used as query vectors to perform key-value retrieval on data semantic description vectors and context semantic embedding vectors respectively, thereby achieving semantic alignment; the final output fusion representation is a weighted aggregation result of all modal semantic units.

[0012] As one embodiment of the present invention, the construction process of the visual semantic decision tree includes: firstly, determining the candidate set of chart types for the root node based on the intent category in the multimodal semantic fusion representation; then, generating child nodes for each candidate chart type based on the field types and distribution characteristics in the data semantic description vector, with each child node corresponding to a coordinate axis mapping scheme; next, generating leaf nodes for color schemes, label densities, and animation strategies based on the focus intensity indicator and the device capability parameters in the context semantic embedding vector; each node is associated with an adjustable semantic weight parameter, the initial value of which is determined by the corresponding component in the fusion representation.

[0013] As one embodiment of the present invention, the interactive adjustment interface displays a visual semantic decision tree in the form of a tree diagram. Each node is represented by a rectangle, which displays the current parameter value and adjustment slider. The user activates the adjustment panel by clicking on a node. The panel provides a list of discrete options or a continuous numerical slider. After the user makes a modification, the system immediately calculates the new semantic components and triggers a local re-rendering, updating only the affected visual elements rather than rebuilding the entire system.

[0014] As one embodiment of the present invention, the updating of the corresponding semantic components in the multimodal semantic fusion representation specifically includes: mapping the user-adjusted parameter values ​​back to the original semantic space to generate a corrected semantic vector; performing weighted fusion of the corrected vector with the vector at the corresponding position in the original fusion representation, the weight being determined by the adjustment magnitude and the historical adjustment confidence; and sending the fused representation as a new input to the parameter configuration module of the visualization rendering engine.

[0015] As one embodiment of the present invention, the visualization rendering engine sequentially performs chart type instantiation, data binding, visual channel mapping, layout optimization, and interactive event registration based on the updated multimodal semantic fusion representation, generating scalable vector graphics or bitmap format visualization content that conforms to Web standards, and adjusting the output size and interaction mode through the device adaptation layer to match the characteristics of the terminal device.

[0016] This invention provides a multimodal information visualization content generation system based on semantic understanding, comprising: The multimodal input acquisition module is used to acquire the user's natural language input commands, the structured dataset to be visualized, and contextual information. The natural language semantic parsing module is used to perform semantic parsing on the natural language input instructions, extract topic keywords, intent categories, focus of attention and modifiers, and form an initial set of semantic elements. The data semantic annotation module is used to perform field identification and semantic annotation on the structured dataset, establish the mapping relationship between field name, data type, numerical distribution characteristics and semantic role, and generate data semantic description vector; The context-aware encoding module is used to perform context-aware encoding on the context environment information to generate a context semantic embedding vector; The multimodal semantic fusion encoding module is used to input the initial set of semantic elements, the data semantic description vector, and the context semantic embedding vector into the multimodal semantic fusion encoder, and perform feature alignment and semantic coupling through a cross-modal attention mechanism to generate a unified multimodal semantic fusion representation. The visual semantic decision tree construction module is used to construct a visual semantic decision tree containing multiple levels of semantic nodes based on the multimodal semantic fusion representation, with each semantic node corresponding to a visual design decision; The interactive adjustment interface module provides users with an interactive adjustment interface for visualizing the semantic decision tree, allowing users to directly modify the parameter values ​​or adjust the weights of any semantic node. The semantic representation dynamic update module is used to update the corresponding semantic components in the multimodal semantic fusion representation in response to the user's adjustment operation; The visualization content generation module is used to call the visualization rendering engine based on the updated multimodal semantic fusion representation, generate the final visualization content, and output it to the user's terminal device.

[0017] Compared with the prior art, the advantages and positive effects of the present invention are as follows: 1. This invention solves the problem of semantic expression being disconnected from user intent in traditional visualization generation methods by constructing a unified multimodal semantic fusion representation space, which deeply integrates natural language instructions, structured data and contextual environment.

[0018] 2. A visual semantic decision tree structure is introduced, breaking down the complex visualization design process into interpretable and adjustable semantic nodes. This allows users to precisely intervene in key dimensions such as chart type, data mapping, and visual style during the generation process. The interactive adjustment interface supports real-time parameter modification and partial re-rendering, significantly improving users' control and satisfaction with the generated results.

[0019] 3. This invention avoids the shortcomings of traditional methods that rely on fixed templates or black box models, resulting in rigid generated content and difficulty in adapting to diverse needs. It achieves dynamic adjustability at the semantic level and highly personalized generated results, effectively narrowing the semantic gap between user intent and visual output. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the overall technical architecture of the multimodal information visualization content generation method based on semantic understanding proposed in this invention; Figure 2 This is a schematic diagram of the core principle framework of the multimodal semantic fusion encoder in this invention; Figure 3 This is a flowchart illustrating the logical process of multimodal input processing and semantic representation construction of natural language instructions, structured data, and contextual information in this invention. Figure 4 This is a flowchart illustrating the logical flow of constructing a visual semantic decision tree and expanding hierarchical semantic nodes in this invention. Figure 5This is a schematic diagram of the multi-level interaction relationship and data flow between the interactive adjustment interface and the dynamic update mechanism of semantic components in this invention; Figure 6 This is a logical flowchart of the visualization content generation and terminal adaptation output based on the updated multimodal semantic fusion representation in this invention. Detailed Implementation

[0021] Please refer to Figures 1 to 6 This invention provides a method for generating multimodal information visualization content based on semantic understanding, aiming to solve the technical problem that users cannot fine-tune semantic parameters in real time during the visualization generation process, resulting in deviations between the generated content and actual needs. This method constructs a semantically adjustable multimodal fusion representation space, uniformly encodes natural language instructions, structured data, and contextual information, and introduces an interactive adjustment mechanism with controllable semantic granularity. This allows users to precisely intervene in key semantic dimensions at different stages of visualization generation, thereby generating visualization content that highly matches the user's intent.

[0022] The method includes the following steps: S1, obtain the user's natural language input commands, the structured dataset to be visualized, and contextual information; S2, perform semantic parsing on the natural language input command, extract the topic keywords, intent categories, focus of attention and modifiers to form an initial set of semantic elements; S3, perform field identification and semantic annotation on the structured dataset, establish the mapping relationship between field name, data type, numerical distribution characteristics and semantic role, and generate data semantic description vector; S4, perform context-aware encoding on the context environment information, which includes user history interaction records, current task scene identifier, device display capability parameters and time or geographical location context, and generate a context semantic embedding vector. S5, the initial semantic element set, the data semantic description vector, and the context semantic embedding vector are input into the multimodal semantic fusion encoder, and feature alignment and semantic coupling are performed through a cross-modal attention mechanism to generate a unified multimodal semantic fusion representation; S6. Based on the multimodal semantic fusion representation, a visual semantic decision tree is constructed. The visual semantic decision tree contains multiple levels of semantic nodes. Each semantic node corresponds to a visual design decision, including chart type selection, coordinate axis mapping, color scheme configuration, label density control, and interactive animation strategy. S7, Provide the user with an interactive adjustment interface for a visual semantic decision tree. The interactive adjustment interface presents each semantic node and its current value in a graphical manner, and allows the user to directly modify the parameter value or adjust the weight of any semantic node. S8, in response to the user's adjustment operation, update the corresponding semantic components in the multimodal semantic fusion representation, and re-execute the subsequent visualization generation process; S9, based on the updated multimodal semantic fusion representation, calls the visualization rendering engine to generate the final visualization content and outputs it to the user's terminal device for display.

[0023] In step S1, the system synchronously collects three types of information sources through a multimodal input interface: the user's natural language input commands, the structured dataset to be visualized, and contextual information. Natural language input commands are submitted by the user through text input boxes, speech-to-text modules, or template selectors. The content consists of complete sentences describing the desired visualization effect, such as "Show a bar chart of sales figures for the past three months distributed by region, highlighting the East China region." The structured dataset is provided in tabular form, containing row records and column fields. Field types include numeric, categorical, temporal, and geographic.

[0024] Contextual information is automatically collected by the system, including the types of visualization content generated by the user in the past thirty days, commonly used chart configuration parameters, and the frequency of adjustments to specific semantic dimensions; the current task scenario identifier is automatically inferred by the system based on the user's currently open application or workflow stage; device display capability parameters include screen resolution, color depth, and whether touch interaction is supported; time or geographic location context is provided by the system clock or positioning module. All input information is assigned a unified timestamp and stored in a temporary cache before entering subsequent processing to ensure data synchronization and processing consistency.

[0025] In step S2, the natural language input command is semantically parsed. This process uses a pre-trained language model to segment the input command, identifying noun phrases as topic keywords, verb phrases as intent categories, and adjectives or adverbs as indicators of focus intensity. Dependency parsing is then used to determine the logical relationships between semantic elements, forming a structured set of semantic element triples. Specifically, the input command is first fed into a semantic parser based on a bidirectional encoder, which outputs the part-of-speech tag, named entity type, and syntactic dependency arc for each lexical unit. Subsequently, the system traverses the dependency tree, extracting the core components of the subject-verb-object structure: the subject is labeled as a topic keyword, the verb is categorized as an intent category (e.g., "show," "compare," "track"), and the modifiers in the object and complement are parsed as focus of attention (e.g., "highlight," "emphasize," "ignore") and their intensity levels (e.g., "significant," "slight"). All extracted semantic elements are organized into an initial set of semantic elements in key-value pair form, where the key is the semantic role type and the value is the corresponding lexical sequence and its confidence score. This set serves as the natural language semantic input for subsequent multimodal fusion.

[0026] In step S3, field identification and semantic annotation are performed on the structured dataset. The system traverses all fields of the dataset, matching the character patterns of the field names against a predefined semantic dictionary to identify geographic information fields, time fields, categorical fields, or numerical fields. For numerical fields, their mean, variance, skewness, and kurtosis are calculated; for categorical fields, the number of unique values ​​and distribution entropy are counted. These statistical features and field types are jointly encoded into a fixed-dimensional data semantic description vector. In specific implementation, field names are first matched using regular expressions against geographic keywords (such as "province", "city", "latitude and longitude"), time keywords (such as "date", "month", "quarter"), and general category identifiers (such as "ID", "category"). If a match is successful, the field type is explicitly labeled; otherwise, the system samples the first 100 rows of data and analyzes their value distribution: if all values ​​are numbers and continuously changing, it is determined to be a numerical field; if the values ​​are discrete strings and the number of unique values ​​is less than 10% of the total number of records, it is determined to be a categorical field. Subsequently, for each field type, corresponding statistics are extracted: numerical fields are output as four-dimensional statistical vectors, categorical fields are output as two-dimensional features (number of unique values, distribution entropy), time fields are converted into timestamp sequences and their start and end ranges and time intervals are calculated, and geographic fields are converted into standard latitude and longitude formats through geocoding services. The semantic description vectors of all fields are concatenated into a matrix form as the input representation of the data modality.

[0027] In step S4, context-aware encoding is performed on the contextual information. User historical interaction records are aggregated into behavioral statistical vectors, including the frequency percentage of chart types such as bar charts, line charts, and scatter plots generated in the past thirty days, the preference index of commonly used color schemes, and the average adjustment magnitude of semantic dimensions such as coordinate axes and label density. The current task scenario identifier is mapped into a one-hot encoded vector, with scenario categories including "business analysis," "scientific research report," and "teaching demonstration." Device display capability parameters are quantified into numerical vectors: screen resolution is converted into pixel width and height, color depth is represented by bits, and touch support is recorded using binary flags. Time or geographic location context is standardized: time is converted into the hours of the day and the day of the week, and geographic location is encoded into a fixed-length string using a geohashing algorithm. After normalization, all contextual sub-items are mapped to a unified-dimensional contextual semantic embedding vector through a fully connected layer. This vector retains the semantic information of the original context and adapts to the input requirements of the subsequent fusion encoder.

[0028] In step S5, the initial set of semantic elements, the data semantic description vector, and the context semantic embedding vector are input into the multimodal semantic fusion encoder. This encoder employs a three-layer stacked cross-modal Transformer structure, with each layer containing a self-attention sublayer and a cross-modal cross-attention sublayer. In the self-attention sublayer, semantic units within each modality are modeled for association; in the cross-modal cross-attention sublayer, using natural language semantic units as query vectors, key-value retrieval is performed on the data semantic description vector and the context semantic embedding vector to achieve semantic alignment. The final output fusion representation is a weighted aggregation result of all modal semantic units. Specifically, natural language semantic elements are converted into query sequences. Data semantic description vectors constitute key sequences AND value sequence The context semantic embedding vectors constitute the key sequence. AND value sequence Cross-attention is calculated as follows:

[0029] in, Let be the dimension of the key vector. The system calculates them separately. and The results are then concatenated and fed forward to generate a fused semantic representation. The three-layer stacked structure deepens the cross-modal association layer by layer, and the final output multimodal semantic fusion representation is a high-dimensional dense vector, with each dimension corresponding to a potential semantic concept, such as "emphasizing regional contrast", "time trend priority" or "simplified layout on mobile devices".

[0030] In step S6, a visualization semantic decision tree is constructed based on the multimodal semantic fusion representation. This decision tree contains multiple levels of semantic nodes, each corresponding to a visualization design decision. The construction process first determines the candidate set of chart types for the root node based on the intent category in the fusion representation; then, based on the field types and distribution characteristics in the data semantic description vector, child nodes are generated for each candidate chart type, with each child node corresponding to a coordinate axis mapping scheme; next, leaf nodes for color schemes, label densities, and animation strategies are generated based on the focus intensity indicator and the device capability parameters in the context semantic embedding vector.

[0031] Each node is associated with an adjustable semantic weight parameter, the initial value of which is determined by the corresponding component in the fused representation. For example, if the intent category is "comparison," the candidate chart type includes bar charts, column charts, and radar charts; if the data contains two numerical fields and one categorical field, the X-axis of the bar chart is mapped to the categorical field, and the Y-axis is mapped to the difference or ratio of the two numerical fields; if the focus intensity is high and the device is a high-resolution screen, a high-contrast color scheme is enabled, the label density is set to high, and a smooth transition effect is enabled for the animation strategy. All decision paths are organized into a tree structure, with nodes linked through parent-child relationships, forming a complete visualization design space.

[0032] In step S7, an interactive adjustment interface for a visual semantic decision tree is provided to the user. This interface displays the decision tree in a tree diagram format, with each node represented by a rectangle containing the current parameter value and an adjustment slider. The user activates the adjustment panel by clicking on a node, which provides a list of discrete options or a continuous numerical slider. After the user makes a change, the system immediately calculates the new semantic components and triggers a local re-render, updating only the affected visual elements rather than rebuilding the entire system. The adjustment interface supports multiple interaction methods, including dragging, sliding, and dropdown selection, and all operations are fed back to the semantic representation layer in real time. The interface layout dynamically adjusts according to the device's display capabilities: on touch devices, the node spacing is increased to accommodate finger operation; on low-resolution screens, non-critical nodes are collapsed by default, and only the branches most recently adjusted by the user are expanded.

[0033] In step S8, in response to the user's adjustment operation, the corresponding semantic components in the multimodal semantic fusion representation are updated. The system maps the user-adjusted parameter values ​​back to the original semantic space, generating a corrected semantic vector. This corrected vector is then weighted and fused with the vector at the corresponding position in the original fusion representation, with the weight determined by both the adjustment magnitude and the historical adjustment confidence. The fused representation is then sent as a new input to the parameter configuration module of the visualization rendering engine. The larger the adjustment magnitude, the higher the weight of the corrected vector; if the user has adjusted the same semantic dimension multiple times in the past, the historical confidence is increased, enhancing the impact of the current adjustment. This mechanism ensures that the user's intent is accurately captured while avoiding drastic fluctuations in the generated results due to accidental misoperations.

[0034] In step S9, based on the updated multimodal semantic fusion representation, the visualization rendering engine is invoked to generate the final visualization content. The rendering engine sequentially executes chart type instantiation, data binding, visual channel mapping, layout optimization, and interactive event registration. Chart type instantiation selects the corresponding graphic template based on the root node of the decision tree; data binding maps structured data fields to visual channels such as axes, colors, and sizes; visual channel mapping configures the rendering style based on color schemes and label density parameters; layout optimization adjusts the legend position, title font size, and margins according to the screen size; interactive event registration binds user operations such as zooming, filtering, and tooltips. The final generated content is a scalable vector graphics or bitmap format conforming to Web standards, and the output size and interaction mode are adjusted through a device adaptation layer to match the characteristics of the terminal device. For example, on mobile devices, complex animations are disabled and touch gestures are enabled; on high color depth displays, a wide color gamut color scheme is enabled.

[0035] The system comprises a multimodal input acquisition module, a natural language semantic parsing module, a data semantic annotation module, a context-aware encoding module, a multimodal semantic fusion encoding module, a visual semantic decision tree construction module, an interactive adjustment interface provision module, a semantic representation dynamic update module, and a visual content generation module. The multimodal input acquisition module is responsible for synchronously collecting natural language instructions, structured datasets, and contextual information, and performing timestamp alignment and format standardization. The natural language semantic parsing module deploys a pre-trained language model, performs word segmentation, named entity recognition, dependency parsing, and semantic role annotation, and outputs a set of structured semantic elements. The data semantic annotation module automatically identifies field types, extracts statistical features, and encodes semantic vectors to generate a data semantic description matrix. The context-aware encoding module aggregates historical behavior, task scenarios, device parameters, and spatiotemporal context, and generates contextual semantic embedding vectors through a fully connected network.

[0036] The multimodal semantic fusion encoding module adopts a three-layer cross-modal Transformer architecture, performing self-attention and cross-attention calculations to output a unified multimodal semantic fusion representation. The visualization semantic decision tree construction module analyzes the chart type, data mapping, and visual style decisions based on the fusion representation, constructing a hierarchical semantic node tree. The interactive adjustment interface provides a module-rendered tree-like adjustment interface, supporting node clicking, parameter sliding, and real-time feedback. The semantic representation dynamic update module receives user adjustment commands, performs semantic vector correction and weighted fusion, and updates the fusion representation. The visualization content generation module calls the rendering engine to complete chart instantiation, data binding, visual mapping, layout optimization, and device adaptation, outputting the final visualization content to the user's terminal.

[0037] This embodiment achieves end-to-end generation from natural language instructions to highly personalized visual content through the above-described method and system. Users can precisely intervene in any semantic dimension during the generation process, effectively narrowing the semantic gap between user intent and visual output.

Claims

1. A method for generating multimodal information visualization content based on semantic understanding, characterized in that, include: Acquire the user's natural language input commands, the structured dataset to be visualized, and contextual information; The natural language input command is semantically parsed to extract the topic keywords, intent categories, focus of attention and modifiers to form an initial set of semantic elements. Field identification and semantic annotation are performed on the structured dataset to establish a mapping relationship between field names, data types, numerical distribution features and semantic roles, and to generate data semantic description vectors; The context environment information is coded with context awareness, including user history interaction records, current task scene identifier, device display capability parameters, and time or geographical location context, to generate a context semantic embedding vector. The initial set of semantic elements, the data semantic description vector, and the context semantic embedding vector are input into the multimodal semantic fusion encoder. Feature alignment and semantic coupling are performed through a cross-modal attention mechanism to generate a unified multimodal semantic fusion representation. Based on the multimodal semantic fusion representation, a visual semantic decision tree is constructed; Provide users with an interactive adjustment interface for a visual semantic decision tree. The interactive adjustment interface presents each semantic node and its current value in a graphical manner, and allows users to directly modify the parameter value or adjust the weight of any semantic node. In response to the user's adjustment operation, the corresponding semantic components in the multimodal semantic fusion representation are updated, and the subsequent visualization generation process is re-executed; Based on the updated multimodal semantic fusion representation, the visualization rendering engine is invoked to generate the final visualization content, which is then output to the user's terminal device for display.

2. The method for generating multimodal information visualization content based on semantic understanding according to claim 1, characterized in that, The visualization semantic decision tree contains multiple levels of semantic nodes, each semantic node corresponding to a visualization design decision, including chart type selection, coordinate axis mapping, color scheme configuration, label density control, and interactive animation strategy.

3. The method for generating multimodal information visualization content based on semantic understanding according to claim 2, characterized in that, Semantic parsing of the natural language input instructions includes: The input instructions are segmented using a pre-trained language model, which identifies noun phrases as topic keywords, verb phrases as intent categories, and adjectives or adverbs as indicators of focus intensity. Dependency parsing is used to determine the logical relationships between semantic elements, forming a set of structured semantic element triples.

4. The method for generating multimodal information visualization content based on semantic understanding according to claim 3, characterized in that, Field identification and semantic annotation are performed on the structured dataset, including: Iterate through all fields of the dataset and match them against a predefined semantic dictionary based on the character pattern of the field name to identify geographic information fields, time fields, categorical fields, or numerical fields. Calculate the mean, variance, skewness, and kurtosis for numerical fields, and count the number of unique values ​​and the distribution entropy for categorical fields; The above statistical features and field types are jointly encoded into a fixed-dimensional data semantic description vector.

5. The method for generating multimodal information visualization content based on semantic understanding according to claim 4, characterized in that, Context-aware encoding of the aforementioned context information includes: The user's historical interaction records are aggregated into a behavioral statistics vector, which includes the frequency ratio of the types of visual content generated in the past thirty days, the preference index of commonly used chart configuration parameters, and the average adjustment magnitude of specific semantic dimensions. Map the current task scenario identifier to a one-hot encoded vector; The device's display capability parameters are quantified into numerical vectors, including screen resolution, color depth, and touch support flags; After standardizing the time or geographic location context, convert it into a fixed-length string using a geohashing algorithm or time period encoding; All context items are normalized and then mapped to a context semantic embedding vector of uniform dimension through a fully connected layer.

6. The method for generating multimodal information visualization content based on semantic understanding according to claim 5, characterized in that, The multimodal semantic fusion encoder adopts a three-layer stacked cross-modal Transformer structure, each layer containing a self-attention sublayer and a cross-modal cross-attention sublayer; In the self-attention sub-layer, semantic units within each modality are modeled for association; In the cross-modal cross-attention sublayer, key-value retrieval is performed on the data semantic description vector and the context semantic embedding vector, respectively, using natural language semantic units as query vectors. The final output fusion representation is a weighted aggregation result of all modal semantic units.

7. The method for generating multimodal information visualization content based on semantic understanding according to claim 6, characterized in that, Based on the multimodal semantic fusion representation, a visual semantic decision tree is constructed, including: The candidate set of graph types for the root node is determined based on the intent category in the multimodal semantic fusion representation; Based on the field types and distribution characteristics in the data semantic description vector, a child node is generated for each candidate chart type, and the child node corresponds to the coordinate axis mapping scheme. Based on the focus intensity indicator and the device capability parameters in the context semantic embedding vector, leaf nodes are generated for color scheme, label density, and animation strategy; Each node is associated with an adjustable semantic weight parameter, the initial value of which is determined by the corresponding component in the fused representation.

8. The method for generating multimodal information visualization content based on semantic understanding according to claim 7, characterized in that, Provides users with an interactive adjustment interface that visualizes a semantic decision tree, including: The visual semantic decision tree is presented in the form of a tree diagram, with each node represented by a rectangle, and the current parameter value and adjustment slider are displayed inside the rectangle. Users can activate the adjustment panel by clicking on a node. The panel provides a list of discrete options or a slider for continuous values. After the user makes a change, the system immediately calculates the new semantic components and triggers a local re-render, updating only the affected visual elements.

9. The method for generating multimodal information visualization content based on semantic understanding according to claim 8, characterized in that, In response to the user's adjustment operation, update the corresponding semantic components in the multimodal semantic fusion representation, including: The user-adjusted parameter values ​​are mapped back to the original semantic space to generate the corrected semantic vector. The corrected vector is then weighted and fused with the corresponding vector in the original fused representation. The weights are determined by the adjustment magnitude and the historical adjustment confidence. The fused representation is then fed into the parameter configuration module of the visualization rendering engine as a new input.

10. The method for generating multimodal information visualization content based on semantic understanding according to claim 9, characterized in that, Based on the updated multimodal semantic fusion representation, the visualization rendering engine is invoked to generate the final visualization content, including: The process involves sequentially instantiating the chart type, binding data, mapping visual channels, optimizing the layout, and registering interactive events. Generate scalable vector graphics or bitmap-formatted visualizations that conform to Web standards; The output size and interaction mode are adjusted through a device adaptation layer to match the characteristics of the terminal device.

Images