A multi-level context enhanced temporal visual annotation method

Abstract

The application discloses a multi-level context-enhanced time series visualization annotation method, belonging to the technical field of data analysis and visualization. First, seven typical time patterns (rising, falling, single peak, single valley, double peak, double valley, and periodic oscillation) in time series data are automatically identified by prompting LLM technology as annotation objects. Then, an annotation system containing a three-level semantic space is constructed. The first level is based on the data information directly displayed in the visualization chart, the second level is based on the calculation results derived from the displayed data in the chart, and the third level is based on the context pattern information associated with the current chart analyzed attribute in the source data set. The application expands the annotation focus to a deeper and broader semantic space, improves the understanding and exploration ability of users on time series data, provides an structured and content-rich knowledge framework for analysts, and effectively makes up for the shortcomings of traditional annotation methods.

Description

Technical Field

[0001] This invention relates to a multi-level context-enhanced temporal visualization annotation method. Background Technology

[0002] With the advancement of the information age and the generation of massive amounts of time series data, the analysis of time series data has become crucial. However, analyzing time series data is not easy, especially when analyzing high-dimensional, multi-level data, which presents numerous challenges. Visual annotations can help present more intuitive information from complex charts, facilitating analysts to obtain deeper insights.

[0003] While traditional visualization annotation techniques offer some convenience for understanding and exploring time-series data, most existing annotation tools focus on describing the obvious basic facts in time-series visualizations (e.g., trends of increase or decrease) without providing the deeper and more comprehensive contextual information users need. Beyond the visual information directly presented in time-series visualizations, the potential external contextual information is rich and diverse. For example, the data presented in existing time-series visualizations can be further processed to form derived insights (e.g., statistics and forecasts); the data dimensions displayed in the charts can form correlation patterns with other unanalyzed attributes in the source dataset (the dataset from which the charts originate); and there is higher-level background information from knowledge documents or human insights. Comprehensively utilizing contextual information from different sources and forms can enhance the information content of annotations and help users form more structured knowledge. However, research and practice in this area are still lacking. Therefore, how to effectively integrate richer contextual information into annotations remains a challenge. Summary of the Invention

[0004] To address the shortcomings of current time-series data visualization annotation, this invention proposes a multi-level context-enhanced time-series visualization annotation method to provide more comprehensive and richer contextual information.

[0005] The technical solution of this invention is as follows:

[0006] A multi-level context-enhanced time series visualization annotation method is provided, wherein an annotated event is represented in the form of <annotation object, annotation content>, where the annotation object refers to one of the seven typical time patterns in time series visualization, namely rising, falling, single peak, single trough, double peak, double trough, and periodic oscillation, and the annotation content refers to the multi-level context annotation information added to the annotation object. The method includes the following steps:

[0007] (1) Temporal data pattern recognition based on prompt-based large language model LLM: using the powerful reasoning and generation capabilities of LLM, the temporal patterns in temporal data can be identified efficiently and accurately.

[0008] (2) Multi-level context annotation construction: Based on the identified temporal patterns, the following three levels of visualization annotations are automatically constructed: based on observation, based on derived calculations, and based on external data. These three levels correspond to the annotation information originating from the data directly displayed in the chart, the results of further derived calculations based on the data displayed in the chart, and the contextual pattern information in the source dataset that is associated with the attributes analyzed in the current chart.

[0009] Furthermore, step (1) includes the following sub-steps:

[0010] (1-1) Time series data normalization: Z-score normalization method is used to apply to the original time series data to eliminate the influence caused by scale differences between time series data, so that the data are in the same numerical range, which is beneficial to the subsequent LLM processing and the accuracy of pattern recognition.

[0011] (1-2) Construct effective prompts to guide the LLM to understand the task and generate the expected output;

[0012] (1-3) Use LLM to analyze the normalized time series data, identify and describe the patterns in the data.

[0013] Furthermore, step (2) includes the following sub-steps:

[0014] (2-1) Observation-based annotation: Observation-based annotation focuses on what is directly presented in the visualization without requiring additional calculations or input. This annotation level includes only one type of task: text annotation. Text annotations provide textual descriptions of the visual patterns observed in the visualization charts. These annotations not only reveal the overall trends visible at the visual level but also provide detailed information that is difficult to observe by visual inspection alone.

[0015] (2-2) Annotations based on derived calculations: These annotations provide deeper insights by performing further derived calculations and analyses on the data in the charts, including both statistical annotations and predictive annotations.

[0016] (2-3) Annotation based on external data: Annotation based on external data aims to expand the information dimension of the current time series visualization by discovering and introducing contextual pattern information in the source dataset that is associated with the attributes analyzed in the current chart. Three external annotation tasks closely related to the time series are designed: details, correlation, and lead-lag.

[0017] In (2-1), the steps for generating the visual pattern text description are as follows:

[0018] (2-1-1) Use an LLM-driven method to generate text annotations. Input the time-series pattern information detected in step (1), the metadata of the visualization chart, and the original time-series dataset into the LLM. The time-series pattern information includes pattern classification labels and time positioning coordinates. The metadata of the visualization chart includes titles and coordinate axis semantic labels. The original time-series dataset includes timestamp-value pair sequences. By carefully crafting prompts, guide the LLM to generate text descriptions of the visual patterns observed in the visualization chart based on the input data.

[0019] (2-1-2) Use different colors to selectively highlight key information in the generated text description. A keyword dictionary is pre-constructed, and then regular expressions are used to check the generated text annotations in turn to identify the keyword information and apply the corresponding colors to highlight it.

[0020] In (2-2), the statistical annotations and predictive annotations are as follows:

[0021] (2-2-1) Statistical annotation: By calculating the presented data, statistical indicators that can quantitatively describe the characteristics of the data are generated. Such annotations can provide accurate statistical attributes that are not easily perceived visually, thereby helping users to understand the intrinsic characteristics of the data more deeply. The structure of a statistical annotation event includes two parts: <statistical scope and statistical method>. The statistical scope defines the range of data to be statistically analyzed, and the statistical method indicates the type of statistical calculation. Then, according to the statistical scope and statistical method, quantitative calculations are performed on the target data, and the calculation results are displayed as annotation information in the chart through auxiliary lines and text labels to help users identify them.

[0022] (2-2-2) Predictive annotation: Predictive annotation aims to infer future time series trends by utilizing existing time series data and predictive models. It uses an AutoARMA model to dynamically estimate future data points, calculates the prediction interval at a 95% confidence level, and sets the prediction range to one-fifth of the current sequence length. The annotation results use dashed curves to represent the prediction trajectory, a strip of semi-transparent area to display the confidence interval, and a time axis marker at the prediction start point.

[0023] In (2-3), the details, relevance, and lead-lag annotations are as follows:

[0024] (2-3-1) Detailed annotations: Detailed annotations are designed to provide more granular information about the patterns observed in time series data. A significant characteristic of time series data is that it typically contains a large number of data points, with a large number of records at each time point. The purpose of detailed annotations is to provide raw time series information before aggregation, so it is only applicable to time series visualizations where aggregation operations are involved.

[0025] The process of generating detailed annotations is as follows: First, the raw data before observation attribute aggregation is obtained from the source dataset; then, an appropriate category field is selected as the label from the source dataset, with the category field having the fewest data items selected by default. Different category fields will determine the granularity of the detailed data presentation; finally, the detailed data is rendered in the corresponding time-series pattern area of ​​the original visualization to provide more detailed data distribution information behind the pattern. Furthermore, to improve the efficiency of detailed data inspection, an interactive detailed inspection mechanism is introduced. Specifically, when a user views a data point, the system will automatically connect other data points in the category to which that point belongs to, highlighting the overall trend changes of that category.

[0026] (2-3-2) Correlation annotation: Correlation annotation reveals the potential correlation between data by comparing the changing trends of different dimensions, which can help users identify synchronous increase / decrease or reverse change patterns within a specific time period.

[0027] Relevance annotations consider the relevance of two perspectives: attribute-oriented and subset-oriented;

[0028] Attribute-oriented association focuses on other attribute fields that are parallel to the current analysis dimension;

[0029] Subset-oriented associations emphasize comparisons under different data slices or filtering conditions;

[0030] The process of generating correlation annotations is as follows: First, for the identified time pattern, the correlation coefficient between the correlation dimension in the source dataset and the target attribute within the time window is calculated. The Pearson correlation coefficient is used to quantify the correlation. Then, the change curve of the correlation attribute is presented, and the degree of correlation is mapped by color, i.e., green represents negative correlation and red represents positive correlation.

[0031] Integrating different correlation attributes into a unified visualization chart presents the challenge of displaying different units of measurement. For subset-based correlation annotations, since these subset attributes share the same unit of measurement as the attribute being analyzed, they can be directly integrated into the corresponding annotation range of the chart. For attribute-based correlation annotations, the correlation attributes often have different units of measurement than the attribute being analyzed. To address this, a remapping strategy is proposed: using the current position of the analyzed attribute in the chart as a baseline, all data points of the correlation attributes are remapped to align with their position. Furthermore, to help users accurately identify the unit of measurement differences between attributes, each correlation attribute is clearly labeled, and a specific numerical value is assigned to each data point.

[0032] (2-3-3) Lead-Lap annotation: Lead-Lap annotation aims to reveal the lead-lag relationship between different sequences. Unlike correlation annotation, which analyzes the numerical correlation between variables within the same time interval, lead-lag annotation focuses on examining the dynamic dependency between attributes across time intervals.

[0033] Lead-lag annotation considers two perspectives: attribute-oriented and subset-oriented. Attribute-oriented lead-lag analysis focuses on the dynamic relationship between different attribute variables; subset-oriented lead-lag analysis emphasizes comparing the temporal differences of the same attribute between different subsets.

[0034] The generation process of lead-lag annotation is as follows: First, time patterns are extracted from the associated attributes in the source dataset, and sequences of the same type as the analysis patterns are selected as candidate sets; then, after normalizing these candidate sequences and the labeled pattern sequences to a 0-1 scale, DTW is used to quantify their similarity. After obtaining the DTW distance, sequences with a distance lower than a preset threshold are selected as lead-lag annotation objects; finally, the selected lead-lag sequences are integrated into the original chart, and corresponding attribute labels and similarity values ​​are labeled.

[0035] Similar to the correlation annotation in (2-3-2), the lead-lag annotation also faces the problem of inconsistent attribute dimensions. In order to facilitate the observation of the lead-lag pattern, this technique adopts a unified remapping strategy for the lead-lag annotation: taking the value range of the annotated pattern sequence on the attribute axis (i.e., the y-axis) as a reference, the data of the associated attribute is remapped to a position aligned with the reference range, while maintaining its original position on the time axis (i.e., the x-axis).

[0036] This invention proposes a multi-level context-enhanced time series visualization annotation method. This method utilizes cue-based LLM (Limited Least Mean Motion) technology to identify seven typical time patterns (rising, falling, single-peak, single-trough, bimodal, bi-trough, and periodic oscillation) from time series visualizations. Furthermore, it designs a three-level annotation space to provide users with richer annotation information for time series visualizations. Specifically, the proposed annotation method provides the following three levels of annotation: information based on data directly displayed in the chart, further calculations or inferences based on the presented data, and contextual pattern information from the source dataset that relates to the attributes analyzed in the current chart. By expanding the focus of annotation and covering a deeper and broader data semantic space, this method improves users' ability to understand and explore time series data.

[0037] The beneficial effects of this invention are as follows: This invention proposes a multi-level context-enhanced time-series visualization annotation method, which enhances users' understanding of multi-dimensional time-series data. This method expands the dimensions of visualization annotation, ensuring that it is not limited to the visual pattern of the chart itself, but also incorporates rich external contextual information. This multi-dimensional time-series visualization annotation method is applicable to various fields, including financial market analysis, scientific research, and market trend forecasting. With the continuous development of data visualization technology, this method is expected to be further optimized, bringing users a more intelligent and efficient data insight experience. Attached Figure Description

[0038] Figure 1 Enhance the temporal visualization annotation framework diagram for multi-level contexts;

[0039] Figure 2 Example image for text annotation;

[0040] Figure 3 Example diagram for derivation calculation annotation;

[0041] Figure 4 Example image annotation for details;

[0042] Figure 5 Example diagram for relevance annotation;

[0043] Figure 6 Example diagram for lead-lag annotation. Detailed Implementation Plan

[0044] The present invention will be further described below with reference to the accompanying drawings;

[0045] Reference Figures 1-5 This invention proposes a multi-level, context-enhanced time-series visualization annotation method to address the shortcomings of current time-series data visualization annotation. This method provides more comprehensive and richer contextual information. An annotated event is represented in the form of <annotation object, annotation content>, where the annotation object refers to one of the seven typical time patterns in time-series visualization (rising, falling, single-peak, single-valley, double-peak, double-valley, periodic oscillation), and the annotation content refers to the multi-level contextual annotation information added to the annotation object. For example... Figure 1 As shown in the temporal visualization annotation framework, the technology of this invention consists of two steps: identifying temporal patterns in temporal sequence data using a prompt-based Large Language Model (LLM) to clarify the annotation objects, and constructing multi-level context annotations based on the annotation objects.

[0046] The time-series visualization annotation method includes the following steps:

[0047] (1) Temporal data pattern recognition based on prompt-based Large Language Model (LLM) aims to leverage the powerful reasoning and generation capabilities of LLM to efficiently and accurately identify temporal patterns in temporal data, including the following sub-steps:

[0048] (1-1) Time series data normalization: This invention uses the Z-score normalization method to apply to the original time series data in order to eliminate the influence caused by scale differences between time series data, so that the data are in the same numerical range, which is beneficial to the subsequent LLM processing and the accuracy of pattern recognition.

[0049] (1-2) Construct effective prompts to guide the LLM to understand the task and generate the expected output;

[0050] (1-3) Use LLM to analyze the normalized time series data, identify and describe the patterns in the data.

[0051] (2) Multi-level context annotation construction: Based on the identified temporal patterns, this invention automatically constructs the following three levels of visual annotations according to the data source used for annotation: observation-based, derived calculation-based, and external data-based. These three levels correspond to the annotation information originating from the data directly displayed in the chart, the results of further derived calculations based on the data displayed in the chart, and the contextual pattern information in the source dataset associated with the attributes analyzed in the current chart, including the following sub-steps:

[0052] (2-1) Observation-Based Annotation: Observation-based annotation focuses on what is directly presented in the visualization, requiring no additional calculations or input. This annotation level includes only one task type: text annotation. Text annotations provide textual descriptions of visual patterns observed in the visualization. These annotations not only reveal overall trends visible at the visual level but also provide detailed information that is difficult to observe through visual inspection alone. The steps for generating textual descriptions of visual patterns are as follows:

[0053] (2-1-1) This invention uses an LLM-driven method (GPT-4) to generate text annotations. Specifically, the method inputs the temporal pattern information (pattern classification labels and time positioning coordinates) detected in step (1), the metadata of the visualization chart (title, axis semantic labels), and the original time-series dataset (timestamp-value pair sequence) into the LLM. It then guides the LLM to generate textual descriptions of the visual patterns observed in the visualization chart based on the input data through carefully crafted prompts.

[0054] (2-1-2) To further improve the readability and information communication efficiency of text annotations, this invention implements a visual optimization measure, namely, selectively highlighting key information in the generated text description using different colors. To achieve this, this invention pre-constructs a keyword dictionary, then uses regular expressions to sequentially examine the generated text annotations, identify the keyword information, and apply corresponding colors for highlighting. For example, orange is used to mark pattern-related information (such as trends of rapid increase or decrease), and red is used to highlight numerical-related information (including time intervals and attribute values).

[0055] Figure 2 An example of adding text annotations to a time-series visualization for analyzing cellular network subscription attributes is provided. Step (1) detects that the time-series pattern of the visualization is an upward pattern, spanning from 1986 to 2018, with attributes involving average cellular network subscriptions and years. By integrating this information into cue words, the powerful understanding and reasoning capabilities of LLM can be used to derive the corresponding textual description of the pattern.

[0056] (2-2) Annotations based on derived calculations: These annotations provide deeper insights by performing further derived calculations and analyses on the data in the charts. They include two forms: statistical annotations and predictive annotations, as follows:

[0057] (2-2-1) Statistical Annotation: This involves generating statistical indicators (mean, median, etc.) that quantitatively describe the characteristics of the presented data through calculations. Such annotations can provide accurate statistical attributes that are not easily perceived visually, helping users gain a deeper understanding of the data's intrinsic features. A statistical annotation event consists of two parts: <statistical scope> and <statistical method>. The statistical scope defines the range of data being statistically analyzed (e.g., GDP figures from 2015 to 2020), while the statistical method indicates the type of statistical calculation (e.g., mean, median, standard deviation, etc.). Then, based on the statistical scope and statistical method, quantitative calculations are performed on the target data (e.g., "The average GDP from 2015 to 2020 is 178.9 billion yuan"). The calculation results are displayed as annotation information in charts using auxiliary lines and text labels to help users identify them. Figure 3 The document provides an example of statistical annotation, which shows that by averaging the average number of cellular subscriptions from 1980 to 2020, the historical average number of cellular subscriptions is 38.79.

[0058] (2-2-2) Predictive Annotation: Predictive annotation aims to infer future time series trends by utilizing existing time series data and predictive models. The key to prediction lies in selecting a suitable predictive model. Currently, the Autoregressive Moving Average (AutoARMA) model is used to dynamically estimate future data points. The rationale for choosing AutoARMA is that it automatically identifies the optimal model order p and q by calculating the autocorrelation function (ACF) and partial autocorrelation function (PACF). This allows the model to adaptively adjust its structure and parameters to adapt to time series data with different characteristics. To improve the reliability of the prediction, a prediction interval is also calculated at a 95% confidence level, visually displaying the range of potential future data fluctuations. Furthermore, the selection of the prediction range is also a trade-off. In this invention, the default prediction range is set to one-fifth of the current sequence length (which can be adjusted as needed). The annotation results use dashed curves to represent the prediction trajectory, a strip of semi-transparent area to display the confidence interval, and time axis markers at the prediction start point. Figure 3 The document provides an example of a predictive annotation, which uses historical average cellular subscription data from 1980 to 2020 to perform ARMA model predictive analysis, showing that the average cellular subscription will show a downward trend in the future.

[0059] (2-3) Annotation based on external data: Annotation based on external data aims to expand the information dimension of the current time series visualization by discovering and introducing contextual pattern information in the source dataset that is related to the attributes analyzed in the current chart. Three external annotation tasks closely related to time series relationships were designed, namely detail, correlation, and lead-lag, as follows:

[0060] (2-3-1) Detailed Annotations: Detailed annotations aim to provide more granular information about observed patterns in time series data. A significant characteristic of time series data is that it typically contains a large number of data points, with a large number of records at each time point. To better observe overall trends, users often perform aggregation operations on observed dimensions (e.g., calculating statistical measures such as mean, minimum, and maximum). However, such aggregation inevitably leads to the loss of detailed information, making it difficult for users to perceive the underlying data distribution patterns behind the aggregation results. The purpose of detailed annotations is to provide the raw time series information before aggregation; therefore, it is only applicable to time series visualizations where aggregation operations are involved.

[0061] The steps for generating detailed information annotations are as follows: First, the raw data before observation attribute aggregation needs to be obtained from the source dataset. Then, an appropriate category field is selected as the label from the source dataset; by default, the category field with the fewest data items is selected. Different category fields will determine the granularity of the detailed data presentation. For example, selecting "Country" as the label can display a more detailed data distribution than selecting "Region". Finally, the detailed data is rendered in the corresponding time-series pattern area of ​​the original visualization to provide more detailed data distribution information behind the pattern. Furthermore, to improve the efficiency of examining the detailed data, an interactive detailed examination mechanism is introduced. Specifically, when a user views a data point, the system will automatically connect other data points in the category to which that point belongs to, highlighting the overall trend changes of that category.

[0062] Figure 4 The documentation provides an example of detailed annotations. In time-series visualizations of cellular subscriptions with average aggregation, these annotations allow users to examine detailed cellular subscription data for different regions or countries. For instance, when a user focuses on examining cellular subscriptions in Europe in 2003, all data points for the annotated period in Europe are automatically connected to reveal trends in cellular subscriptions across the entire European region.

[0063] (2-3-2) Correlation annotation: Correlation annotation reveals the potential correlation between data by comparing the changing trends of different dimensions, which can help users identify synchronous increase / decrease or reverse change patterns within a specific time period.

[0064] The relevance annotations in this technique consider relevance from two perspectives: attribute-oriented and subset-oriented.

[0065] Attribute-oriented association focuses on other attribute fields that are parallel to the current analysis dimension. For example, when analyzing cellular network subscriptions, related attributes such as the number of internet users and broadband network subscriptions might be considered.

[0066] Subset-oriented associations emphasize comparisons across different data slices or filtering conditions. For example, when analyzing trends in "cellular network subscriptions" in Asia, it can be compared with patterns of change in other markets such as Europe and Africa.

[0067] The process of generating correlation annotations includes the following steps: First, for the identified time pattern, the correlation coefficient between the correlation dimension (numerical dimension) in the source dataset and the target attribute within that time window is calculated. Here, the commonly used Pearson correlation coefficient is used to quantify the correlation. Next, the change curves of the correlation attributes are presented, and colors are used to map the degree of correlation (green represents negative correlation, and red represents positive correlation).

[0068] Integrating different correlation attributes into a unified visualization chart presents the challenge of displaying different units of measurement. For subset-based correlation annotations, since these subset attributes have the same units of measurement as the attribute being analyzed, they can be directly integrated into the corresponding annotation range of the chart. For attribute-based correlation annotations, the correlation attributes often have different units of measurement than the attribute being analyzed. To address this, a remapping strategy is proposed: using the current position of the analyzed attribute in the chart as a baseline, all data points of the correlation attributes are remapped to align with their position. Furthermore, to help users accurately identify the differences in units of measurement between attributes, each correlation attribute is explicitly labeled, and a specific numerical value is assigned to each data point.

[0069] Figure 5 The document provides an example of correlation annotation. For the declining trend in cellular network subscriptions in Asia during the 2019-2020 period, subset-oriented correlation annotations show a positive correlation between Europe, North America, and Oceania and Asia, while South America and Africa show a negative correlation. In attribute-oriented correlation annotations, broadband network subscriptions, the number of internet users, and the percentage of internet users show a negative correlation.

[0070] (2-3-3) Lead-Lap Annotation: Lead-lap annotation aims to reveal the lead-lap relationship between different sequences. Unlike correlation annotation, which analyzes the numerical correlation between variables within the same time interval, lead-lap annotation focuses on examining dynamic dependencies between attributes across time intervals.

[0071] Similar to relevance annotation, lead-lag annotation also considers both attribute-oriented and subset-oriented perspectives. Attribute-oriented lead-lag analysis focuses on the dynamic relationships between different attribute variables. For example, the growth of cellular network subscriptions may precede the growth of broadband network subscriptions, which is consistent with the historical process of technological development. Subset-oriented lead-lag analysis emphasizes comparing the temporal differences of the same attribute across different subsets. For example, the number of cellular network subscriptions in developed regions (such as Europe) may lead in less developed regions (such as Africa).

[0072] The generation process of lead-lag annotation is as follows: First, time patterns are extracted from the associated attributes in the source dataset, and sequences of the same type as the analysis patterns are selected as candidate sets. This preprocessing step reduces the scale of subsequent matching calculations and improves overall efficiency. Then, after normalizing these candidate sequences and the labeled pattern sequences to a 0-1 scale, DTW (Time-Difference Transmission) is used to quantify their similarity. After obtaining the DTW distance, sequences with a distance lower than a preset threshold are selected as lead-lag annotation objects (the default DTW distance threshold is set to 1, but dynamic adjustment is allowed). Finally, the selected lag sequences are integrated into the original graph, and corresponding attribute labels and similarity values ​​are labeled.

[0073] Similar to the relevance annotation in (2-3-2), lead-lag annotation also faces the problem of inconsistent attribute dimensions. To facilitate the observation of lead-lag patterns, this technique adopts a unified remapping strategy for lead-lag annotation: using the value range of the annotated pattern sequence on the attribute axis (i.e., the y-axis) as a reference, the data of the associated attributes are remapped to a position aligned with this reference range, while maintaining their original position on the time axis (i.e., the x-axis).

[0074] Figure 6 An example of lead-lag annotation provided in the text shows that, for the period from 1988 to 2019, subset-oriented lead-lag annotation shows that the upward trend in cellular network subscriptions in Europe leads that in Asia, while Asia leads that in Africa. In attribute-oriented lead-lag annotation, broadband network subscriptions in Asia lag behind the growth in cellular network subscriptions.

[0075] This embodiment proposes a multi-level context-enhanced time-series visualization annotation method, which enhances users' understanding of multi-dimensional time-series data. This method expands the dimensions of visualization annotation, extending it beyond the visual patterns of the chart itself to include rich external contextual information. This multi-dimensional time-series visualization annotation method is applicable to various fields, including financial market analysis, scientific research, and market trend forecasting. With the continuous development of data visualization technology, this method is expected to be further optimized, bringing users a more intelligent and efficient data insight experience.

Claims

1. A multi-level context-enhanced temporal visualization annotation method, characterized in that, An annotated event is represented in the form of <annotation object, annotation content>, where the annotation object refers to one of the seven typical time patterns in time series visualization: rising, falling, single peak, single trough, double peak, double trough, and periodic oscillation. The annotation content refers to the multi-level contextual annotation information added to the annotation object. The method includes the following steps: (1) Temporal data pattern recognition based on prompt-based large language model LLM: using the powerful reasoning and generation capabilities of LLM, the temporal patterns in temporal data can be identified efficiently and accurately. (2) Multi-level context annotation construction: Based on the identified temporal pattern, the following three levels of visualization annotation are automatically constructed: based on observation, based on derived calculation and based on external data. These three levels correspond to the annotation information coming from the data directly displayed in the chart, the result of further derived calculation based on the data displayed in the chart, and the context pattern information in the source data that is associated with the attribute analyzed in the current chart. The (2) step includes the following sub-steps: (2-1) Observation-based annotation: Observation-based annotation focuses on what is directly presented in the visualization without additional calculations or input. At this annotation level, there is only one type of task: text annotation. Text annotation is used to provide a textual description of the visual patterns observed in the visualization chart. These annotations not only reveal the overall trends that are visible at the visual level, but also provide detailed information that is difficult to observe by visual inspection alone. (2-2) Annotations based on derived calculations: These annotations provide deeper insights by performing further derived calculations and analyses on the data in the charts, including both statistical annotations and predictive annotations. (2-3) Annotation based on external data: Annotation based on external data aims to expand the information dimension of the current time series visualization by discovering and introducing contextual pattern information in the source dataset that is related to the attributes analyzed in the current chart. Three external annotation tasks closely related to the time series are designed: details, correlation and lead-lag.

2. The multi-level context-enhanced temporal visualization annotation method as described in claim 1, characterized in that, The (1) step includes the following sub-steps: (1-1) Time series data normalization: Z-score normalization method is used to apply to the original time series data to eliminate the influence caused by scale differences between time series data, so that the data are in the same numerical range, which is beneficial to the subsequent LLM processing and the accuracy of pattern recognition. (1-2) Construct effective prompts to guide the LLM to understand the task and generate the expected output; (1-3) Use LLM to analyze the normalized time series data, identify and describe the patterns in the data.

3. The multi-level context-enhanced temporal visualization annotation method as described in claim 1, characterized in that, In (2-1), the steps for generating the visual pattern text description are as follows: (2-1-1) Use an LLM-driven method to generate text annotations. Input the time-series pattern information detected in step (1), the metadata of the visualization chart, and the original time-series dataset into the LLM. The time-series pattern information includes pattern classification labels and time positioning coordinates. The metadata of the visualization chart includes titles and coordinate axis semantic labels. The original time-series dataset includes timestamp-value pair sequences. By carefully crafting prompts, guide the LLM to generate text descriptions of the visual patterns observed in the visualization chart based on the input data. (2-1-2) Use different colors to selectively highlight key information in the generated text description. A keyword dictionary is pre-constructed, and then regular expressions are used to check the generated text annotations in turn to identify the keyword information and apply the corresponding colors to highlight it.

4. The multi-level context-enhanced temporal visualization annotation method as described in claim 1, characterized in that, In (2-2), the statistical annotations and predictive annotations are as follows: (2-2-1) Statistical annotation: By calculating the presented data, statistical indicators that quantitatively describe the characteristics of the data are generated. Such annotations provide accurate statistical attributes that are not easily perceived visually, thereby helping users to understand the intrinsic characteristics of the data more deeply. The structure of a statistical annotation event includes two parts: <statistical scope and statistical method>. The statistical scope defines the range of data to be statistically analyzed, and the statistical method indicates the type of statistical calculation. Then, according to the statistical scope and statistical method, quantitative calculations are performed on the target data, and the calculation results are displayed as annotation information in the chart through auxiliary lines and text labels to help users identify them. (2-2-2) Predictive annotation: Predictive annotation aims to infer future time series trends by utilizing existing time series data and predictive models. It uses an AutoARMA model to dynamically estimate future data points, calculates the prediction interval at a 95% confidence level, and sets the prediction range to one-fifth of the current sequence length. The annotation results use dashed curves to represent the prediction trajectory, a strip of semi-transparent area to display the confidence interval, and a time axis marker at the prediction start point.

5. The multi-level context-enhanced temporal visualization annotation method as described in claim 1, characterized in that, In (2-3), the details, relevance, and lead-lag annotations are as follows: (2-3-1) Detailed annotations: Detailed annotations are designed to provide more granular information about patterns observed in time series data. A significant characteristic of time series data is that it contains a large number of data points, with a large number of records at each time point. The purpose of detailed annotations is to provide raw time series information before aggregation, so it is only applicable to time series visualizations where aggregation operations are involved. The process of generating detailed information annotations is as follows: First, the raw data before the aggregation of observed attributes needs to be obtained from the source dataset; then, the category field is selected as the label from the source dataset, with the category field having the fewest data items selected by default. Different category fields will determine the granularity of the detailed data presentation; finally, the detailed data is rendered in the corresponding time series pattern area in the original visualization to provide more detailed data distribution information behind the pattern. To improve the efficiency of checking the detailed data, an interactive detailed inspection mechanism is introduced. When a user views a data point, the system will automatically connect other data points in the category to which the point belongs to highlight the overall trend change of the category. (2-3-2) Correlation annotation: Correlation annotation reveals the potential correlation between data by comparing the changing trends of different dimensions, which can help users identify synchronous increase / decrease or reverse change patterns within a time period; Relevance annotations consider the relevance of two perspectives: attribute-oriented and subset-oriented; Attribute-oriented association focuses on other attribute fields that are parallel to the current analysis dimension; Subset-oriented associations emphasize comparisons under different data slices or filtering conditions; The process of generating correlation annotations is as follows: First, for the identified time patterns, the correlation coefficient between the correlation dimension in the source dataset and the target attribute within the time window is calculated. The Pearson correlation coefficient is used to quantify the correlation. Then, the change curve of the correlation attribute is presented, and the degree of correlation is mapped by color, i.e., green represents negative correlation and red represents positive correlation. For subset-oriented relevance annotations, since these subset attributes have the same scale as the attribute being analyzed, they are directly integrated into the corresponding annotation range of the chart. For attribute-oriented correlation annotations, since the related attributes and the analyzed attributes have different dimensions, a remapping strategy is adopted: based on the current position area of ​​the analyzed attribute in the chart, all data points of the related attributes are remapped to align with its position area. At the same time, in order to help users accurately identify the dimensional differences between attributes, each related attribute is clearly labeled and a specific value is attached to each data point. (2-3-3) Lead-Lap annotation: Lead-lap annotation aims to reveal the lead-lap relationship between different sequences. Unlike correlation annotation, which analyzes the numerical correlation between variables within the same time interval, lead-lap annotation focuses on examining the dynamic dependency between attributes across time intervals. Lead-lag annotation considers two perspectives: attribute-oriented and subset-oriented. Attribute-oriented lead-lag analysis focuses on the dynamic relationship between different attribute variables; subset-oriented lead-lag analysis emphasizes comparing the temporal differences of the same attribute between different subsets. The generation process of lead-lag annotation is as follows: First, time patterns are extracted from the associated attributes in the source dataset, and sequences of the same type as the analysis patterns are selected as candidate sets; then, after normalizing these candidate sequences and the labeled pattern sequences to a 0-1 scale, DTW is used to quantify their similarity. After obtaining the DTW distance, sequences with a distance lower than a preset threshold are selected as lead-lag annotation objects; finally, the selected lead-lag sequences are integrated into the original chart, and corresponding attribute labels and similarity values ​​are labeled. A unified remapping strategy is adopted for lead-lag annotations: the range of values ​​of the annotated pattern sequence on the attribute axis, i.e. the y-axis, is used as a reference. The data of the associated attributes are remapped to a position aligned with the reference range, while maintaining their original position on the time axis, i.e. the x-axis.

Images