Contextual visual interaction completion method and device based on large model dialogue

Abstract

The embodiment of the application provides a context visualization interaction completion method and device based on a large model dialogue, relates to the technical field of artificial intelligence, and the method comprises the following steps: converting a first chart demand description of natural language into a first chart text syntax, generating a first chart according to the first chart text syntax; based on user change operation information of the first chart, generating second chart text syntax and a second syntax tree corresponding to the changed second chart, and generating a chart change record according to the difference between the second syntax tree and a first syntax tree corresponding to the first chart; generating a chart evolution data set according to the first chart text syntax, the chart change record and the second chart text syntax, and then training a large language model. The application can provide stable sample data sources for the training of the large language model, so that the model learns chart text syntax understanding, structure reasoning and change record generation capabilities, and provides higher level generalization and reasoning support for the system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a context visualization interaction completion method and device based on large model dialogue. BACKGROUND

[0002] With the increasing capabilities of LLM (Large Language Models) in natural language understanding and code generation, more and more systems attempt to interact with users through natural language to generate visual charts including flowcharts, time sequence charts, relationship charts, etc. In related technologies, some researches or products have attempted to convert natural language input into chart text syntax and then visualize it through graphical rendering tools. However, these researches or products mostly take "one-way generation" as the core idea, i.e., they can only generate static charts from text, and users cannot interactively edit the generated charts, and LLMs cannot perceive the user's operation intention on the charts, thus it is difficult to meet the user's demand for LLM processing charts. SUMMARY

[0003] The embodiments of the present application provide a context visualization interaction completion method and device based on large model dialogue to alleviate or solve one or more technical problems existing in the prior art.

[0004] In a first aspect, the embodiments of the present application provide a context visualization interaction completion method based on large model dialogue, comprising:

[0005] performing syntax conversion on a first chart demand description in natural language to obtain a first chart text syntax, and generating a first chart corresponding to the first chart demand description according to the first chart text syntax;

[0006] generating a second chart text syntax and a second syntax tree corresponding to a second chart after a change operation of a user on the first chart, and generating a chart change record described in natural language according to the difference between the second syntax tree and a first syntax tree corresponding to the first chart; the change operation information includes operation information of performing a change operation on the chart structure of the first chart;

[0007] generating a chart evolution dataset according to the first chart text syntax, the chart change record, and the second chart text syntax;

[0008] performing model training on a large language model using the chart evolution dataset.

[0009] In a second aspect, the embodiments of the present application provide a context visualization interaction completion device based on large model dialogue, comprising:

[0010] a syntax conversion module, configured to perform syntax conversion on the first chart requirement description in natural language to obtain a first chart text syntax, and generate a first chart corresponding to the first chart requirement description according to the first chart text syntax;

[0011] a first generation module, configured to generate a second chart text syntax and a second syntax tree corresponding to a second chart after a change according to change operation information of the user on the first chart, and generate a chart change record described in natural language according to a difference between the second syntax tree and a first syntax tree corresponding to the first chart; the change operation information includes operation information of performing a change operation on a chart structure of the first chart;

[0012] a second generation module, configured to generate a chart evolution data set according to the first chart text syntax, the chart change record, and the second chart text syntax;

[0013] a training module, configured to perform model training on a large language model by using the chart evolution data set.

[0014] In a third aspect, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory. The processor implements any of the methods provided in the embodiments of the present application when executing the computer program.

[0015] In a fourth aspect, a computer-readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement any of the methods provided in the embodiments of the present application.

[0016] In a fifth aspect, a computer program product is provided, which includes a computer program. The computer program is executed by a processor to implement any of the methods provided in the embodiments of the present application.

[0017] According to the technical scheme of the embodiment of the present application, the first chart text syntax is obtained by performing syntax conversion on the first chart requirement description in natural language, and the first chart corresponding to the first chart requirement description is generated according to the first chart text syntax; based on the change operation information of the user to the first chart, the second chart text syntax and the second syntax tree corresponding to the changed second chart are generated, and the chart change record described in natural language is generated according to the difference between the second syntax tree and the first syntax tree corresponding to the first chart, and the change operation information includes the operation information of performing the change operation on the chart structure of the first chart; then, the chart evolution data set is generated according to the first chart text syntax, the chart change record and the second chart text syntax, and the model training of the large language model is performed by using the chart evolution data set. It can be seen that the technical scheme establishes the structure correspondence relationship between the chart from the syntax generation to the change interaction, effectively improving the accuracy and consistency of the syntax generation and the chart presentation. Moreover, since the generated chart supports the visual editing of the user to the chart structure, and the chart change record after the chart interaction of the user can be recorded according to the difference between the syntax trees before and after the chart change, the intelligent recording of the chart change process, the recording of the chart evolution data set and the chart change record supplement of the dialogue context of the large model are realized, not only the manual intervention cost is reduced, but also stable sample data sources are provided for the training of the large language model. In addition, by training the large language model by using the chart evolution data set, the large language model can learn the chart text syntax understanding, structure reasoning and change record generation capability, and the modeling capability of the chart semantics and the change process is enhanced, which provides higher level of generalization and reasoning support for the overall system.

[0018] The above description is only a summary of the technical scheme of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the content of the specification, and in order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described below. BRIEF DESCRIPTION OF DRAWINGS

[0019] In the drawings, like reference numerals refer to same or similar components throughout the several views. The drawings are not necessarily to scale. It should be understood that the drawings only depict some embodiments in accordance with the present application and should not be considered as limiting the scope of the present application.

[0020] Figure 1 A flowchart of a large model dialogue context visual interactive completion method provided by an embodiment of the present application is shown;

[0021] Figure 2 A structural diagram of a large model dialogue context visual interactive completion system provided by an embodiment of the present application is shown;

[0022] Figure 3 A flowchart of a method for context visualization interaction completion based on large model dialogue according to another embodiment of the present application is shown;

[0023] Figure 4 A block diagram of an apparatus for context visualization interaction completion based on large model dialogue according to an embodiment of the present application is shown;

[0024] Figure 5 A block diagram of an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0025] In the following, only certain exemplary embodiments are simply described. As those skilled in the art can recognize, the described embodiments can be modified in various different ways without departing from the concept or scope of the present application. Therefore, the drawings and the description are considered to be exemplary in nature, rather than limiting.

[0026] In order to facilitate understanding of the technical solutions of the embodiments of the present application, the related technologies of the embodiments of the present application are described below. The following related technologies can be combined with the technical solutions of the embodiments of the present application in any manner as optional solutions, which all belong to the protection scope of the embodiments of the present application.

[0027] As mentioned earlier, the text syntax chart processing system currently used by mainstream large models mainly uses a "one-way generation" core idea, i.e., it can only generate static charts from text, and users cannot interactively edit the generated charts. If the user pastes the text locally for modification, the LLM cannot perceive the user's operation intention for the chart, and the user needs to have a certain learning cost to edit the chart text syntax, thus making it difficult to meet the user's demand for LLM processing of charts. To address this technical problem, in related technologies, a text syntax chart generation mechanism based on the ECharts tool (an open-source visualization chart library based on JavaScript) is proposed. This mechanism can automatically generate new charts from raw data according to chart modification configurations. However, this approach mainly focuses on one-way conversion from instructions to chart modifications. In the current complex and diverse actual business application scenarios, the user's demand for charts is not limited to static display after modifying the chart, but also includes interactive operations on the chart, such as adding, deleting, and editing nodes, and feeding back these operations to the LLM and updating the chart. In addition, in the text syntax chart generation mechanism in related technologies, the user cannot interact with the chart, such as editing the chart locally. The lack of context also limits the understanding of the interactive context by the large language model, making the conversation between the user and the large language model lack coherence, thus making it difficult to meet the demand for interactivity and dynamic updating in the chart processing system. Based on this, considering that ECharts is a highly configurable chart library that supports rendering structured data into interactive charts, and the current large model field mainly uses the technology of generating static charts from text syntax, which cannot support chart interaction, ECharts is added between text syntax and chart interaction, so that the chart generated from text syntax can support visual interaction.

[0028] To address the above technical problems, the present application provides a context visualization interactive completion method based on large model dialogue. This method takes natural language-driven chart generation and interaction as the core, realizes the structural conversion of various chart text syntaxes (such as Mermaid, PlantUML class chart syntax) to ECharts visualization configurations through a unified structured AST (Abstract Syntax Tree) parsing mechanism, supports user editing and operation of chart structures on the chart rendering interface, and finally converts back to chart text syntax and completes the conversation context, thereby realizing the complete process system of generating chart text syntax from natural language, parsing chart text syntax into structured charts, interactively editing charts, analyzing chart changes based on interactive operations, and reconstructing the syntax closed loop. In this process, high-quality chart evolution datasets can also be generated to support the continuous optimization and fine-tuning of large language models.

[0029] The technical solutions of the present application and how the technical solutions of the present application solve the foregoing technical problems will be described in detail below with specific embodiments. The several specific embodiments listed can be combined with each other, and the same or similar concepts or processes can not be described again in some embodiments. The embodiments of the present application will be described in detail below with reference to the drawings.

[0030] Figure 1 A flowchart of the context visualization interaction completion method based on large model dialogue provided by the embodiments of the present application is shown, as shown in Figure 1 The method can include steps S101, S102, S103 and S104.

[0031] Step S101, the first chart text syntax is obtained by performing syntax conversion on the natural language first chart requirement description, and the first chart corresponding to the first chart requirement description is generated according to the first chart text syntax.

[0032] The chart text syntax is a syntax rule for describing chart content through structured text code, which combines the logicality of chart syntax and the convenience of text-driven, and is an efficient tool in the fields of data visualization, software design, document writing, etc. In this embodiment, the chart text syntax can be Mermaid or PlantUML text syntax.

[0033] After generating the first chart, the first chart can be rendered to enable the user to perform chart interaction on the rendered first chart interface. After the user operates the first chart, the system generates change operation information corresponding to the first chart in response to the user's operation action on the first chart, and the change operation information is used to generate a chart change record.

[0034] The change operation information includes operation information of performing a change operation on the chart structure of the first chart, and the change operation information can include at least one of the following: change type, change target, original value and change value. The change type includes but is not limited to the following: node addition and deletion, edge relationship change, node attribute adjustment, node position modification, etc. The change target refers to the chart component to which the user performs the operation action. The original value refers to the state value of the chart component before the user performs the operation action, and the change value refers to the state value of the chart component after the user performs the operation action. It can be understood that the object of the user's operation action is the first chart itself, and the operation result is that the chart structure of the first chart is changed. Compared with the related art of user inputting text format chart operation information, the present application can meet the user's demand for interactivity and dynamic update of the chart structure of the generated chart.

[0035] In step S101, the LLM can generate a chart syntax for the natural language first chart requirement description to obtain the first chart text syntax.

[0036] In step S102, based on the change operation information of the user on the first chart, a second chart text syntax corresponding to the changed second chart and a second syntax tree are generated, and a chart change record described in natural language is generated according to the difference between the second syntax tree and the first syntax tree corresponding to the first chart.

[0037] The image change record is the result of intelligent recording of the change operation information. Alternatively, the change operation information is first recorded as standardized operation record data, and then the standardized operation record data is converted into an image change record described in natural language. The standardized operation record data can include at least one of the following: change type, change target, original value, and change value.

[0038] In step S103, a chart evolution dataset is generated according to the first chart text syntax, the chart change record, and the second chart text syntax.

[0039] The chart evolution dataset is composed of the first chart text syntax, the chart change record, and the second chart text syntax.

[0040] In step S104, the chart evolution dataset is used to train the large language model.

[0041] In this embodiment, the large language model can be obtained by fine-tuning the base model. Therefore, before step S101 is performed, the base model can be fine-tuned using a sample dataset to obtain the large language model, i.e., the large language model is a preliminary fine-tuned model for chart text syntax operation understanding. The fine-tuning of the base model can be performed as follows: first, obtain a sample dataset, which includes a first sample chart before change, a sample chart change record in natural language, and a second sample chart after change, which is obtained by performing a change operation on the first sample chart based on the sample chart change record; second, input the sample dataset into the base model to perform syntax analysis on the first sample chart, the sample chart change record, and the second sample chart; third, fine-tune the base model according to the syntax analysis result to obtain the large language model. By fine-tuning the large language model, the large language model has the ability to preliminarily understand chart text syntax operations.

[0042] According to the technical scheme of the embodiment of the present application, the first chart text syntax is obtained by performing syntax conversion on the first chart requirement description in natural language, and the first chart corresponding to the first chart requirement description is generated according to the first chart text syntax; based on the change operation information of the user on the first chart, the second chart text syntax and the second syntax tree corresponding to the changed second chart are generated, and the chart change record described in natural language is generated according to the difference between the second syntax tree and the first syntax tree corresponding to the first chart, and the change operation information includes the operation information of performing the change operation on the chart structure of the first chart; then, the chart evolution data set is generated according to the first chart text syntax, the chart change record and the second chart text syntax, and the model training of the large language model is performed by using the chart evolution data set. It can be seen that the technical scheme establishes the structural correspondence relationship between the chart from the syntax generation to the change interaction, effectively improving the accuracy and consistency of the syntax generation and the chart presentation. Moreover, since the generated chart supports the visual editing of the user on the chart structure, and the chart change record after the chart interaction of the user can be recorded according to the difference between the syntax trees before and after the chart change, the intelligent recording of the chart change process, the recording of the chart evolution data set and the chart change record supplement of the dialogue context of the large model are realized, which not only reduces the manual intervention cost, but also provides stable sample data source for the training of the large language model. In addition, by training the large language model by using the chart evolution data set, the large language model can learn the chart text syntax understanding, structure reasoning and change record generation capability, enhance the modeling capability of the chart semantics and change process, and provide higher level of generalization and reasoning support for the overall system.

[0043] In some embodiments, when the first chart corresponding to the first chart requirement description is generated according to the first chart text syntax (i.e., step S101 is performed), the following steps A1 to A3 can be performed.

[0044] Step A1, determining the chart component sequence of the first chart according to the first chart text syntax.

[0045] The chart component sequence refers to the process of combining and using various chart elements or components in a certain order in the chart construction process. The chart component sequence includes at least one component type of the chart component and the corresponding syntax value. Optionally, the chart component sequence is a set of structured tokens. The system can parse the chart text syntax character by character to generate a set of token sequences, and these tokens include the component type representation of the chart component and the corresponding syntax value. For example, keywords (type: "keyword"), node names (type: "name"), arrow symbols (type: "arrow") and parentheses identifiers (type: "paren"), etc., which constitute the basic components of the chart text syntax.

[0046] Step A2: constructing a first syntax tree of the first graph according to the graph component sequence; the syntax tree is used to represent the logical structure of the graph components.

[0047] Optionally, based on the token sequence, the system constructs an abstract syntax tree (AST). This tree uses a hierarchical, nested data structure to explicitly represent the logical structure and connections of the graph. Taking a statechart as an example, the system parses the starting state, state name, and transition relationships between states into structural nodes in the AST. Each connection relationship corresponds to an object with two fields, from and to, representing the object's starting and ending states.

[0048] Step A3: Generate a first graph based on the syntax tree.

[0049] Optionally, after generating the AST, a data mapping operation is performed from the AST to the ECharts chart configuration based on the AST structure. Specifically, each state node in the AST is converted into a node object in the ECharts chart configuration, containing a display label and optional style information, such as a symbol representing the state shape. The connection between states is converted into a link object, containing the source node, the target node, and the connection style.

[0050] A sequence of tokens can be represented as follows:

[0051] const tokens = [

[0052] { type: "keyword", value: "stateDiagram-v2"},

[0053] { type: "name", value: "[*]"},

[0054] { type: "arrow", value: "-->"},

[0055] { type: "name", value: "Requirement Analysis"},

[0056] { type: "name", value: "Requirement Analysis"},

[0057] { type: "arrow", value: "-->"},

[0058] { type: "name", value: "UI Design"},

[0059] { type: "name", value: "UI design"},

[0060] { type: "arrow", value: "-->"},

[0061] { type: "name", value: "Product design"},

[0062] { type: "name", value: "Product design"},

[0063] { type: "arrow", value: "-->"},

[0064] { type: "name", value: "Software development"},

[0065] { type: "name", value: "Software development"},

[0066] { type: "arrow", value: "-->"},

[0067] { type: "paren", value: "[*]"}, ];

[0069] The AST can be exemplarily represented as follows:

[0070] const stateDiagramAst = {type: "StateDiagram",body: [{ from: "*", to:"Still"},{ from: "Still", to: "*"},{ from: "Still", to: "Moving"},{ from:"Moving", to: "Still"},{ from: "Moving", to: "Crash"},{ from: "Crash", to:"*"}]};

[0071] In the embodiment, the AST analysis mechanism is used to unify the syntax structure of the chart text of the first chart, and automatically map the ECharts configuration parameters of the chart, so that a structure correspondence relationship is established between the generation and the interactive rendering of the chart, and the accuracy and consistency of the generation and the presentation of the chart text syntax are effectively improved.

[0072] In some embodiments, when generating the chart change record described in natural language according to the difference between the second syntax tree and the first syntax tree corresponding to the first chart (i.e., performing step S102), the following steps B1 to B3 can be performed.

[0073] Step B1, comparing the first syntax tree and the second syntax tree to obtain the difference between the first syntax tree and the second syntax tree.

[0074] Step B2, determining the chart structure difference between the first chart and the second chart according to the difference between the first syntax tree and the second syntax tree.

[0075] Step B3, generating the chart change record described in natural language according to the chart structure difference.

[0076] In this embodiment, the system supports users to visually edit charts in the chart rendering interface and accurately identify changes in chart structure, such as identifying changes in chart structure through a diff algorithm to determine chart structure differences, while expressing user operation actions in natural language semantics, thereby intelligently recording chart change processes and reducing manual intervention costs.

[0077] In some embodiments, after generating the chart evolution dataset, the large language model is trained using the chart evolution dataset. The training process can include the following steps C1 to C3:

[0078] Step C1, cleaning the chart evolution dataset to obtain a cleaned chart evolution dataset.

[0079] Step C2, inputting the cleaned chart evolution dataset into the large language model, and predicting the changed predicted chart text syntax through the large language model according to the first chart text syntax and the chart change record.

[0080] Step C3, iteratively training the large language model according to the difference between the predicted chart text syntax and the second chart text syntax.

[0081] Optionally, the LoRA (Low-Rank Adaptation) method is used to efficiently fine-tune the parameters of the large language model. After training, the trained large language model can be merged with the base model and deployed to the service interface of the system to enhance the user interactive chart generation, modification suggestions, and structure reasoning capabilities.

[0082] In some embodiments, after training the large language model using the chart evolution dataset, the model performance of the large language model can be verified. The model performance verification process can include the following steps D1 to D3:

[0083] Step D1, obtaining a validation data set of the large language model; the validation data set includes: a third chart requirement description in natural language and a chart change intention.

[0084] Step D2, inputting the validation data set into the large language model, changing the third chart corresponding to the third chart requirement description according to the chart change intention, obtaining a fourth chart after the change, and outputting a chart text grammar corresponding to the fourth chart.

[0085] Wherein, after obtaining the fourth chart, the fourth chart is converted into the corresponding chart text grammar, and the way of converting the first chart into the corresponding first chart text grammar in the above embodiment is the same, which is not repeated here.

[0086] Step D3, verifying the model performance of the large language model according to the chart text grammar corresponding to the fourth chart.

[0087] Wherein, when performing step D3, the chart text grammar corresponding to the fourth chart and the validation chart text grammar can be compared to determine whether the similarity of the two reaches a preset similarity threshold. If the preset similarity threshold is reached, it is determined that the large language model has high model performance, and the model verification is passed. The validation chart text grammar refers to the chart text grammar corresponding to the correct changed chart obtained after changing the third chart according to the chart change intention.

[0088] Figure 2 The structure diagram of the context visualization interactive completion system based on large model dialogue provided by the embodiment of the application is shown, as shown in Figure 2 The context visualization interactive completion system based on large model dialogue includes: a natural language input module, a large model generation module, a syntax analysis module, a chart rendering module, an interactive tracking module, a difference analysis and semantic generation module, a data set construction module and a model training module. The user can input the chart requirement description through the natural language dialogue interface of the front end, and the system will automatically generate the chart corresponding to the chart requirement description, support visual interactive editing, and generate new chart text grammar and natural language chart change record based on the operation action of the user on the chart, and then construct the chart evolution data set, and use the chart evolution data set to train the large language model, realize the continuous optimization of the syntax understanding ability.

[0089] The natural language input module is configured to receive a chart requirement description in natural language input by a user. Optionally, the natural language input module is further configured to receive a chart change intention in natural language input by the user. The large model generation module is configured to convert the chart requirement description in natural language into a corresponding chart text syntax, such as Mermaid, PlantUML, etc. The syntax analysis module is configured to analyze the chart text syntax, such as into an AST, and generate ECharts configuration parameters of the chart based on the analysis result. The chart rendering module is configured to render the chart according to the ECharts configuration parameters of the chart. The interaction tracking module is configured to determine operation information of the chart by the user in response to an operation action of the user on the chart rendering interface, and transmit the operation information to the difference analysis and semantic generation module. The difference analysis and semantic generation module is configured to generate a changed syntax tree (such as an AST) in real time according to the operation information of the chart by the user, and compare the changed syntax tree with the syntax tree before the change to determine the chart structure difference before and after the change, and further record the chart change information according to the chart structure difference, and generate a chart change record in natural language description and a chart text syntax after the change. The dataset construction module is configured to generate a chart evolution dataset according to the chart text syntax before the change, the chart change record, and the chart text syntax after the change. The model training module is configured to continuously optimize and fine-tune the large language model according to the chart evolution dataset.

[0090] Figure 3 A flowchart of a method for context visualization interaction completion based on a large model dialogue provided by an embodiment of the present application is shown, as shown in Figure 3 The method is applied to Figure 2 A context visualization interaction completion system based on a large model dialogue provided by an embodiment of the present application is shown, as shown in

[0091] Step S301, a chart evolution dataset is obtained, and the chart evolution dataset is input into a large language model.

[0092] In the early stage of training, a small amount of chart evolution dataset can be manually labeled by a user. Optionally, the user inputs a chart requirement description in natural language and a chart change intention, and labels a chart text syntax after the change. The chart text syntax after the change is the chart text syntax corresponding to the changed chart after the chart is changed according to the chart change intention.

[0093] During the model training process, the chart requirement description and the chart change intention are taken as input data, and the changed chart text syntax is taken as output data. Based on a small amount of annotated chart evolution data set, the large language model can preliminarily learn the syntax conversion capability of the chart and the understanding of the chart change intention. With more and more optimization times, the syntax conversion capability of the large language model and the understanding capability of the chart change intention are gradually strengthened, so that the user no longer needs to manually annotate the chart evolution data set. The large language model can optimize the model through the chart evolution data set intelligently completed by the system.

[0094] In step S302, the large language model outputs the first chart text syntax corresponding to the chart requirement description.

[0095] For example, the user inputs the chart requirement description "draw a flowchart with A pointing to B" through the web page dialog box provided by the system. The large language model converts the chart requirement description into the corresponding Mermaid or PlantUML text syntax.

[0096] In step S303, the first chart text syntax is parsed to generate a chart component sequence.

[0097] The chart component sequence refers to the process of combining and using various chart elements or components in a certain order during chart construction. The chart component sequence includes at least one component type of the chart component and the corresponding syntax value.

[0098] Optionally, the chart component sequence is a set of structured lexical units (i.e., tokens). The system built-in syntax parsing module can parse the chart text syntax character by character to generate a set of token sequences. These tokens include the component type representation of the chart component and the corresponding syntax value.

[0099] In step S304, the AST of the chart is constructed according to the chart component sequence.

[0100] In step S305, the AST of the chart is converted into a chart configuration, and the chart is rendered based on the chart configuration.

[0101] Optionally, the system is built-in with an ECharts component that receives the chart configuration and renders the chart structure. The user can perform interactive operations on the chart rendering interface, such as adding or deleting nodes, moving nodes, adjusting labels, and other interactive operations.

[0102] In step S306, in response to the user's operation action on the chart, the changed AST is generated.

[0103] In step S307, the changed AST and the AST before the change are compared, and the difference between the chart structure before and after the chart change is determined according to the comparison result.

[0104] Optionally, a custom diff algorithm is used to extract the structural changes between the two ASTs before and after the change. Specifically, after the user performs an operation on the chart rendering interface, an AST corresponding to the current chart state is generated in real time. Then, the AST corresponding to the current chart state is recursively compared with the AST before the operation at the field and structure levels. The diff algorithm can identify the following types of changes: node additions and deletions, edge relationship changes, node attribute adjustments, node position modifications, etc.

[0105] Step S308 : generating a chart change record described in natural language according to the chart structure difference, and performing syntax conversion on the changed AST to obtain a changed second chart text syntax.

[0106] Each change item in the chart is recorded as a standardized operation record data, including the change type, change target, original value, and changed value. For example, when a user moves the node "UI Design" to the sub-process node of the node "Product Design" in the chart and adds a node named "Software Testing", the operation record data generated by the system can be expressed as follows:

[0107] [{ "action": "moveNode", "node": { "oldNode": "UI Design", "newNode": "Product Design"}},{ "action": "addNode", node:{"name":"Software Testing"}} ]

[0109] After generating the operation log data, the system's built-in semantic generation module translates it into a highly readable, natural-language diagram change record. For example, the operation log data in the example above can be automatically translated into the following diagram change record: "The user moved the node UI Design under Product Design and added a new node named Software Testing." This conversion of standardized operation log data into natural-language diagram change records serves as contextual input for the large language model, helping it more accurately understand the user's editing intent for the diagram. It can also be used to construct a diagram evolution dataset to record the evolutionary path of the diagram structure.

[0110] Step S309 : generating a graph representation data set according to the first graph text grammar, the graph change record and the second graph text grammar.

[0111] In this step, the system organizes the changed second chart text grammar, the chart change record, and the first chart text grammar before the change into a complete sample data, which can be expressed in the following format:

[0112] [First chart text syntax] + [Chart change record] -> [Second chart text syntax]

[0113] The above format means that the first chart text syntax and the chart change record are input data during training of the large language model, and the second chart text syntax is output data during training of the large language model.

[0114] Taking the chart change record as "moving the node UI design under the product design" as an example, the chart evolution dataset can be exemplarily represented as follows:

[0115] "input": "UI design moves under product design. stateDiagram-v2\n [*] --> requirement analysis\n requirement analysis --> UI design\n UI design --> product design\n product design --> software development\n software development --> [*]",

[0116] "output": "stateDiagram-v2\n [*] --> requirement analysis\n requirement analysis --> product design\n product design --> UI design\n UI design --> software development\n software development --> software testing\n software testing --> [*]"

[0117] In addition, the generated chart evolution dataset can be stored in a structured format, and multiple rounds of interaction can form a sample sequence.

[0118] After step S309 is performed, it can return to step S301, that is, continue to optimize the large language model by using the newly generated chart evolution dataset.

[0119] It should be noted that the operation record data, the format of the chart evolution dataset, and the specific content of the chart evolution dataset listed in the embodiments are exemplary and do not limit the present application. In actual application, the representation method can be customized according to actual needs.

[0120] In this embodiment, the dataset annotated by artificial labeling is used to complete the preliminary training of the large language model, ensuring that the large language model has the basic ability to understand the syntax interaction of the graph and the ability to generate graph text syntax in natural language. After that, the user inputs the expected graph requirement description in natural language, and the graph processing system (hereinafter referred to as the system) generates the corresponding graph text syntax (such as Mermaid or PlantUML text), constructs an abstract syntax tree for the graph text syntax, extracts the graph structure and data content, and automatically converts it into the graph configuration parameters of ECharts, thereby realizing graph rendering. Then, the user can perform interactive operations such as node addition, relationship editing, and element deletion on the graph rendering interface, and the system records the graph change information based on the difference analysis algorithm (such as the diff algorithm).

[0121] After that, the system further records the operation changes of the graph and generates readable natural language descriptions to express the user's modification intention. Finally, the system takes the original graph syntax and the operation difference natural language as input and the changed graph syntax as output to construct a high-quality interactive graph evolution dataset for supporting the training and evaluation of the large model in terms of graph understanding and semantic tracking.

[0122] On this basis, to further improve the model's graph semantic understanding and reconstruction ability, the system supports standardized processing of the constructed dataset, including sample cleaning, structure arrangement, and generalization preservation, and converts it into a data format that meets the requirements of fine-tuning tools (such as Lora) for necessary label labeling. The system can fine-tune the base large model through Lora and other fine-tuning methods, and judge the training effect by combining subjective evaluation and objective indicators. After training is completed, the fine-tuned model can be merged, packaged, and deployed into the graph processing system, ultimately connecting the channels between graph text syntax, graph interaction, structure modeling, and natural language description, and completing the closed-loop process of "graph text syntax - AST - graph configuration - user operation - AST change - syntax reconstruction".

[0123] The system has significant advantages in unified graph language structure analysis, supporting graph interaction semantic backtracking, and automatically generating graph syntax evolution data, solving the problem of lack of closed-loop capability from syntax to graphics to interaction in existing graph intelligent generation systems. Relying on the above structure closed loop and data mechanism, the system can be used to train a large model with graph text syntax understanding, structure reasoning, and interactive operation generation capability, enhancing its modeling capability for graph semantics and evolution process, and providing higher level of generalization and reasoning support for intelligent graph systems.

[0124] Corresponding to the application scenarios and methods of the method provided in the embodiments of the present application, the embodiments of the present application also provide a context visualization interaction completion device based on a large model dialogue.

[0125] Figure 4 A block diagram of the context visualization interactive completion device based on large model dialogue provided by the embodiments of the present application is shown in FIG. 1. As shown in FIG. 1, the context visualization interactive completion device based on large model dialogue includes: Figure 4

[0126] The syntax conversion module 41 is configured to perform syntax conversion on the first chart requirement description in natural language to obtain a first chart text syntax, and generate a first chart corresponding to the first chart requirement description according to the first chart text syntax.

[0127] The first generation module 42 is configured to generate a second chart text syntax and a second syntax tree corresponding to a second chart after a change operation according to user change operation information on the first chart, and generate a chart change record described in natural language according to a difference between the second syntax tree and a first syntax tree corresponding to the first chart; the change operation information includes operation information of performing a change operation on a chart structure of the first chart.

[0128] The second generation module 43 is configured to generate a chart evolution data set according to the first chart text syntax, the chart change record, and the second chart text syntax.

[0129] The training module 44 is configured to perform model training on a large language model using the chart evolution data set.

[0130] In some embodiments, when the syntax conversion module 41 generates the first chart corresponding to the first chart requirement description according to the first chart text syntax, the following steps are performed:

[0131] According to the first chart text syntax, a chart component sequence of the first chart is determined; the chart component sequence includes a component type of at least one chart component and a corresponding syntax value thereof;

[0132] According to the chart component sequence, a first syntax tree of the first chart is constructed; the syntax tree is used to represent a logical structure of the chart component;

[0133] According to the syntax tree, the first chart is generated.

[0134] In some embodiments, when the first generation module 42 generates a chart change record described in natural language according to a difference between the second syntax tree and the first syntax tree corresponding to the first chart, the following steps are performed:

[0135] The first syntax tree and the second syntax tree are compared to obtain a difference between the first syntax tree and the second syntax tree;

[0136] ​determine a chart structure difference between the first chart and the second chart according to a difference between the first syntax tree and the second syntax tree;

[0137] generate the chart change record according to the chart structure difference.

[0138] In some embodiments, the apparatus further comprises:

[0139] a first obtaining module, configured to obtain a sample data set before syntax conversion is performed on the first chart requirement description in natural language to obtain a first chart text syntax; the sample data set comprises: a first sample chart before change, a sample chart change record in natural language, and a second sample chart after change;

[0140] a syntax analysis module, configured to input the sample data set into a base model, and perform syntax analysis on the first sample chart, the sample chart change record, and the second sample chart by using the base model;

[0141] a fine-tuning module, configured to fine-tune the base model according to a syntax analysis result to obtain the large language model.

[0142] In some embodiments, the apparatus further comprises:

[0143] a rendering module, configured to render the first chart after the first chart is generated according to the first chart text syntax;

[0144] a third generation module, configured to generate the change operation information corresponding to the first chart in response to an operation action of a user on the first chart; the change operation information comprises at least one of the following: a change type, a change target, an original value, and a change value.

[0145] In some embodiments, the training module 44 performs the following steps when training the large language model by using the chart evolution data set:

[0146] cleaning the chart evolution data set to obtain a cleaned chart evolution data set;

[0147] inputting the cleaned chart evolution data set into the large language model, and predicting a predicted chart text syntax after change by using the large language model and according to the first chart text syntax and the chart change record;

[0148] iteratively training the large language model according to a difference between the predicted chart text syntax and the second chart text syntax.

[0149] In some embodiments, the apparatus further comprises:

[0150] The second obtaining module is configured to, after model training of the large language model by using the chart evolution dataset, obtain a verification dataset of the large language model; the verification dataset includes a third chart demand description in natural language and a chart change intention;

[0151] The changing module is configured to input the verification dataset into the large language model, change a third chart corresponding to the third chart demand description according to the chart change intention, obtain a fourth chart after change, and output a chart text grammar corresponding to the fourth chart;

[0152] The verifying module is configured to verify the model performance of the large language model according to the chart text grammar corresponding to the fourth chart.

[0153] According to the technical scheme of the embodiments of the present application, the first chart text grammar is obtained by performing grammar conversion on the first chart demand description in natural language, and the first chart corresponding to the first chart demand description is generated according to the first chart text grammar. Based on the change operation information of the user on the first chart, the second chart text grammar and the second syntax tree corresponding to the second chart after change are generated, and the chart change record described in natural language is generated according to the difference between the second syntax tree and the first syntax tree corresponding to the first chart. The change operation information includes operation information of performing change operation on the chart structure of the first chart. Then, the chart evolution dataset is generated according to the first chart text grammar, the chart change record and the second chart text grammar, and the large language model is trained by using the chart evolution dataset. It can be seen that the technical scheme establishes the structure correspondence relationship between the chart from grammar generation and change interaction, effectively improves the accuracy and consistency of grammar generation and chart presentation. Moreover, since the generated chart supports visual editing of the chart structure by the user, and can record the chart change record after the chart interaction of the user according to the difference between the syntax trees before and after the chart change, the intelligent recording of the chart change process, the recording of the chart evolution dataset and the chart change record supplement of the dialogue context of the large model are realized. Not only the manual intervention cost is reduced, but also stable sample data source is provided for the training of the large language model. In addition, by training the large language model by using the chart evolution dataset, the large language model can learn the chart text grammar understanding, structure reasoning and change record generation capability, and enhance the modeling capability of the chart semantics and change process, thereby providing higher level of generalization and reasoning support for the overall system.

[0154] The functions of each module in each device of the embodiments of the present application can be referred to the corresponding description in the above method, and have corresponding beneficial effects, which will not be repeated here.

[0155] Figure 5A block diagram of an electronic device for implementing the embodiments of the present application. Figure 5 As shown, the electronic device includes a memory 501 and a processor 502. The memory 501 stores a computer program that can be executed on the processor 502. When the processor 502 executes the computer program, the method of the above embodiment is implemented. The number of memory 501 and processor 502 can be one or more. In a specific implementation, the electronic device may also include a communication interface 503 for communicating with external devices and performing data exchange.

[0156] In a specific implementation, if the memory 501, processor 502, and communication interface 503 are implemented independently, the memory 501, processor 502, and communication interface 503 can be connected to each other via a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0157] Optionally, in a specific implementation, if the memory 501 , the processor 502 , and the communication interface 503 are integrated on a chip, the memory 501 , the processor 502 , and the communication interface 503 may communicate with each other through an internal interface.

[0158] An embodiment of the present application provides a computer-readable storage medium storing a computer program, which implements the method provided in the embodiment of the present application when the program is executed by a processor.

[0159] An embodiment of the present application provides a computer program product, including a computer program, which implements the method provided in the embodiment of the present application when executed by a processor.

[0160] An embodiment of the present application also provides a chip, which includes a processor for calling and executing instructions stored in the memory from the memory, so that a communication device equipped with the chip executes the method provided in the embodiment of the present application.

[0161] The embodiment of the present application further provides a chip, comprising: an input interface, an output interface, a processor and a memory, the input interface, the output interface, the processor and the memory are connected through internal connection paths, the processor is used for executing the code in the memory, and when the code is executed, the processor is used for executing the method provided by the embodiment of the present application.

[0162] It should be understood that the processor described above can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. It should be noted that the processor can be a processor supporting an advanced RISC machine (ARM) architecture.

[0163] Further, the memory can include a read-only memory and a random access memory, optionally. The memory can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memory. The non-volatile memory can include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically EPROM (EEPROM), or a flash memory, among others. The volatile memory can include a random access memory (RAM), which is used as an external cache. By way of example, and not limitation, many forms of RAM are available. The RAM can include a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate SDRAM (DDR SDRAM), an enhanced SDRAM (ESDRAM), a Sync link DRAM (SLDRAM), and a direct Rambus RAM (DR RAM), among others.

[0164] In the above-described embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the present disclosure are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium.

[0165] In the description of the application, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the application. Moreover, the specific features, structures, materials or characteristics described can be combined in any suitable manner in one or more embodiments or examples. In addition, different embodiments or examples described in the specification and characteristics of different embodiments or examples can be combined and combined by those skilled in the art without contradiction.

[0166] In addition, the terms "first", "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the application, the meaning of "multiple" is two or more, unless otherwise specifically limited.

[0167] Any process or method described in the flowchart or otherwise described herein can be understood as representing a module, segment, or portion of code that includes executable instructions for implementing specific logical functions or steps. And the scope of the preferred embodiments of the application includes additional implementations, in which the functions can be performed in an order other than that shown or discussed, including in a substantially simultaneous manner or in reverse order, according to the functions involved.

[0168] The logic and / or steps described in the flowchart or otherwise described herein, for example, can be considered as a list of executable instructions for implementing the logic function, which can be specifically implemented in any computer readable medium for instruction execution system, device or equipment (such as computer based system, system including processor or other system that can take instructions from instruction execution system, device or equipment and execute instructions) or in conjunction with these instructions execution system, device or equipment.

[0169] It should be understood that parts of the application can be realized by hardware, software, firmware or their combination. In the above described embodiments, a plurality of steps or methods can be realized by software or firmware stored in the memory and executed by the appropriate instruction execution system. All or part of the steps of the above described embodiment method can be instructed by the relevant hardware through the program, which can be stored in a computer readable storage medium, and the program includes one of the steps of the method embodiment or its combination when executed.

[0170] In addition, each of the function units in each embodiment of the present application can be integrated in one processing module, or each unit can be physically present separately, or two or more units can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software function module. When the integrated module is realized in the form of a software function module and sold or used as an independent product, it can also be stored in a computer readable storage medium. The storage medium can be a read-only memory, a magnetic disk or an optical disk, etc.

[0171] The above is only exemplary embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of various changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A large model dialogue-based context visualization interaction completion method, characterized in that, The method comprises the following steps: performing syntax conversion on a first chart requirement description in natural language to obtain a first chart text syntax, and generating a first chart corresponding to the first chart requirement description according to the first chart text syntax; based on user change operation information of the first chart, generating a second chart text syntax and a second syntax tree corresponding to a second chart after change, and generating a chart change record described in natural language according to the difference between the second syntax tree and the first syntax tree corresponding to the first chart; the change operation information includes operation information of performing a change operation on the chart structure of the first chart; generating a chart evolution dataset according to the first chart text syntax, the chart change record and the second chart text syntax; training a large language model using the chart evolution dataset; the method further comprises the following steps: performing syntax conversion on a first chart requirement description in natural language to obtain a first chart text syntax, and generating a first chart corresponding to the first chart requirement description according to the first chart text syntax; 2. The method of claim 1, wherein, the method further comprises the following steps: determining a chart component sequence of the first chart according to the first chart text syntax; the chart component sequence includes the component type of at least one chart component and the corresponding syntax value; constructing a first syntax tree of the first chart according to the chart component sequence; the syntax tree is used to represent the logical structure of the chart component; generating the first chart according to the syntax tree.

3. The method of claim 1, wherein, the method further comprises the following steps: comparing the first syntax tree and the second syntax tree to obtain the difference between the first syntax tree and the second syntax tree; determining the chart structure difference between the first chart and the second chart according to the difference between the first syntax tree and the second syntax tree; generating the chart change record according to the chart structure difference.

4. The method of claim 1, wherein, the method further comprises the following steps before performing syntax conversion on a first chart requirement description in natural language to obtain a first chart text syntax: obtaining a sample dataset; the sample dataset includes a first sample chart before change, a sample chart change record in natural language and a second sample chart after change; inputting the sample dataset into a base model to perform syntax analysis on the first sample chart, the sample chart change record and the second sample chart by the base model; fine-tuning the base model according to the syntax analysis result to obtain the large language model.

5. The method of claim 1, wherein, the method further comprises the following steps after generating the first chart corresponding to the first chart requirement description according to the first chart text syntax: rendering the first chart; In response to a user operation action on the first chart, generate the change operation information corresponding to the first chart; the change operation information includes at least one of the following: change type, change target, original value and change value.

6. The method of claim 1, wherein, The model training of the large language model by using the chart evolution data set comprises: cleaning the chart evolution data set to obtain a cleaned chart evolution data set; inputting the cleaned chart evolution data set into the large language model, predicting a changed predicted chart text syntax by the large language model according to the first chart text syntax and the chart change record, and iteratively training the large language model according to the difference between the predicted chart text syntax and the second chart text syntax.

7. The method of claim 6, wherein, After the model training of the large language model by using the chart evolution data set, the method further comprises: obtaining a verification data set of the large language model; the verification data set comprises a third chart demand description in natural language and a chart change intention; inputting the verification data set into the large language model, changing a third chart corresponding to the third chart demand description according to the chart change intention to obtain a changed fourth chart, and outputting a chart text syntax corresponding to the fourth chart; verifying the model performance of the large language model according to the chart text syntax corresponding to the fourth chart.

8. A large model dialogue-based context visualization interaction completion device, characterized in that, Comprise: a syntax conversion module configured to convert a first chart demand description in natural language into a first chart text syntax, and generate a first chart corresponding to the first chart demand description according to the first chart text syntax; a first generation module configured to generate a second chart text syntax and a second syntax tree corresponding to a second chart changed by a user according to change operation information of the user on the first chart, and generate a chart change record described in natural language according to the difference between the second syntax tree and a first syntax tree corresponding to the first chart; the change operation information comprises operation information of performing a change operation on a chart structure of the first chart; a second generation module configured to generate a chart evolution data set according to the first chart text syntax, the chart change record and the second chart text syntax; a training module configured to train a large language model by using the chart evolution data set; the first generation module is configured to generate a second syntax tree corresponding to a second chart changed by a user according to change operation information of the user on the first chart; convert the second syntax tree into the second chart text syntax.

9. An electronic device, comprising: The computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the method in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the method in any one of claims 1 to 7.

Images