Intelligent chart dynamic generation method based on voice recognition and multi-modal interaction

Abstract

The invention belongs to the technical field of speech recognition, and discloses an intelligent chart dynamic generation method based on speech recognition and multi-modal interaction. Comprising the following steps: collecting user voice data and denoising to obtain accurate user voice data; performing semantic structure segmentation and combination, and outputting complete user semantic data; executing semantic analysis and outputting user query intention information; extracting a query target data subset; constructing a field structure portrait based on the query target data subset, generating a target chart format, and outputting target chart parameters; performing structure correction on the target chart parameters based on the multi-modal interaction information collected in real time, and outputting a dynamically corrected target chart; performing trend evolution display on the dynamically corrected target chart to obtain an optimized target chart, and sending the dynamically optimized target chart to a preset data large screen; the voice recognition accuracy and robustness are improved, the interaction function of the chart is optimized, the interaction requirement of the user is met, and the interaction experience of the user is improved.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and more particularly to a method for dynamically generating intelligent charts based on speech recognition and multimodal interaction. Background Art

[0002] With the rapid development of natural language processing technology, voice interaction technology has been widely used in many fields such as smart cities, traffic management and data visualization. Among them, voice-driven data chart generation technology has gradually become an important component of smart transportation and visual analysis systems. Especially in interactive display terminals, users often prefer to generate the required data charts through the simple method of oral expression, which reduces the burden of manual operation and improves query efficiency. However, existing technologies still face certain challenges in accurate voice recognition and chart generation.

[0003] First, in the existing field of speech recognition technology, when users face the highly complex business scenarios of smart transportation, they may experience speech pauses during voice input due to thinking, hesitation, or additional explanations. Existing speech recognition technology often determines such pauses as sentence boundaries caused by situations such as punctuation, which causes the semantic structure of the recognized speech to be broken, resulting in the inability to generate charts that meet user needs during the subsequent chart generation process, greatly reducing query efficiency; in addition, since users may have certain regional accents or the habit of colloquial expression, the results of speech recognition are not accurate enough, which reduces the recognition accuracy; at the same time, users may express their needs in a certain way. It will spontaneously combine multiple professional terms in the field of smart transportation to form a new and unrecorded compound phrase structure. This non-standard expression makes the semantic parsing process more difficult. Existing technologies find it difficult to accurately identify the semantic information of compound phrase structures, resulting in an inability to understand the user's true intentions. Existing large language models find it difficult to generate suitable chart formats based on inaccurate intention information. Moreover, existing technologies lack accurate identification of user semantic tendencies, and it is easy for generated charts to not conform to current business scenarios. In terms of interaction, existing technologies lack adaptive adjustment mechanisms and trend evolution mechanisms based on user feedback, resulting in charts with weak expressive power that do not meet user needs.

[0004] In view of this, the present invention proposes a method for dynamically generating intelligent charts based on speech recognition and multimodal interaction to solve the above problems. Summary of the Invention

[0005] In order to overcome the above-mentioned defects of the prior art and achieve the above-mentioned objectives, the present invention provides the following technical solution: a method for dynamically generating intelligent charts based on speech recognition and multimodal interaction, comprising: S1. Collect user voice data and perform voice data denoising to obtain accurate user voice data; S2. Perform semantic structure segmentation and combination on the precise user voice data to output complete user semantic data; S3. Perform semantic analysis on the complete user semantic data and output user query intent information; S4. Extracting a subset of query target data from a preset database based on user query intent information; S5. Construct a field structure portrait based on the query target data subset, generate a target chart format based on the field structure portrait using a preset language model, and output the target chart parameters; S6. collecting multimodal interaction information in real time, performing structural correction on target chart parameters based on the multimodal interaction information, and outputting a dynamically corrected target chart; S7. Execute trend evolution display on the dynamically modified target chart to obtain an optimized target chart; and send the dynamically optimized target chart to a preset data large screen.

[0006] Furthermore, the semantic structure segmentation and combination methods include: A dynamic sliding window is set and used to traverse the precise user speech data, and speech intervals whose interval time is greater than the dynamic sliding window size are marked as boundaries; the precise user speech data is structurally divided based on the marked speech intervals, and a set of structured speech segments is output; the two adjacent structured speech segments before and after each structured speech segment are extracted, and the structured speech segment and the adjacent structured speech segments are combined into a context speech segment group; Perform a coherence assessment on each context speech segment group and output a semantic coherence score; set a coherence score threshold, and combine the semantic coherence score and the coherence score threshold to screen the mergeable speech segment groups; merge the mergeable speech segment groups into segments and output complete user semantic data; structurally divide the complete user semantic data based on reasonable speech intervals, and calculate the comprehensive coherence score of the complete user semantic data after division. If it does not meet the coherence score threshold, adjust the segment merging strength; otherwise, do not adjust it.

[0007] Furthermore, the consistency assessment method includes: Perform speech-to-text recognition on any contextual speech segment group and output the original recognized text; collect user information, load the local accent reference database based on the user information, convert the original recognized text to standard text based on the local accent reference database, and output the standard recognized text; identify colloquial words in the standard recognized text, perform word form correction on the colloquial words based on a preset professional expression dictionary, and output the professional expression model text; Identify and label sub-words in professional expression pattern texts; count the number of sub-word switches in the target speech segment, the preceding speech segment, and the following speech segment in the professional expression pattern texts, and output the degree of semantic transition; if the degree of semantic transition is higher than a preset threshold, it is determined to be a broken structure and the fracture weight is output; otherwise, it is determined to be a coherent structure and the coherence weight is output; Semantic vector modeling is performed on professional expression pattern text to obtain the target speech segment semantic vector, pre-segment semantic vector and post-segment semantic vector respectively; the semantic similarity between the target speech segment semantic vector and the pre-segment semantic vector is calculated, and the forward semantic coherence score is output; the semantic similarity between the target speech segment semantic vector and the post-segment semantic vector is calculated, and the backward semantic coherence score is output; the forward semantic coherence score and the backward semantic coherence score are weightedly fused based on the break weight and coherence weight, and the semantic coherence score is output.

[0008] Furthermore, the method of screening the mergeable speech segment groups includes: The coherence score threshold includes a merging threshold and an independent threshold. When the semantic coherence score is higher than or equal to the merging threshold, the corresponding context speech segment group is judged as a mergible speech segment group; when the semantic coherence score is lower than the independent threshold, the structural speech segments corresponding to the context speech segment group are respectively judged as independent semantic structures and do not participate in the merging; when the semantic coherence score is higher than the independent threshold and lower than the merging threshold, if the semantic coherence score is close to the merging threshold, the speech interval of the corresponding context speech segment group is adjusted and the coherence evaluation is re-performed until the semantic coherence score is higher than or equal to the merging threshold; if the semantic coherence score is close to the independent threshold, the semantic interval adjustment is performed first and the coherence evaluation is performed. If the difference between the adjusted semantic coherence score and the merging threshold is greater than the preset distance threshold, the structural speech segment corresponding to the context speech segment is judged as a independent semantic structure.

[0009] Furthermore, the method of performing semantic parsing includes: Extract phrase and sentence structures from complete user semantic data, match them with the preset traffic term dictionary, filter phrases or sentence structures containing traffic terms, and obtain independent traffic term entries; detect whether the independent traffic term entries contain conjunctions, and classify the independent traffic term entries based on the conjunctions, including conjunction compound phrases and non-conjunction compound phrases; Using conjunctions as decomposition markers, the traffic term phrases adjacent to the decomposition markers in the connected compound phrases are extracted; the traffic term phrases are sorted according to the original word order and semantically concatenated to output a class of compound semantic fragments; The non-connected compound phrases are decomposed into lemmas to obtain preliminary split subsequences; a term-lemma relationship graph is constructed based on the preliminary split subsequences, with any lemma as a node, and the semantic relevance between lemmas is calculated and used as edges; the term-lemma relationship graph is clustered by neighbors to obtain the shortest relationship path for each lemma; the shortest relationship paths are reasonably screened and reasonable relationship paths are output; the lemmas in the reasonable relationship paths are semantically concatenated according to the original word order to obtain two types of compound semantic fragments; the first type of compound semantic fragments and the second type of compound semantic fragments are integrated to obtain user query intention information.

[0010] Furthermore, the method of constructing the field structure portrait includes: Extract all relevant fields in the query target data subset, classify the fields based on the source of the relevant fields, and use the traffic business category to which the fields belong to mark the field type; count the basic attribute indicators of the query target data subset, extract the features of the basic attribute indicators, and output the target mathematical distribution characteristics; detect the time range corresponding to each relevant field, and perform trend change analysis on the corresponding values ​​of the relevant fields within the time range to obtain the trend change characteristics of the relevant fields; compare the values ​​of the relevant fields with the preset safety value range, and if they exceed the preset safety value range, determine them as alarm fields, and add alarm marks; match the field type with the target mathematical distribution characteristics, trend change characteristics and alarm marks respectively, and output the field structure portrait of each relevant field.

[0011] Furthermore, the method of generating the target chart format includes: Obtain the speech text corresponding to the user's query intention information, use the pre-trained corpus model to perform semantic extraction on the speech text, and output the user intention vector; judge the user's semantic tendency based on the user intention vector; combine the field structure portrait with the corresponding user semantic tendency to generate demand target information; use the demand target information as the data basis, use the preset language model to construct the target chart format, calculate the matching fitness, and screen candidate charts based on the matching fitness; evaluate the expressive ability of the candidate charts, and output the target chart parameters based on the evaluation results.

[0012] Furthermore, the method of performing structural correction includes: Construct a corresponding graph based on the target graph parameters; extract the interaction mode type and interaction area of ​​the multimodal interaction information after each interaction behavior is completed, and construct interaction association information; based on the interaction association information, divide the interaction behavior into simple interaction behavior and semantic interaction behavior; When the interactive behavior is a simple interactive behavior, the corresponding functional architecture of the corresponding chart is called to adaptively adjust the internal parameters of the structure; when the interactive behavior is a semantic interactive behavior, the semantic target of the semantic interactive behavior is extracted, and the preset language model is used to regenerate the appropriate chart format based on the semantic target, and the target chart parameters are reconfigured; the chart adjusted based on the interactive behavior is output as a dynamically corrected target chart.

[0013] Furthermore, the method of performing trend evolution display includes: Construct an adjustable timeline; bind all fields in the dynamically corrected target chart to corresponding timestamps; and switch the value of each field in the dynamically corrected target chart frame by frame based on the timestamp by adjusting the adjustable timeline to obtain an optimized target chart.

[0014] An intelligent chart dynamic generation system based on speech recognition and multimodal interaction is used to implement an intelligent chart dynamic generation method based on speech recognition and multimodal interaction, and is characterized by comprising: The data acquisition module is used to collect user voice data and perform voice data denoising to obtain accurate user voice data; Semantic segmentation module, used to segment and combine the precise user voice data into semantic structures and output complete user semantic data; Semantic parsing module, used to perform semantic parsing on complete user semantic data and output user query intention information; A data query module is used to extract a query target data subset from a preset database based on user query intention information; A chart generation module is used to construct a field structure portrait based on the query target data subset, generate a target chart format based on the field structure portrait using a preset language model, and output target chart parameters; The interactive optimization module is used to collect multimodal interaction information in real time, perform structural correction on the target chart parameters based on the multimodal interaction information, and output the corrected target chart; The chart visualization module is used to perform trend evolution display on the corrected target chart to obtain a dynamically optimized target chart; send the dynamically optimized target chart to a preset data large screen; and connect each module via wired and / or wireless means.

[0015] The technical effects and advantages of the intelligent chart dynamic generation method based on speech recognition and multimodal interaction of the present invention are as follows: By first performing structural processing and semantic analysis on the collected user voice data, the user query intention information is extracted, and a suitable chart format is generated based on the user query intention information. At the same time, the interaction in the chart is optimized, and a dynamic generation method of intelligent charts based on voice recognition and multimodal interaction is realized. Compared with existing experience, a multi-level analysis is performed on user voice, which significantly improves the accuracy of user intention recognition in the case of voice input, especially in dealing with colloquial expressions, sentence pauses, and accent or dialect interference, thereby improving the robustness of voice recognition. At the same time, the semantic parsing operation is deeply integrated with traffic business knowledge to generate field structure portraits, which provides a basis for generating charts using large language models. Further, by combining chart matching adaptability and expression ability, accurate matching of user semantics and chart types is achieved. By capturing user interaction actions, adaptive adjustment of chart structure parameters and chart formats is achieved, meeting user interaction needs and improving user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of a method for dynamically generating intelligent charts based on speech recognition and multimodal interaction according to the present invention; Figure 2 This is a schematic diagram of the intelligent chart dynamic generation system based on speech recognition and multimodal interaction of the present invention. DETAILED DESCRIPTION

[0017] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0018] Example 1 See also Figure 1 As shown, the method for dynamically generating intelligent charts based on speech recognition and multimodal interaction described in this embodiment includes: S1. Collect user voice data and perform voice data denoising to obtain accurate user voice data; S2. Perform semantic structure segmentation and combination on the precise user voice data to output complete user semantic data; S3. Perform semantic analysis on the complete user semantic data and output user query intent information; S4. Extracting a subset of query target data from a preset database based on user query intent information; S5. Construct a field structure portrait based on the query target data subset, generate a target chart format based on the field structure portrait using a preset language model, and output the target chart parameters; S6. real-time collection of multimodal interaction information, structural correction of target chart parameters based on the multimodal interaction information, and output of a dynamically corrected target chart; S7. Execute trend evolution display on the dynamically modified target chart to obtain an optimized target chart; and send the dynamically optimized target chart to a preset data large screen.

[0019] The user voice data refers to the user voice collected by an audio collection device such as a microphone. At the same time, the user voice data is used to guide the generation of corresponding smart transportation field charts, which include the user's requirements for the charts. In this embodiment, the environmental noise in the user voice data is eliminated through a filtering algorithm to obtain accurate user voice data.

[0020] The ways of segmenting and combining semantic structures include: A dynamic sliding window is set and used to traverse the precise user voice data. The boundaries of voice intervals whose interval time is greater than the dynamic sliding window size are marked. Since users may pause to think when inputting voice, a dynamic sliding window is constructed in this embodiment to traverse the precise user voice data to identify the voice intervals caused by such pauses. The size of the initial dynamic sliding window is first set, and then the window size is adjusted through subsequent operations to ensure that the voice intervals caused by pauses can be accurately identified, avoiding semantic structure division errors caused by fixed window recognition.

[0021] The precise user speech data is structurally divided based on the marked speech intervals, and a set of structured speech segments is output. The speech intervals are used as the boundaries of the structural division, and the precise user language data is divided into several structured speech segments. These structured speech segments are integrated to obtain the set of structured speech segments.

[0022] The two adjacent structural speech segments before and after each structural speech segment are extracted, and the structural speech segment and the adjacent structural speech segments are combined into a context speech segment group, wherein for each structural speech segment, a triple structure is formed by combining its previous segment and the next segment, namely the context speech segment group. If it cannot be divided into three segments, only two segments can also be used for subsequent operations; the context speech segment group includes context information and reflects the word order relationship of the current structural speech segment in its paragraph or sentence, providing basic data for subsequent semantic merging.

[0023] A coherence assessment is performed on each context speech segment group, and a semantic coherence score is output. The coherence assessment is used to detect whether the three speech segments arranged in word order in each context speech segment group are coherent at the semantic level. The obtained semantic coherence score serves as the basis for subsequent merging.

[0024] A coherence score threshold is set, and the semantic coherence score and the coherence score threshold are combined to screen the groups of speech segments that can be merged. In this embodiment, the coherence score threshold includes two thresholds: a merging threshold and an independent threshold. These thresholds are used to determine whether merging is possible. If merging is possible, two sub-thresholds are set based on the existing merging threshold: a forward scoring threshold and a backward scoring threshold. If the semantic coherence score meets the merging threshold requirement, it is determined that the structural speech segments in the corresponding context speech segment group can be merged, indicating that two or three of the structural speech segments are continuous at the semantic level. The pauses are not sentence interruptions by the user, but are simply speech gaps caused by thinking or other insignificant reasons.

[0025] The mergeable speech segment group is merged to output complete user semantic data. The semantic coherence score includes the forward semantic coherence score, the backward semantic coherence score and the semantic coherence score obtained by weighted fusion of the above two scores. In this embodiment, the semantic coherence score is first compared with the merging threshold in the coherence score threshold to determine whether it can be merged. Then, the forward semantic coherence score is compared with the forward score threshold, and the backward semantic coherence score is compared with the backward score threshold. If the forward semantic coherence score meets the forward score threshold, then The structural speech segment is merged with its previous segment; if the backward semantic coherence score meets the backward score threshold, the structural speech segment is merged with its subsequent segment; if the forward semantic coherence score, the backward semantic coherence score and the semantic coherence score all meet the set threshold, the three segments are merged; it should be noted that since the semantic coherence score is obtained by weighted fusion of the forward semantic coherence score and the backward semantic coherence score, there will be no situation where both the forward semantic coherence score and the backward semantic coherence score do not meet the threshold, but the semantic coherence score does.

[0026] The complete user semantic data is structurally divided based on reasonable speech intervals, and a comprehensive coherence score of the complete user semantic data after division is calculated. If the coherence score threshold is not met, the segment merging strength is adjusted; otherwise, no adjustment is made. It should be noted that the complete user semantic data no longer includes unreasonable speech intervals such as pauses for thinking, but only reasonable speech intervals caused by sentence breaks. The complete semantic data is divided into several segments based on reasonable speech intervals. In this embodiment, the pre-trained BERT language processing model is used to evaluate each segment to determine whether the semantics of each segment remains smoothly expressed in the complete semantic data as a whole, and a comprehensive coherence score is output. If the score does not meet the comprehensive score threshold in the coherence score threshold, it indicates that there is an abnormality in the merged semantics. In this case, the forward score threshold and the backward score threshold are adjusted, and the merge simulation is re-performed to make the semantic expression after the merge simulation normal. The adjusted data replaces the complete user semantic data before modification.

[0027] Methods for coherence assessment include:

[0028] Perform speech-to-text recognition on any context speech segment group and output the original recognized text. In this embodiment, ASR automatic speech recognition technology is used to convert the speech content of the context speech segment group into the original recognized text as the data basis for subsequent operations.

[0029] Collect user information, load a local accent reference database based on the user information, convert the original recognition text into standard text based on the local accent reference database, and output standard recognition text. Since users may have dialects or accents in the voice input process due to geographical reasons, which may lead to deviations in text recognition, based on the user information, load a local accent reference database that matches the accent type of the user's region. Use this database to replace the accent expressions that may be misrecognized in the original recognition text with synonyms or standard words to obtain standard recognition text. The user information includes the user's identity information, registration area, GPS positioning, and other data that can identify the user's location.

[0030] Identify the colloquial words in the standard recognition text, perform word form correction on the colloquial words based on the preset professional expression dictionary, and output professional expression pattern text. Among them, common colloquial expression words are extracted from the standard recognition text, and the colloquial expression words are mapped and corrected using the preset professional expression dictionary to ensure that the existing large model can recognize the output professional expression pattern text.

[0031] Identify secondary words in professional expression pattern texts and mark them. Secondary words include modal particles, adverbs, conjunctions, adjectives, etc., which are not the main semantic carriers in the sentence, but play the role of connecting the previous and next sentences, semantic flow, and expression of intention tendency.

[0032] The number of times secondary words switch in the target speech segment text, the speech segment text before the paragraph and the speech segment text after the paragraph in the professional expression pattern text is counted respectively, and the degree of semantic transition is output. The degree of semantic transition refers to the number of times secondary words appear or disappear in each speech segment in the professional expression pattern text. This value reflects whether there is a sudden change in syntactic logic or semantic emotion between each speech segment.

[0033] If the degree of semantic transition is higher than the preset threshold, it is determined to be a broken structure and the fracture weight is output. Otherwise, it is determined to be a coherent structure and the coherence weight is output. When the degree of semantic transition is higher than the preset threshold, it means that the corresponding speech segment has no semantic connection with the previous or next text, and a semantic jump occurs, which is a broken structure. At the same time, the fracture weight is a negative value, which is used to reflect the risk of subsequent merging of the speech segment. The smaller the value, the greater the merging risk and the lower the semantic coherence. At the same time, the coherence weight is a positive value. The larger the value, the smaller the merging risk and the higher the semantic coherence. In this embodiment, the values ​​of the fracture weight and the coherence weight are set based on the specific semantic situation. Based on the experience of those skilled in the art, the initial value of the fracture weight can be set to , set the initial value of the coherence weight to , and subsequently make dynamic adjustments based on specific circumstances.

[0034] Semantic vector modeling is performed on the professional expression pattern text to obtain the target speech segment semantic vector, pre-segment semantic vector and post-segment semantic vector respectively. In this embodiment, the three speech segment texts corresponding to the professional expression pattern text are converted into vector representation using a pre-trained natural language processing model to obtain the semantic vector of the corresponding speech segment.

[0035] Calculate the semantic similarity between the target speech segment semantic vector and the pre-segment semantic vector, and output the forward semantic coherence score. Calculate the semantic similarity between the target speech segment semantic vector and the post-segment semantic vector, and output the backward semantic coherence score. The calculation formula for semantic similarity is: ;in and Represents a semantic vector respectively, when When is the semantic vector of the target speech segment, if is the pre-segment semantic vector, then the output The semantic vector of the target speech segment The forward semantic coherence score of is the pre-segment semantic vector, then the output The semantic vector of the target speech segment If the target speech segment semantic vector does not have a pre-segment semantic vector or a post-segment semantic vector, then the other vector in the formula other than the target speech segment semantic vector is set to .

[0036] The forward semantic coherence score and the backward semantic coherence score are weightedly fused based on the fracture weight and the coherence weight to output the semantic coherence score. It should be noted that when the speech segment for which the semantic coherence score needs to be calculated is a fractured structure, the fracture weight is used to weightedly fused the forward semantic coherence score and the backward semantic coherence score to output the semantic coherence score of the fractured structure at this time; when the speech segment for which the semantic coherence score needs to be calculated is a coherent structure, the coherence weight is used to weightedly fused the forward semantic coherence score and the backward semantic coherence score to output the semantic coherence score of the coherent structure at this time.

[0037] Methods for filtering mergeable voice segment groups include: The coherence score threshold includes a merging threshold and an independent threshold. When the semantic coherence score is higher than or equal to the merging threshold, the corresponding context speech segment group is determined to be a mergeable speech segment group. The merging threshold has a larger value and the independent threshold has a smaller value, which can be specifically set based on expert experience or existing data. When the semantic coherence score is higher than or equal to the merging threshold, it indicates that the context speech segment group is semantically logically continuous, and it is determined that the three structural speech segments can be merged forward, backward, or as a whole, which is a mergeable speech segment group.

[0038] When the semantic coherence score is lower than the independence threshold, the structural speech segments corresponding to the context speech segment group are judged as independent semantic structures and do not participate in the merging. When the semantic coherence score is lower than the independence threshold, it means that there are inconsistencies or semantic jumps in the semantic logic between the structural speech segments in the context speech segment group. This may be because the user did make natural pauses or punctuation in them, so different structural speech segments have different semantics and cannot be merged.

[0039] When the semantic coherence score is higher than the independent threshold and lower than the merge threshold, if the semantic coherence score is close to the merge threshold, the speech interval of the corresponding context speech segment group is adjusted and the coherence evaluation is re-performed until the semantic coherence score is higher than or equal to the merge threshold. The semantic coherence score is higher than the independent threshold and lower than the merge threshold, which indicates that the context speech segment group is in a fuzzy interval at the semantic level. Therefore, two sub-thresholds are set: a merge sub-threshold close to the merge threshold and an independent sub-threshold close to the independent threshold. If the semantic coherence score is lower than the merge threshold but greater than or equal to the merge sub-threshold, it indicates that the context speech segment group has certain semantic continuity potential. Therefore, the speech interval is re-identified by adjusting the dynamic sliding window size and the coherence evaluation is re-performed. The process of adjusting the dynamic sliding window size and coherence evaluation is repeated until the output semantic coherence score is higher than or equal to the merge threshold, thereby maximally restoring the user's actual complete semantics in the original speech.

[0040] If the semantic coherence score is close to the independence threshold, the semantic interval adjustment is first performed and the coherence evaluation is performed. If the difference between the adjusted output semantic coherence score and the merging threshold is greater than the preset distance threshold, the structural speech segment corresponding to the context speech segment is determined to be a separate semantic structure. When the semantic coherence score is in the range from the independence threshold to the independent sub-threshold, the dynamic sliding window size adjustment and coherence evaluation operation are performed once, and the absolute value of the difference between the adjusted output coherence score and the merging threshold is calculated. If this value is greater than the preset distance threshold, it means that the context speech segment group no longer has the potential to be merged, and therefore it is directly determined to be a separate semantic structure.

[0041] Ways to perform semantic parsing include: Extract the phrase and sentence structure in the complete user semantic data, use the preset traffic terminology dictionary to match the phrase and sentence structure, screen the phrase or sentence structure containing traffic terminology, and obtain independent traffic terminology entries, where the phrase and sentence structure refers to the structure that clearly constitutes nouns or complete sentences in addition to secondary words in the complete user semantic data. These structures are the main semantic carriers in the complete user semantic data; it should be noted that the traffic terminology dictionary is a dictionary constructed based on existing smart transportation field data, which includes commonly used traffic terms such as "traffic flow", "congestion index" and road section names. Independent traffic terminology entries are obtained by matching the phrase and sentence structure with the preset traffic terminology dictionary.

[0042] Detect whether the independent traffic term entry includes conjunctions, and classify the independent traffic term entry based on the conjunctions, including conjunction compound phrases and non-conjunction compound phrases. Since the user may produce some self-made compound traffic phrases composed of several existing traffic terms when performing voice output, the previous rule for matching using the traffic term dictionary is to complete the match when a field related to the traffic term appears. Therefore, the output independent traffic term entry may include multiple traffic terms. Therefore, in this embodiment, the independent traffic term entry is classified into conjunction compound phrases and non-conjunction compound phrases based on conjunctions; wherein the conjunction compound phrase refers to a phrase or short sentence structure in which two traffic terms are connected by a conjunction, and the non-conjunction compound phrase refers to a phrase or short sentence structure composed of several fields extracted from several traffic terms that can express the general meaning, such as "traffic flow efficiency", which is a combination of "traffic volume" and "traffic efficiency".

[0043] Using conjunctions as decomposition markers, traffic terminology phrases adjacent to the decomposition markers in the connective compound phrases are extracted, wherein the conjunctions are used as semantic cutting markers to separate the traffic terminology phrases adjacent to the conjunctions from the conjunctions; the traffic terminology phrases are sorted according to the original word order and semantically concatenated to output a class of compound semantic fragments, which are sorted based on the order in which the original traffic terminology phrases appear, and the ordered traffic terminology phrase sequence after decomposition is semantically concatenated based on syntactic dependency logic, which is a class of compound semantic fragments; this type of semantic fragment ensures consistency with the original semantic theme and reasonable sentence word order.

[0044] The non-connected compound phrases are decomposed into word units to obtain preliminary split subsequences. In this embodiment, the non-connected compound phrases are decomposed into word units using a semantic dependency model, and the non-connected compound phrases are decomposed into separate sub-units, which are combined into preliminary split subsequences. A term-word relationship graph is constructed based on the preliminary split subsequences, and any word unit is used as a node. The semantic relevance between the word units is calculated and used as an edge. By converting the preliminary split subsequences into a graph structure, the aggregation degree between the word units is presented. The semantic relevance is obtained by converting the word units into word vectors, and the cosine similarity between the word vectors is calculated to obtain the semantic relevance and used as an edge.

[0045] The term-lemma relationship graph is clustered by nearest neighbor clustering to obtain the shortest relationship path for each lemma. It should be noted that for the relationship graph composed of lemmas of a single non-connected compound phrase, the number of lemma nodes included therein is relatively limited. In order to optimize the semantic dispersion, this embodiment selects a nearest neighbor clustering method based on semantic relevance to obtain a semantic path composed of lemmas with the closest semantic relationship within a preset threshold whose semantic relevance is within a preset threshold, which is the shortest relationship path.

[0046] The shortest relationship path is reasonably screened and a reasonable relationship path is output. The screening requirement for reasonable screening is that the word units that need to constitute the shortest relationship path are appropriately positioned and semantically fluent, and the connection between the word units can reflect the complete semantics; the word units in the reasonable relationship path are semantically concatenated according to the original word order to obtain two types of compound semantic fragments, and the word units in the screened path are semantically concatenated according to the original word order according to the syntactic dependency logic, so that the obtained two types of compound semantic fragments can clearly express the overall semantics and reflect the user's intention; the first type of compound semantic fragments and the second type of compound semantic fragments are integrated to obtain the user's query intention information.

[0047] The ways to construct field structure portraits include: Extract all relevant fields in the query target data subset, classify the fields based on the sources of the relevant fields, and use the traffic business category to which the fields belong to mark the field type, wherein the query target data subset refers to the specific values ​​or information of the relevant data that matches the user query intention information obtained from the preset database based on the user query intention information in step S4, that is, the relevant fields; wherein the source of the relevant field refers to the data source of the required information in the preset database, and the fields are classified based on the traffic business categories corresponding to these data sources, such as multiple traffic business categories such as "traffic flow statistics" and "congestion trends".

[0048] The basic attribute indicators of the target data subset of the statistical query are extracted, and the target mathematical distribution characteristics are output. The basic attribute indicators refer to the basic statistical characteristics of the field value range, mean and variance, etc. used to express the field value in the overall query target data subset. At the same time, based on these basic statistical characteristics, mathematical methods are used to obtain the mathematical distribution characteristics corresponding to the basic attribute indicators, such as normal distribution and discrete distribution, which are the target mathematical distribution characteristics.

[0049] The time range corresponding to each relevant field is detected, and the trend change analysis is performed on the corresponding numerical values ​​of the relevant fields within the time range to obtain the trend change characteristics of the relevant fields, where the time range refers to the time period covered by the field value of the relevant field in the query target data subset. In this embodiment, a regression algorithm is used to perform trend analysis on the field value of the relevant field within the time range, and the trend change characteristics obtained include characteristics with time evolution structure characteristics such as change direction, period and mutation point.

[0050] Compare the value of the relevant field with the preset safety value range. If it exceeds the preset safety value range, it is determined to be an alarm field and an alarm mark is added. The preset safety value range refers to the reasonable value fluctuation range in the traffic business field to which the relevant field belongs. For example, the traffic flow does not exceed ; By marking the fields that exceed the preset safety value range with alarms, it is convenient to construct a more complete field structure portrait in the future; match the field type with the target mathematical distribution characteristics, trend change characteristics and alarm marks respectively, and output the field structure portrait of each relevant field. The field structure portrait refers to the data collection of the statistical characteristics, change trends and alarm information of the relevant fields that match the corresponding traffic business category, and provides a reference basis for subsequent chart matching.

[0051] The methods for generating the target chart format include:

[0052] Obtain the speech text corresponding to the user query intention information, use the pre-trained corpus model to perform semantic extraction on the speech text, and output the user intention vector. In this embodiment, the natural language processing model trained with traffic field corpus is used to extract the word vector combination and its semantic context dependency features in the speech text, and encode them into vector form to obtain the user intention vector.

[0053] The user semantic tendency is judged based on the user intention vector. In this embodiment, keywords are mapped to tendency labels. For example, "trend, change, increase, decrease" can be mapped to trend analysis tendency, and "current, now" can be mapped to real-time status tendency; the user semantic tendency refers to the semantic information that the user needs or pays more attention to. With the user intention vector as the data basis, the tendency label corresponding to the keyword encoding is identified, and the attention mechanism is used to determine which tendency label has the greatest influence. The semantic information of the tendency label is then used as the user semantic tendency.

[0054] The field structure portrait and the corresponding user semantic tendency are combined to generate demand target information, wherein the field structure portrait and the user semantic tendency are jointly modeled to generate structured demand target information; the demand target information includes data such as field type, time range, trend change pattern and alarm status, and this information is used as the basis for subsequent matching with the chart.

[0055] Taking the target information as the data basis, the preset language model is used to construct the target chart format, calculate the matching fitness, and screen the candidate charts based on the matching fitness. The calculation formula for matching fitness is: In this embodiment, a large language model for chart generation is provided. The matching fitness formula is used to calculate the matching fitness between the generated chart format and the target information, and based on the matching fitness, it is determined whether the currently generated chart format meets the requirements. In the calculation formula, Indicates the matching fitness between the target information and any generated graph; represents the semantic matching weight, which is obtained by calculating the cosine similarity between the demand vector corresponding to the demand target information and the label vector of the generated graph; Indicates the field structure matching weight. This weight refers to the degree of match between a field (e.g., continuous, discrete, temporal, or spatial structure) and the generated chart function. This value is dynamically set based on historical experience. Indicates the field quantity matching weight, which is used to reflect whether the number of fields corresponding to the target information is within the range of the number of fields supported by the generated chart. When the number of fields corresponding to the target information exceeds the range of the number of fields supported by the generated chart, the value is Otherwise, the ratio of the number of fields corresponding to the target information to the number of fields supported by the generated chart is calculated as value; set the matching fitness threshold and select the generated graphs with matching fitness higher than the threshold as candidate graphs.

[0056] The expressiveness of candidate charts is evaluated, and target chart parameters are output based on the evaluation results. In this embodiment, a simulation environment is constructed, in which all fields corresponding to the required target information are merged with each candidate chart to simulate the generated chart effect. At the same time, the expressiveness of candidate charts is evaluated from three aspects: the degree of information display adequacy, visual complexity, and interactive performance. Based on the existing evaluation criteria, the chart with the best expressiveness is selected, and the target chart parameters of the chart are output.

[0057] Ways to make structural corrections include: A corresponding chart is constructed based on the target chart parameters; after each interaction behavior is completed, the interaction modal type and interaction area of ​​the multimodal interaction information are extracted to construct interaction association information, where the interaction modal type includes interaction behaviors such as clicking, dragging, and zooming, and the interaction area refers to the area on the chart where each interaction behavior is affected; the interaction modal type and interaction area are combined to form interaction association information, which is a data structure including the above two types of data.

[0058] Based on the interaction correlation information, interaction behaviors are divided into simple interaction behaviors and semantic interaction behaviors. Simple interaction behaviors refer to basic interaction operations such as clicking and dragging, and semantic interaction behaviors refer to interaction operations involving voice commands, such as the user's request to "only watch peak hours."

[0059] When the interactive behavior is a simple interactive behavior, the corresponding functional architecture of the corresponding chart is called to adaptively adjust the internal parameters of the structure. It should be noted that when the interactive behavior is a simple interactive behavior, it is only necessary to adjust the parameters in the interactive control structure of the corresponding area in the chart to implement simple interactive actions such as zooming in, sliding and zooming out, thereby realizing lightweight state updates of the chart.

[0060] When the interactive behavior is a semantic interactive behavior, the semantic target of the semantic interactive behavior is extracted, and the preset language model is used to regenerate a suitable chart format based on the semantic target, and the target chart parameters are reconfigured. By performing semantic analysis on the speech corresponding to the semantic interactive behavior, the user intention is determined, and the semantic target is obtained. Based on the semantic target, the preset language model is called to generate a chart format that is more suitable for the current user intention. At the same time, the fields that meet the user intention are obtained and the parameters of the new chart are reconfigured.

[0061] The chart adjusted based on the interactive behavior is output as a dynamically corrected target chart. It should be noted that what is described here is a method for switching the chart format generated by a large model, which is only used in the case where the user changes the requirements after the first chart is generated during the speech recognition process. Therefore, before outputting the dynamically corrected target chart, the previous chart with adaptive parameter adjustment has been sent to the preset data large screen for display. It is only used here to update the chart format.

[0062] Ways to perform trend evolution display include: Construct an adjustable timeline. In this embodiment, the adjustable timeline constructed is a timeline that can dynamically adjust the time range based on user requirements; bind all fields in the dynamically corrected target chart to the corresponding timestamps, wherein the fields are bound to the timestamps to establish a mapping relationship between the fields and the timestamps, so as to facilitate the subsequent synchronous adjustment of the field values ​​by adjusting the timeline.

[0063] By adjusting the adjustable time axis, the value of each field in the dynamically corrected target chart is switched frame by frame based on the timestamp to obtain an optimized target chart. In this embodiment, when the user operates the adjustable time axis, the system obtains the field value corresponding to the next timestamp based on the mapping relationship based on the timestamp corresponding to the current field value, and synchronously updates the rendering content of the chart, thereby realizing frame-by-frame switching of the chart as the time axis changes, further improving the observability of the trend changes of the chart; in addition, the adjustable time axis in this embodiment is bidirectionally adjustable, so the optimized target chart finally obtained is a dynamic chart covering the entire process of time evolution.

[0064] This embodiment first performs structural processing and semantic analysis on the collected user voice data, extracts user query intention information, generates a suitable chart format based on the user query intention information, and optimizes the interaction in the chart, thereby realizing a dynamic generation method of intelligent charts based on voice recognition and multimodal interaction. Compared with existing experience, multi-level analysis is performed on user voice, which significantly improves the accuracy of user intention recognition in the case of voice input, especially in dealing with colloquial expressions, sentence pauses, and accent or dialect interference, thereby improving the robustness of voice recognition. At the same time, the semantic parsing operation is deeply integrated with traffic business knowledge to generate field structure portraits, which provides a basis for generating charts using large language models. Further, by combining chart matching adaptability and expression ability, accurate matching of user semantics and chart types is achieved. By capturing user interaction actions, adaptive adjustment of chart structure parameters and chart formats is achieved, meeting user interaction needs and improving user experience.

[0065] Example 2 See also Figure 2As shown, for parts not described in detail in this embodiment, please refer to the description of Example 1. A system for dynamically generating intelligent charts based on speech recognition and multimodal interaction is provided, including: The data acquisition module is used to collect user voice data and perform voice data denoising to obtain accurate user voice data; Semantic segmentation module, used to segment and combine the precise user voice data into semantic structures and output complete user semantic data; Semantic parsing module, used to perform semantic parsing on complete user semantic data and output user query intention information; A data query module is used to extract a query target data subset from a preset database based on user query intention information; A chart generation module is used to construct a field structure portrait based on the query target data subset, generate a target chart format based on the field structure portrait using a preset language model, and output target chart parameters; The interactive optimization module is used to collect multimodal interaction information in real time, perform structural correction on the target chart parameters based on the multimodal interaction information, and output the corrected target chart; The chart visualization module is used to perform trend evolution display on the corrected target chart to obtain a dynamically optimized target chart; send the dynamically optimized target chart to a preset data large screen; and connect each module via wired and / or wireless means.

[0066] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art will be able to modify the technical solutions described in the foregoing embodiments or to substitute equivalents for some of the technical features. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

[0067] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0068] In the description of the present invention, it should be understood that the terms "first", "second", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0069] In the description of the present invention, unless otherwise specified, "plurality" means two or more.

[0070] In the description of the present invention, “several” means one or more, and “a large number” means two or more.

[0071] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0072] The formulas in this manual are all dimensionless and calculated using numerical values. The formulas are obtained by collecting a large amount of data and performing software simulation to obtain the most recent real situation. The preset parameters and thresholds in the formulas are set by technicians in this field based on actual conditions.

[0073] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. A method for dynamically generating intelligent charts based on speech recognition and multimodal interaction, characterized in that: include: S1. Collect user voice data and perform voice data denoising to obtain accurate user voice data; S2. Perform semantic structure segmentation and combination on the precise user voice data to output complete user semantic data; S3. Perform semantic analysis on the complete user semantic data and output user query intent information; S4. Extracting a subset of query target data from a preset database based on user query intent information; S5. Construct a field structure portrait based on the query target data subset, generate a target chart format based on the field structure portrait using a preset language model, and output the target chart parameters; S6. real-time collection of multimodal interaction information, structural correction of target chart parameters based on the multimodal interaction information, and output of a dynamically corrected target chart; S7. Execute trend evolution display on the dynamically modified target chart to obtain an optimized target chart; Send the dynamic optimization target chart to the preset data screen.

2. The method for dynamically generating intelligent charts based on speech recognition and multimodal interaction according to claim 1, characterized in that: The semantic structure segmentation and combination methods include: A dynamic sliding window is set and used to traverse the precise user speech data, and speech intervals whose interval time is greater than the dynamic sliding window size are marked as boundaries; the precise user speech data is structurally divided based on the marked speech intervals, and a set of structured speech segments is output; the two adjacent structured speech segments before and after each structured speech segment are extracted, and the structured speech segment and the adjacent structured speech segments are combined into a context speech segment group; Perform a coherence assessment on each context speech segment group and output a semantic coherence score; set a coherence score threshold, and combine the semantic coherence score and the coherence score threshold to screen the mergeable speech segment groups; merge the mergeable speech segment groups into segments and output complete user semantic data; structurally divide the complete user semantic data based on reasonable speech intervals, and calculate the comprehensive coherence score of the complete user semantic data after division. If it does not meet the coherence score threshold, adjust the segment merging strength; otherwise, do not adjust it.

3. The method for dynamically generating intelligent charts based on speech recognition and multimodal interaction according to claim 2, characterized in that: The methods of coherence assessment include: Perform speech-to-text recognition on any contextual speech segment group and output the original recognized text; collect user information, load the local accent reference database based on the user information, convert the original recognized text to standard text based on the local accent reference database, and output the standard recognized text; identify colloquial words in the standard recognized text, perform word form correction on the colloquial words based on a preset professional expression dictionary, and output the professional expression model text; Identify and label sub-words in professional expression pattern texts; count the number of sub-word switches in the target speech segment, the preceding speech segment, and the following speech segment in the professional expression pattern texts, and output the degree of semantic transition; if the degree of semantic transition is higher than a preset threshold, it is determined to be a broken structure and the fracture weight is output; otherwise, it is determined to be a coherent structure and the coherence weight is output; Semantic vector modeling is performed on professional expression pattern text to obtain the target speech segment semantic vector, pre-segment semantic vector and post-segment semantic vector respectively; the semantic similarity between the target speech segment semantic vector and the pre-segment semantic vector is calculated, and the forward semantic coherence score is output; the semantic similarity between the target speech segment semantic vector and the post-segment semantic vector is calculated, and the backward semantic coherence score is output; the forward semantic coherence score and the backward semantic coherence score are weightedly fused based on the break weight and coherence weight, and the semantic coherence score is output.

4. The method for dynamically generating intelligent charts based on speech recognition and multimodal interaction according to claim 3, characterized in that: The method of screening the mergeable speech segment groups includes: The coherence score threshold includes a merging threshold and an independent threshold. When the semantic coherence score is higher than or equal to the merging threshold, the corresponding context speech segment group is judged as a mergible speech segment group; when the semantic coherence score is lower than the independent threshold, the structural speech segments corresponding to the context speech segment group are respectively judged as independent semantic structures and do not participate in the merging; when the semantic coherence score is higher than the independent threshold and lower than the merging threshold, if the semantic coherence score is close to the merging threshold, the speech interval of the corresponding context speech segment group is adjusted and the coherence evaluation is re-performed until the semantic coherence score is higher than or equal to the merging threshold; if the semantic coherence score is close to the independent threshold, the semantic interval adjustment is performed first and the coherence evaluation is performed. If the difference between the adjusted semantic coherence score and the merging threshold is greater than the preset distance threshold, the structural speech segment corresponding to the context speech segment is judged as a independent semantic structure.

5. The method for dynamically generating intelligent charts based on speech recognition and multimodal interaction according to claim 4, characterized in that: The method of performing semantic parsing includes: Extract phrase and sentence structures from complete user semantic data, match them with the preset traffic term dictionary, filter phrases or sentence structures containing traffic terms, and obtain independent traffic term entries; detect whether the independent traffic term entries contain conjunctions, and classify the independent traffic term entries based on the conjunctions, including conjunction compound phrases and non-conjunction compound phrases; Using conjunctions as decomposition markers, the traffic term phrases adjacent to the decomposition markers in the connected compound phrases are extracted; the traffic term phrases are sorted according to the original word order and semantically concatenated to output a class of compound semantic fragments; The non-connected compound phrases are decomposed into lemmas to obtain preliminary split subsequences; a term-lemma relationship graph is constructed based on the preliminary split subsequences, with any lemma as a node, and the semantic relevance between lemmas is calculated and used as edges; the term-lemma relationship graph is clustered by neighbors to obtain the shortest relationship path for each lemma; the shortest relationship paths are reasonably screened and reasonable relationship paths are output; the lemmas in the reasonable relationship paths are semantically concatenated according to the original word order to obtain two types of compound semantic fragments; the first type of compound semantic fragments and the second type of compound semantic fragments are integrated to obtain user query intention information.

6. The method for dynamically generating intelligent charts based on speech recognition and multimodal interaction according to claim 5, characterized in that: The method of constructing the field structure portrait includes: Extract all relevant fields in the query target data subset, classify the fields based on the source of the relevant fields, and use the traffic business category to which the fields belong to mark the field type; count the basic attribute indicators of the query target data subset, extract the features of the basic attribute indicators, and output the target mathematical distribution characteristics; detect the time range corresponding to each relevant field, and perform trend change analysis on the corresponding values ​​of the relevant fields within the time range to obtain the trend change characteristics of the relevant fields; compare the values ​​of the relevant fields with the preset safety value range, and if they exceed the preset safety value range, determine them as alarm fields, and add alarm marks; match the field type with the target mathematical distribution characteristics, trend change characteristics and alarm marks respectively, and output the field structure portrait of each relevant field.

7. The method for dynamically generating intelligent charts based on speech recognition and multimodal interaction according to claim 6, characterized in that: The method of generating the target chart format includes: Obtain the speech text corresponding to the user's query intention information, use the pre-trained corpus model to perform semantic extraction on the speech text, and output the user intention vector; judge the user's semantic tendency based on the user intention vector; combine the field structure portrait with the corresponding user semantic tendency to generate demand target information; use the demand target information as the data basis, use the preset language model to construct the target chart format, calculate the matching fitness, and screen candidate charts based on the matching fitness; evaluate the expressive ability of the candidate charts, and output the target chart parameters based on the evaluation results.

8. The method for dynamically generating intelligent charts based on speech recognition and multimodal interaction according to claim 7, characterized in that: The method of performing structural correction includes: Construct a corresponding graph based on the target graph parameters; extract the interaction mode type and interaction area of ​​the multimodal interaction information after each interaction behavior is completed, and construct interaction association information; based on the interaction association information, divide the interaction behavior into simple interaction behavior and semantic interaction behavior; When the interactive behavior is a simple interactive behavior, the corresponding functional architecture of the corresponding chart is called to adaptively adjust the internal parameters of the structure; when the interactive behavior is a semantic interactive behavior, the semantic target of the semantic interactive behavior is extracted, and the preset language model is used to regenerate the appropriate chart format based on the semantic target, and the target chart parameters are reconfigured; the chart adjusted based on the interactive behavior is output as a dynamically corrected target chart.

9. The method for dynamically generating intelligent charts based on speech recognition and multimodal interaction according to claim 8, characterized in that: The method of performing trend evolution display includes: Construct an adjustable timeline; bind all fields in the dynamically corrected target chart to corresponding timestamps; and switch the value of each field in the dynamically corrected target chart frame by frame based on the timestamp by adjusting the adjustable timeline to obtain an optimized target chart.

10. An intelligent chart dynamic generation system based on speech recognition and multimodal interaction, which is used to implement the intelligent chart dynamic generation method based on speech recognition and multimodal interaction according to any one of claims 1 to 9, characterized in that: include: The data acquisition module is used to collect user voice data and perform voice data denoising to obtain accurate user voice data; Semantic segmentation module, used to segment and combine the precise user voice data into semantic structures and output complete user semantic data; Semantic parsing module, used to perform semantic parsing on complete user semantic data and output user query intention information; A data query module is used to extract a query target data subset from a preset database based on user query intention information; A chart generation module is used to construct a field structure portrait based on the query target data subset, generate a target chart format based on the field structure portrait using a preset language model, and output target chart parameters; The interactive optimization module is used to collect multimodal interaction information in real time, perform structural correction on the target chart parameters based on the multimodal interaction information, and output the corrected target chart; A chart visualization module is used to display the trend evolution of the modified target chart to obtain a dynamically optimized target chart; The dynamic optimization target chart is sent to the preset data screen; each module is connected via wired and / or wireless means.

Images