Intelligent chart dynamic generation method based on voice recognition and multi-modal interaction

Abstract

The application belongs to the technical field of speech recognition, and discloses an intelligent chart dynamic generation method based on speech recognition and multi-modal interaction; comprising: collecting user speech data and denoising to obtain accurate user speech data; performing semantic structure segmentation and combination to output complete user semantic data; executing semantic analysis and outputting user query intent information; extracting a query target data subset; constructing a field structure portrait based on the query target data subset and generating a target chart format to output target chart parameters; based on real-time collected multi-modal interaction information, correcting the structure of the target chart parameters to output a dynamically corrected target chart; performing trend evolution display on the dynamically corrected target chart to obtain an optimized target chart and send the dynamically optimized target chart to a preset data large screen; the speech recognition accuracy and robustness are improved, the chart interaction function is optimized, the user's interaction demand is met, and the user's interaction experience is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech recognition, more particularly, the present application relates to an intelligent chart dynamic generation method based on speech recognition and multi-modal interaction. BACKGROUND

[0002] With the rapid development of natural language processing technology, speech interaction technology has been widely used in smart cities, traffic management and data visualization and many other fields; among them, speech-driven data chart generation technology has gradually become an important part of intelligent transportation and visual analysis systems; especially in interactive display terminals, users often prefer to generate the required data chart through this simple method of voice oral expression, which reduces the burden of manual operation while improving query efficiency; but the existing technology still has some challenges in speech accurate recognition and chart generation.

[0003] First of all, in the existing field of speech recognition technology, when users face high-complexity business scenarios of intelligent transportation, there may be speech pauses in the process of voice input due to thinking, hesitation or supplementary explanation, etc. The existing speech recognition technology often determines that this pause is caused by sentence boundaries such as punctuation, so that the recognized speech appears to have a semantic structure fracture phenomenon, resulting in the inability to generate charts that meet user needs in the subsequent chart generation process, which greatly reduces the query efficiency; in addition, due to the user's habits of certain regional accents or colloquial expressions, the result of speech recognition is not accurate enough, which reduces the recognition accuracy; at the same time, the user may spontaneously combine multiple professional terms in the field of intelligent transportation to form a new and unrecorded compound phrase structure. This non-standard expression makes the semantic analysis process more difficult, and the existing technology is difficult to accurately recognize the semantic information of the compound phrase structure, resulting in the inability to understand the user's true intention, and the existing language model is difficult to generate appropriate chart formats based on inaccurate intention information; moreover, the existing technology lacks accurate recognition of user semantic tendencies, which may result in the generation of charts that do not meet the current business scenario; in terms of interaction, the existing technology lacks an adaptive adjustment mechanism and a trend evolution mechanism based on user feedback, resulting in charts with weak expression ability that do not meet user needs.

[0004] In view of this, the present application proposes an intelligent chart dynamic generation method based on speech recognition and multi-modal interaction to solve the above problems. SUMMARY

[0005] In order to overcome the above-mentioned defects of the prior art, in order to achieve the above-mentioned purpose, the present application provides the following technical scheme: an intelligent chart dynamic generation method based on speech recognition and multi-modal interaction, comprising:

[0006] S1. Collect user voice data and perform voice data denoising to obtain accurate user voice data;

[0007] S2. Perform semantic structure segmentation and combination on the accurate user voice data to output complete user semantic data;

[0008] S3. Perform semantic analysis on the complete user semantic data to output user query intent information;

[0009] S4. Extract a query target data subset from a preset database based on the user query intent information;

[0010] S5. Construct a field structure portrait based on the query target data subset, generate a target chart format based on the field structure portrait using a preset language large model, and output target chart parameters;

[0011] S6. Collect multi-modal interaction information in real time, modify the structure of the target chart parameters based on the multi-modal interaction information, and output a dynamically modified target chart;

[0012] S7. Perform trend evolution display on the dynamically modified target chart to obtain an optimized target chart; and send the dynamically optimized target chart to a preset data big screen.

[0013] Further, the semantic structure segmentation and combination method comprises:

[0014] A dynamic sliding window is set and the window is used to traverse the accurate user voice data, and voice intervals with an interval time greater than the size of the dynamic sliding window are marked at the boundaries; the accurate user voice data is divided into structures based on the marked voice intervals, and a set of structured voice segments is output; the two adjacent structured voice segments before and after each structured voice segment are extracted, and the structured voice segment and the adjacent structured voice segment are combined into a context voice segment group;

[0015] The coherence of each context voice segment group is evaluated, and a semantic coherence score is output; a coherence score threshold is set, and the semantic coherence score and the coherence score threshold are combined to screen mergable voice segment groups; the mergable voice segment groups are merged, and complete user semantic data is output; the complete user semantic data is divided into structures based on reasonable voice intervals, the comprehensive coherence score of the divided complete user semantic data is calculated, and if the coherence score threshold is not met, the speech segment merging strength is adjusted, otherwise no adjustment is made.

[0016] Further, the coherence evaluation method comprises:

[0017] The speech text recognition is performed on any one of the context speech segment groups to output original recognition text; user information is collected, a local accent reference database is loaded based on the user information, and standard text conversion is performed on the original recognition text based on the local accent reference database to output standard recognition text; colloquial words in the standard recognition text are recognized, and the colloquial words are morphologically corrected based on a preset professional expression word dictionary to output a professional expression mode text;

[0018] Secondary words in the professional expression mode text are recognized and marked; the number of times of switching of the secondary words in the target speech segment text, the pre-segment speech segment text and the post-segment speech segment text in the professional expression mode text is respectively counted to output a semantic turning degree; if the semantic turning degree is higher than a preset threshold, it is determined as a broken structure, and a broken weight is output, otherwise, it is determined as a coherent structure and a coherent weight is output;

[0019] Semantic vector modeling is performed on the professional expression mode text to obtain a target speech segment semantic vector, a pre-segment semantic vector and a post-segment semantic vector; a forward semantic coherence score is output by calculating the semantic similarity between the target speech segment semantic vector and the pre-segment semantic vector, and a backward semantic coherence score is output by calculating the semantic similarity between the target speech segment semantic vector and the post-segment semantic vector; the forward semantic coherence score and the backward semantic coherence score are weighted and fused based on the broken weight and the coherent weight to output a semantic coherence score.

[0020] Further, the way of screening the mergable speech segment groups comprises:

[0021] The coherence score threshold comprises a merging threshold and an independent threshold, when the semantic coherence score is higher than or equal to the merging threshold, the corresponding context speech segment group is determined as a mergable speech segment group; when the semantic coherence score is lower than the independent threshold, the structure speech segments corresponding to the context speech segment group are determined as separate semantic structures respectively, and do not participate in merging; when the semantic coherence score is higher than the independent threshold and lower than the merging threshold, if the semantic coherence score is close to the merging threshold, the speech interval of the corresponding context speech segment group is adjusted, and the coherence is re-evaluated until the semantic coherence score is higher than or equal to the merging threshold; if the semantic coherence score is close to the independent threshold, the semantic interval is adjusted first and the coherence is evaluated, and if the difference between the output semantic coherence score after adjustment and the merging threshold is greater than a preset distance threshold, the structure speech segments corresponding to the context speech segments are determined as separate semantic structures.

[0022] Further, the way of performing semantic analysis comprises:

[0023] The phrase structure in the complete user semantic data is extracted, the preset traffic terminology dictionary is matched with the phrase structure, the phrase or the phrase structure containing the traffic terminology is screened, and the independent traffic terminology entry is obtained; whether the independent traffic terminology entry includes a conjunction is detected, and the independent traffic terminology entry is classified based on the conjunction, including a connected compound phrase and a non-connected compound phrase;

[0024] The traffic terminology phrase adjacent to the decomposition marker in the connected compound phrase is extracted with the conjunction as the decomposition marker; the traffic terminology phrase is sorted according to the original order and is semantically concatenated, and a first type of compound semantic segment is output;

[0025] The non-connected compound phrase is word element decomposed, and a preliminary split subsequence is obtained; a terminology word element relationship graph is constructed based on the preliminary split subsequence, any word element is taken as a node, the semantic correlation between the word elements is calculated and is taken as an edge; the terminology word element relationship graph is near-neighbor clustered, and the shortest relationship path of each word element is obtained; the shortest relationship path is reasonably screened, and a reasonable relationship path is output; the word elements in the reasonable relationship path are semantically concatenated according to the original order, and a second type of compound semantic segment is obtained; and the first type of compound semantic segment and the second type of compound semantic segment are integrated to obtain user query intention information.

[0026] Further, the way of constructing the field structure image includes:

[0027] All related fields in the query target data subset are extracted, and the related fields are classified based on the field sources; the field types are labeled by using the traffic business categories to which the fields belong; the basic attribute indexes of the query target data subset are counted, and the basic attribute indexes are feature extracted, and target mathematical distribution features are output; the time range corresponding to each related field is detected, and the corresponding numerical values of the related fields in the time range are analyzed for trend change, and the trend change features of the related fields are obtained; the numerical values of the related fields are compared with preset safe numerical value intervals, and if the numerical values exceed the preset safe numerical value intervals, the related fields are determined as alarm fields, and alarm markers are added; the field types are matched with the target mathematical distribution features, the trend change features and the alarm markers respectively, and the field structure images of each related field are output.

[0028] Further, the way of generating the target chart format includes:

[0029] acquire voice text corresponding to user query intention information, perform semantic extraction on the voice text by using a pre-trained corpus model, and output a user intention vector; determine a user semantic tendency based on the user intention vector; generate requirement target information in combination with a field structure image and the corresponding user semantic tendency; use the requirement target information as a data basis, construct a target chart format by using a pre-set language large model, calculate a matching fitness, and filter candidate charts based on the matching fitness; perform expression capability evaluation on the candidate charts, and output target chart parameters based on the evaluation result.

[0030] Further, the structure modification manner comprises:

[0031] Based on the target chart parameters, a corresponding chart is constructed; after each interaction behavior is completed, the interaction type and the interaction area of the multi-modal interaction information are extracted, and interaction association information is constructed; based on the interaction association information, the interaction behavior is divided into simple interaction behavior and semantic interaction behavior;

[0032] When the interaction behavior is simple interaction behavior, the corresponding function architecture of the corresponding chart is called to perform structure internal parameter self-adaptive adjustment; when the interaction behavior is semantic interaction behavior, the semantic target of the semantic interaction behavior is extracted, a suitable chart format is regenerated based on the semantic target by using a pre-set language large model, and the target chart parameters are reconfigured; the chart adjusted based on the interaction behavior is output as a dynamic revision target chart.

[0033] Further, the execution trend evolution display manner comprises:

[0034] A time axis is constructed; all fields in the dynamic revision target chart are bound to corresponding time stamps; by adjusting the time axis, the value of each field in the dynamic revision target chart is switched frame by frame based on the time stamps, and an optimized target chart is obtained.

[0035] An intelligent chart dynamic generation system based on voice recognition and multi-modal interaction, which is used to implement an intelligent chart dynamic generation method based on voice recognition and multi-modal interaction, and characterized in that it comprises:

[0036] A data acquisition module, configured to acquire user voice data and perform voice data denoising to obtain accurate user voice data;

[0037] A semantic segmentation module, configured to perform semantic structure segmentation and combination on the accurate user voice data, and output complete user semantic data;

[0038] A semantic analysis module, configured to perform semantic analysis on the complete user semantic data, and output user query intention information;

[0039] A data query module, configured to extract a query target data subset from a pre-set database based on the user query intention information;

[0040] a chart generation module, configured to construct a field structure portrait based on the query target data subset, generate a target chart format based on the field structure portrait by using a preset language large model, and output target chart parameters;

[0041] an interaction optimization module, configured to collect multi-modal interaction information in real time, perform structural correction on the target chart parameters based on the multi-modal interaction information, and output a corrected target chart;

[0042] a chart visualization module, configured to perform trend evolution display on the corrected target chart to obtain a dynamically optimized target chart, and send the dynamically optimized target chart to a preset data big screen; and the modules are connected through wired and / or wireless modes.

[0043] The intelligent chart dynamic generation method based on voice recognition and multi-modal interaction has the following technical effects and advantages:

[0044] By performing structural processing and semantic analysis on the collected user voice data, the user query intention information is extracted, and a suitable chart format is generated based on the user query intention information, and the interaction in the chart is optimized, thereby realizing the intelligent chart dynamic generation method based on voice recognition and multi-modal interaction. Compared with the existing experience, the user voice is analyzed in multiple levels, which significantly improves the accuracy of user intention recognition under voice input, especially in dealing with oral expression, sentence pause and accent or dialect interference, etc., and improves the robustness of voice recognition. Meanwhile, the semantic analysis operation is deeply integrated with the traffic business knowledge, the field structure portrait is generated to provide a basis for generating the chart by using the language large model. Further, by combining the chart matching fitness and expression ability, the accurate matching of user semantics and chart type is realized. By capturing the user interaction action, the adaptive adjustment of the chart structure parameters and the chart format is realized, the interaction demand of the user is met, and the user experience is improved. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 FIG. 1 is a schematic diagram of the intelligent chart dynamic generation method based on voice recognition and multi-modal interaction of the present application;

[0046] Figure 2 FIG. 2 is a schematic diagram of the intelligent chart dynamic generation system based on voice recognition and multi-modal interaction of the present application. DETAILED DESCRIPTION

[0047] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0048] Embodiment 1

[0049] Please refer to Figure 1 The intelligent chart dynamic generation method based on voice recognition and multi-modal interaction in the embodiment includes:

[0050] S1. Collect user voice data and perform voice data denoising to obtain accurate user voice data;

[0051] S2. Perform semantic structure segmentation and combination on the accurate user voice data to output complete user semantic data;

[0052] S3. Perform semantic analysis on the complete user semantic data to output user query intent information;

[0053] S4. Extract a query target data subset from a preset database based on the user query intent information;

[0054] S5. Construct a field structure portrait based on the query target data subset, generate a target chart format based on the field structure portrait using a preset language large model, and output target chart parameters;

[0055] S6. Real-time collection of multi-modal interaction information, structural correction of target chart parameters based on multi-modal interaction information, and output of dynamically corrected target chart;

[0056] S7. Perform trend evolution display on the dynamically corrected target chart to obtain an optimized target chart; and send the dynamically optimized target chart to a preset data big screen.

[0057] The user voice data is collected by an audio collection device such as a microphone, and is used to guide the generation of a corresponding intelligent transportation field chart, which includes the user's demand for the chart. In this embodiment, the environmental noise in the user voice data is eliminated by a filtering algorithm to obtain accurate user voice data.

[0058] The semantic structure segmentation and combination method includes:

[0059] The dynamic sliding window is set and the accurate user voice data is traversed by using the window, and the voice intervals with interval time greater than the size of the dynamic sliding window are marked as boundaries. Since the user may pause for thinking when inputting voice, a dynamic sliding window is constructed in the embodiment to traverse the accurate user voice data to identify the voice intervals caused by the pause. The size of the initial dynamic sliding window is set, and the window size is adjusted through subsequent operations to ensure that the voice intervals caused by the pause can be accurately identified, and the semantic structure division error caused by the fixed window identification is avoided.

[0060] The accurate user voice data is divided into structures based on the marked voice intervals, and a structure voice segment set is output. The voice intervals are used as the boundaries of the structure division, and the accurate user voice data is divided into a plurality of structure voice segments. The structure voice segment set is obtained by integrating the structure voice segments.

[0061] The two adjacent structure voice segments before and after each structure voice segment are extracted, and the structure voice segment and the adjacent structure voice segment are combined into a context voice segment group. For each structure voice segment, a three-tuple structure is formed by combining the previous segment and the next segment, which is the context voice segment group. If it cannot be divided into three segments, only two segments can also be operated subsequently. The context voice segment group includes context information and reflects the order relationship of the current structure voice segment in the segment or sentence, which provides basic data for subsequent semantic merging.

[0062] The coherence of each context voice segment group is evaluated, and a semantic coherence score is output. The coherence evaluation is used to detect whether the three voice segments arranged in order in each context voice segment group are coherent at the semantic level. The semantic coherence score is used as the basis for subsequent merging.

[0063] A coherence score threshold is set, and the semantic coherence score and the coherence score threshold are combined to screen the mergable voice segment group. In the embodiment, the coherence score threshold includes two thresholds, namely the merging threshold and the independent threshold, which are used to determine whether the voice segment group can be merged. If the voice segment group can be merged, two sub-thresholds, namely the forward score threshold and the backward score threshold, are set based on the merging threshold. If the semantic coherence score meets the merging threshold requirement, it is determined that the structure voice segments in the corresponding context voice segment group can be merged, which means that two or three structure voice segments are continuous at the semantic level. The pause is not caused by the user's punctuation, but only by thinking or other reasons that do not affect the voice interval.

[0064] The speech segment merging is performed on the mergable speech segment groups, and complete user semantic data is output. The semantic coherence score includes a forward semantic coherence score, a backward semantic coherence score, and a semantic coherence score obtained by weighted fusion of the two scores. In this embodiment, the semantic coherence score is first compared with a merging threshold in the coherence score threshold, to determine whether merging is possible. Then, the forward semantic coherence score is compared with a forward score threshold, and the backward semantic coherence score is compared with a backward score threshold. If the forward semantic coherence score meets the forward score threshold, the structural speech segment is merged with the previous segment. If the backward semantic coherence score meets the backward score threshold, the structural speech segment is merged with the next segment. If the forward semantic coherence score, the backward semantic coherence score, and the semantic coherence score all meet the set threshold, the three segments are merged. It should be noted that, since the semantic coherence score is obtained by weighted fusion of the forward semantic coherence score and the backward semantic coherence score, the situation that the forward semantic coherence score and the backward semantic coherence score both do not meet the threshold, but the semantic coherence score meets the threshold, will not occur.

[0065] The complete user semantic data is divided into segments based on reasonable speech intervals, and a comprehensive coherence score of the divided complete user semantic data is calculated. If the coherence score threshold is not met, the speech segment merging strength is adjusted, otherwise it is not adjusted. It should be noted that the complete user semantic data does not include unreasonable speech intervals such as thinking pauses. At this time, only reasonable speech intervals generated by punctuation are present. The complete semantic data is divided into segments based on reasonable speech intervals. In this embodiment, each segment is evaluated using a pre-trained BERT language processing model to determine whether the semantic of each segment is expressed smoothly in the complete semantic data as a whole, and a comprehensive coherence score is output. If the score does not meet the comprehensive score threshold in the coherence score threshold, it means that the merged semantic is abnormal. At this time, the forward score threshold and the backward score threshold are adjusted, and the merging simulation is performed again to make the semantic expression normal after merging simulation. The adjusted data replaces the complete user semantic data before modification.

[0066] The coherence evaluation method includes:

[0067] The speech text recognition is performed on any one context speech segment group, and the original recognition text is output. In this embodiment, the speech content of the context speech segment group is converted into the original recognition text by using the ASR automatic speech recognition technology, serving as the data basis for subsequent operations.

[0068] Collecting user information, loading a local accent reference database based on the user information, converting the original recognition text into standard text based on the local accent reference database, and outputting the standard recognition text. Since the user may have a dialect or accent in the speech input process due to regional relations, which may cause the text recognition to deviate, the user information is used as the basis to load the local accent reference database that matches the user's regional accent type. The database is used to replace the possible misrecognized accent expression words in the original recognition text with synonyms or standard words to obtain the standard recognition text. The user information includes user identity information, registered region, GPS positioning, and other data that can identify the user's region.

[0069] Identifying colloquial words in the standard recognition text, and correcting the word form of the colloquial words based on a preset professional expression word dictionary to output a professional expression mode text. The common colloquial expression words are extracted from the standard recognition text, and the preset professional expression dictionary is used to map and correct the colloquial expression words to ensure that the existing large model can recognize and output the professional expression mode text.

[0070] Identifying secondary words in the professional expression mode text and marking the secondary words. The secondary words include words such as mood words, adverbs, conjunctions, and adjectives that are not the main semantic carriers in the sentence and play a role in connecting the upper and lower sentences, semantic flow, and intent expression.

[0071] Respectively, the number of secondary word switches in the target voice segment text, the pre-segment voice segment text, and the post-segment voice segment text in the professional expression mode text is counted, and the semantic turning degree is output. The semantic turning degree refers to the number of times the secondary words appear or disappear in each voice segment in the professional expression mode text. This value reflects whether there is a syntactic logic or semantic emotion mutation between each voice segment.

[0072] If the semantic turning degree is higher than the preset threshold, it is determined as a broken structure, and the broken weight is output. Otherwise, it is determined as a coherent structure and the coherent weight is output. When the semantic turning degree is higher than the preset threshold, it means that the corresponding voice segment does not have a semantic connection with the previous or subsequent text, resulting in a semantic jump, which is a broken structure. At the same time, the broken weight is a negative value, which reflects the risk of merging the subsequent voice segment. The smaller the value, the greater the risk of merging and the lower the semantic coherence. At the same time, the coherent weight is a positive value. The larger the value, the smaller the risk of merging and the higher the semantic coherence. In this embodiment, the values of the broken weight and the coherent weight are set based on specific semantic situations. Based on the experience of those skilled in the art, the initial value of the broken weight can be set to , and the initial value of the coherent weight can be set to , which can be dynamically adjusted based on specific situations.

[0073] The semantic vector modeling is performed on the professional expression mode text to obtain a target speech segment semantic vector, a segment preceding semantic vector and a segment following semantic vector. In this embodiment, the pre-trained natural language processing model is used to convert the three speech segment texts corresponding to the professional expression mode text into vector representation forms to obtain the semantic vectors of the corresponding speech segments.

[0074] The semantic similarity between the target speech segment semantic vector and the segment preceding semantic vector is calculated to output a forward semantic coherence score, and the semantic similarity between the target speech segment semantic vector and the segment following semantic vector is calculated to output a backward semantic coherence score. The calculation formula of the semantic similarity is as follows: ; wherein and respectively represent a semantic vector, and when is the target speech segment semantic vector, if is the segment preceding semantic vector, the forward semantic coherence score output at this time is , which is the forward semantic coherence score of the target speech segment semantic vector ; if is the segment preceding semantic vector, the backward semantic coherence score output at this time is , which is the backward semantic coherence score of the target speech segment semantic vector ; if the target speech segment semantic vector has no segment preceding semantic vector or segment following semantic vector, the other vector in the formula except the target speech segment semantic vector is set as .

[0075] The forward semantic coherence score and the backward semantic coherence score are weighted and fused based on the fracture weight and the coherence weight to output a semantic coherence score. It should be noted that when the speech segment for which the semantic coherence score needs to be calculated is a fracture structure, the fracture weight is used to weightedly fuse the forward semantic coherence score and the backward semantic coherence score to output the semantic coherence score of the fracture structure at this time; when the speech segment for which the semantic coherence score needs to be calculated is a coherent structure, the coherence weight is used to weightedly fuse the forward semantic coherence score and the backward semantic coherence score to output the semantic coherence score of the coherent structure at this time.

[0076] The manner of screening the mergable speech segment group includes:

[0077] The coherence score threshold includes a merging threshold and an independent threshold. When the semantic coherence score is higher than or equal to the merging threshold, the corresponding context speech segment group is determined as a mergable speech segment group. The value of the merging threshold is relatively large, and the value of the independent threshold is relatively small. The specific setting can be based on expert experience or existing data. When the semantic coherence score is higher than or equal to the merging threshold, it indicates that the context speech segment group has continuity in semantic logic, and it is determined that the three structural speech segments can be merged forward, backward or integrally, that is, the context speech segment group is a mergable speech segment group.

[0078] When the semantic coherence score is lower than the independent threshold, the structural speech segments corresponding to the context speech segment group are determined as separate semantic structures respectively, and meanwhile do not participate in merging, wherein when the semantic coherence score is lower than the independent threshold, it means that there is inconsistency or semantic jump in the semantic logic between the structural speech segments in the context speech segment group, which may be due to the fact that the user actually made a natural pause or punctuation, so the different structural speech segments have different semantics and cannot be merged.

[0079] When the semantic coherence score is higher than the independent threshold and lower than the merging threshold, if the semantic coherence score is close to the merging threshold, the speech interval is adjusted for the corresponding context speech segment group, and the coherence is re-evaluated until the semantic coherence score is higher than or equal to the merging threshold, wherein the semantic coherence score higher than the independent threshold and lower than the merging threshold means that the context speech segment group at this time is in a fuzzy interval in the semantic level, so two sub-thresholds are set, which are the merging sub-threshold close to the merging threshold and the independent sub-threshold close to the independent threshold, if the semantic coherence score is lower than the merging threshold but greater than or equal to the merging sub-threshold, it means that the context speech segment group has a certain semantic continuity potential, so the dynamic sliding window size is adjusted to re-identify the speech interval, and the coherence is re-evaluated, and the process of adjusting the dynamic sliding window size and the coherence evaluation is repeated until the output semantic coherence score is higher than or equal to the merging threshold, so as to maximize the recovery of the actual complete semantics of the user in the original speech.

[0080] If the semantic coherence score is close to the independent threshold, the semantic interval is adjusted first and the coherence is evaluated, if the difference between the output semantic coherence score after adjustment and the merging threshold is greater than a preset distance threshold, the structural speech segments corresponding to the context speech segment are determined as separate semantic structures, wherein when the semantic coherence score is in the interval from the independent threshold to the independent sub-threshold, the dynamic sliding window size is adjusted and the coherence is evaluated once, the absolute value of the difference between the output coherence score after adjustment and the merging threshold is calculated, if the value is greater than the preset distance threshold, it means that the context speech segment group has no merging potential, so it is directly determined as a separate semantic structure.

[0081] The manner of performing semantic analysis includes:

[0082] The phrase or short sentence structure is matched with the preset traffic terminology dictionary, and the phrase or short sentence structure containing the traffic terminology is screened to obtain an independent traffic terminology entry. It should be noted that the traffic terminology dictionary is a dictionary constructed based on existing intelligent transportation field data, which includes commonly used traffic terminologies such as “vehicle flow”, “congestion index”, and road segment names. The phrase or short sentence structure is matched with the preset traffic terminology dictionary to obtain an independent traffic terminology entry.

[0083] It is detected whether the independent traffic terminology entry includes a conjunction, and the independent traffic terminology entry is classified based on the conjunction, including a connected compound phrase and a non-connected compound phrase. Since the user may generate some self-made compound traffic phrases composed of several existing traffic terminologies when performing voice output, the rule for matching with the traffic terminology dictionary is to complete matching when a field related to the traffic terminology appears. Therefore, the independent traffic terminology entry output may include multiple traffic terminologies. Therefore, in the embodiment, the independent traffic terminology entry is classified into a connected compound phrase and a non-connected compound phrase based on the conjunction. The connected compound phrase refers to a phrase or short sentence structure in which two traffic terminologies are connected by a conjunction, and the non-connected compound phrase refers to a phrase or short sentence structure in which several fields capable of indicating the general meaning of several traffic terminologies are combined, for example, “vehicle flow traffic efficiency”. The phrase is composed of “vehicle flow” and “traffic efficiency”.

[0084] The traffic terminology phrase adjacent to the decomposition marker in the connected compound phrase is extracted with the conjunction as the decomposition marker. The traffic terminology phrase adjacent to the conjunction is separated from the conjunction. The traffic terminology phrases are sorted according to the original sequence and are semantically concatenated to output a type of compound semantic segment. The traffic terminology phrases are sorted based on the original sequence, and the ordered traffic terminology phrase sequence after decomposition is semantically concatenated based on the syntax dependency logic, that is, a type of compound semantic segment. This type of semantic segment ensures consistency with the original semantic theme and reasonable sentence order.

[0085] The non-connected compound phrase is tokenized to obtain a preliminary split subsequence, and in this embodiment, the non-connected compound phrase is tokenized by using a semantic dependency model to divide the non-connected compound phrase into individual subunits and combine them into a preliminary split subsequence; a term token relationship graph is constructed based on the preliminary split subsequence, any token is taken as a node, the semantic correlation between tokens is calculated and taken as an edge, and the aggregation degree between tokens is presented by converting the preliminary split subsequence into a graph structure; wherein the semantic correlation is obtained by converting the token into a word vector, calculating the cosine similarity between the word vectors to obtain the semantic correlation and taking it as an edge.

[0086] The term token relationship graph is clustered by proximity to obtain the shortest relationship path of each token. It should be noted that for a relationship graph composed of tokens of a single non-connected compound phrase, the number of token nodes included is limited. In order to optimize the semantic dispersion, this embodiment selects a proximity clustering method based on semantic correlation to obtain a semantic path composed of tokens with the closest semantic relationship within a preset threshold, i.e. the shortest relationship path.

[0087] The shortest relationship path is reasonably screened to output a reasonable relationship path, wherein the screening requirement of reasonable screening refers to that the token position required to form the shortest relationship path is appropriate and the semantic flow is smooth, and the connection between tokens can reflect the complete semantics; the tokens in the reasonable relationship path are sequentially connected according to the original syntax to obtain a two-class compound semantic segment, and the tokens in the screened path are sequentially connected according to the original syntax according to the syntax dependency logic, so that the two-class compound semantic segment can clearly represent the overall semantics and reflect the user intent; the one-class compound semantic segment and the two-class compound semantic segment are integrated to obtain the user query intent information.

[0088] The way of constructing the field structure image includes:

[0089] All relevant fields in the query target data subset are extracted, and the fields are classified based on the source of the relevant fields, and the field type is labeled by using the traffic business category to which the field belongs, wherein the query target data subset refers to the specific numerical value or information of the relevant data matched with the user query intent information obtained from the preset database based on the user query intent information in step S4, i.e. the relevant field; wherein the source of the relevant field refers to the data source of the required information in the preset database, and the fields are classified based on the traffic business categories corresponding to these data sources, such as multiple traffic business categories such as “vehicle flow statistics” and “congestion trend”.

[0090] The basic attribute index of the target data subset is queried and statistically, and feature extraction is performed on the basic attribute index to output target mathematical distribution features. The basic attribute index refers to, for example, the value range, mean value, and variance of a field, which are used to express the basic statistical characteristics of the field value in the overall query target data subset. Meanwhile, based on these basic statistical characteristics, mathematical methods are used to obtain mathematical distribution features corresponding to the basic attribute index, such as normal distribution and discrete distribution, which are the target mathematical distribution features.

[0091] The time range corresponding to each related field is detected, and trend change analysis is performed on the corresponding numerical values of the related field in the time range to obtain the trend change feature of the related field. The time range refers to the time period covered by the field value of the related field in the query target data subset. In this embodiment, a regression algorithm is used to perform trend analysis on the field value of the related field in the time range. The trend change feature obtained includes, for example, change direction, period, and mutation point, which have time evolution structure characteristics.

[0092] The numerical value of the related field is compared with the preset safe numerical value interval. If it exceeds the preset safe numerical value interval, it is determined as an alarm field, and an alarm mark is added. The preset safe numerical value interval refers to a reasonable numerical value fluctuation interval in the traffic business field to which the related field belongs. For example, the traffic flow should not exceed By marking the field that exceeds the preset safe numerical value interval as an alarm, it is convenient to subsequently construct a field structure image with more complete information. The field type is matched with the target mathematical distribution feature, the trend change feature, and the alarm mark, respectively, to output the field structure image of each related field. The field structure image refers to a data collection of the statistical level features, change trends, and alarm information of the related field matched with the corresponding traffic business category, which provides a reference basis for subsequent chart matching.

[0093] The generation of the target chart format includes:

[0094] The voice text corresponding to the user query intention information is obtained, and the pre-trained corpus model is used to perform semantic extraction on the voice text to output a user intention vector. In this embodiment, a natural language processing model trained in the traffic field is used to extract the word vector combination and its semantic context dependency features in the voice text and encode them into a vector form to obtain the user intention vector.

[0095] The user semantic tendency is determined based on the user intention vector. In this embodiment, the keywords are mapped to tendency labels, for example, "trend, change, increase, decrease" can be mapped to a trend analysis tendency, and "current, now" can be mapped to a real-time state tendency. The user semantic tendency refers to semantic information that the user needs or pays more attention to. Based on the user intention vector as data, the corresponding tendency label of the keyword code is identified. The tendency label that has the greatest impact is determined through the attention mechanism. The semantic information of the tendency label is taken as the user semantic tendency.

[0096] The requirement target information is generated in combination with the field structure image and the corresponding user semantic tendency. The field structure image and the user semantic tendency are jointly modeled to generate structured requirement target information. The requirement target information includes field types, time ranges, trend change patterns, and alarm states, and the information is used as a basis for subsequent matching with charts.

[0097] The requirement target information is used as data basis to construct a target chart format using a preset language large model, calculate a matching fitness, and select a candidate chart based on the matching fitness. The calculation formula of the matching fitness is: In this embodiment, there is a language large model for chart generation. The matching fitness of the generated chart format and the requirement target information is calculated using the matching fitness formula, and whether the currently generated chart format meets the requirements is determined based on the matching fitness. In the calculation formula, represents the matching fitness of the requirement target information and any generated chart. represents the semantic matching weight. The value is obtained by calculating the cosine similarity between the requirement vector corresponding to the requirement target information and the label vector of the generated chart. represents the field structure matching weight. The weight refers to the matching degree of fields such as continuous, discrete, time, or space structure with the function of the generated chart. The value is dynamically set based on historical experience. represents the field quantity matching weight, which is used to reflect whether the field quantity corresponding to the requirement target information is within the range of the field quantity supported by the generated chart. When the field quantity corresponding to the requirement target information exceeds the range of the field quantity supported by the generated chart, the value is , otherwise, the ratio of the field quantity corresponding to the requirement target information to the field quantity supported by the generated chart is calculated as the value of . A matching fitness threshold is set, and the generated chart with a matching fitness higher than the threshold is selected as a candidate chart.

[0098] The expression ability of the candidate chart is evaluated, and the target chart parameters are output based on the evaluation results. In this embodiment, a simulation environment is constructed, all fields corresponding to the demand target information are merged with each candidate chart in the simulation environment, the effect of the generated chart is simulated, and the expression ability of the candidate chart is evaluated from the information display sufficiency, visual complexity, and interaction performance. Based on the existing evaluation standard, the chart with the best expression ability is selected, and the target chart parameters of the chart are output.

[0099] The structure modification method includes:

[0100] Based on the target chart parameters, a corresponding chart is constructed. After each interaction behavior is completed, the interaction belonging modal type and the interaction action area of the multi-modal interaction information are extracted to construct interaction association information. The interaction belonging modal type includes interaction behaviors such as clicking, dragging, and zooming, and the interaction action area refers to the area of the chart affected by each interaction behavior. The interaction belonging modal type and the interaction action area are combined to form the interaction association information, which is a data structure including the above two types of data.

[0101] Based on the interaction association information, the interaction behavior is divided into simple interaction behavior and semantic interaction behavior. The simple interaction behavior refers to basic interaction operations such as clicking and dragging, and the semantic interaction behavior refers to interaction operations involving voice instructions, such as the user's request to "only see peak hours".

[0102] When the interaction behavior is a simple interaction behavior, the corresponding function architecture of the corresponding chart is called to perform internal parameter self-adaptive adjustment. It should be noted that when the interaction behavior is a simple interaction behavior, only the parameters in the interaction control structure of the corresponding area in the chart need to be adjusted to realize simple interaction actions such as zooming in, sliding, and zooming out, and to realize lightweight state update of the chart.

[0103] When the interaction behavior is a semantic interaction behavior, the semantic target of the semantic interaction behavior is extracted, a suitable chart format is regenerated based on the semantic target using a preset language large model, and the target chart parameters are reconfigured. By performing semantic analysis on the voice corresponding to the semantic interaction behavior, the user's intention is determined, the semantic target is obtained, and the preset language large model is called to generate a chart format more suitable for the current user's intention based on the semantic target. At the same time, the fields that meet the user's intention are obtained and the parameters of the new chart are reconfigured.

[0104] The chart adjusted based on the interaction behavior is output as a dynamic correction target chart. It should be noted that this describes a switching method of chart formats generated by a large model, which is only used in the case where the user changes the requirements after generating the first chart in the speech recognition process. Therefore, the previous chart adjusted by parameter self-adaptation has been sent to the preset data big screen for display before the dynamic correction target chart is output. This is only used for updating the chart format.

[0105] The manner of performing trend evolution display includes:

[0106] A time axis is constructed. In this embodiment, the time axis constructed is a time axis that can be dynamically adjusted based on user requirements. All fields in the dynamic correction target chart are bound to corresponding time stamps. The fields are bound to the time stamps in order to establish a mapping relationship between the fields and the time stamps, so that the field values can be adjusted synchronously by adjusting the time axis subsequently.

[0107] By adjusting the adjustable time axis, the values of each field in the dynamic correction target chart are switched frame by frame based on the time stamps, and an optimized target chart is obtained. In this embodiment, when the user adjusts the adjustable time axis, the system obtains the field value corresponding to the next time stamp based on the time stamp corresponding to the current field value and the mapping relationship, and synchronously updates the rendering content of the chart, so as to realize frame-by-frame switching of the chart caused by the change of the time axis, and further improve the trend change observability of the chart. In addition, the adjustable time axis in this embodiment is bidirectional, and therefore the optimized target chart obtained finally is a dynamic chart covering the whole process of time evolution.

[0108] In this embodiment, the user query intention information is extracted by first performing structural processing and semantic analysis on the collected user speech data, and a suitable chart format is generated based on the user query intention information. Meanwhile, the interaction in the chart is optimized, and an intelligent chart dynamic generation method based on speech recognition and multi-modal interaction is realized. Compared with existing experience, the user speech is analyzed in multiple levels, which significantly improves the accuracy of user intention recognition under speech input. In particular, in terms of dealing with colloquial expressions, sentence pauses, and accent or dialect interference, the robustness of speech recognition is improved. Meanwhile, the semantic analysis operation is deeply integrated with the traffic business knowledge, and the field structure portrait is generated, which provides a basis for generating charts by using a language large model. Further, by combining the chart matching fitness and the expression ability, accurate matching of user semantics and chart types is realized. By capturing the user's interaction actions, adaptive adjustment of the chart structure parameters and the chart format is realized, the interaction requirements of the user are met, and the user experience is improved.

[0109] Embodiment 2

[0110] Please refer to Figure 2As shown, the embodiment does not describe part of the embodiment 1, provides an intelligent chart dynamic generation system based on speech recognition and multi-modal interaction, comprising:

[0111] The data acquisition module is used for collecting user voice data and performing voice data denoising to obtain accurate user voice data.

[0112] The semantic segmentation module is used for performing semantic structure segmentation and combination on the accurate user voice data to output complete user semantic data.

[0113] The semantic analysis module is used for performing semantic analysis on the complete user semantic data to output user query intent information.

[0114] The data query module is used for extracting a query target data subset from a preset database based on the user query intent information.

[0115] The chart generation module is used for constructing a field structure portrait based on the query target data subset, generating a target chart format based on the field structure portrait using a preset language large model, and outputting target chart parameters.

[0116] The interaction optimization module is used for collecting multi-modal interaction information in real time, performing structure correction on the target chart parameters based on the multi-modal interaction information, and outputting a corrected target chart.

[0117] The chart visualization module is used for performing trend evolution display on the corrected target chart to obtain a dynamically optimized target chart, and sending the dynamically optimized target chart to a preset data big screen. The various modules are connected through wired and / or wireless means.

[0118] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or replace some of the technical features with equivalent ones. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

[0119] It should be noted that in this document, the terms "include", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or device including the element.

[0120] In the description of the application, it should be understood that the terms "first", "second" and the like are used to distinguish descriptions and are not intended to imply or imply relative importance.

[0121] In the description of the application, unless otherwise specified, the meaning of "a plurality of" is two or more.

[0122] In the description of the application, the meaning of "several" is one or more, and the meaning of "a large number" is two or more.

[0123] In the description of the present application, the description of the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0124] For the formula of the present application, the value is calculated by dimensionless, the formula is obtained by collecting a large amount of data to simulate the recent real situation, and the preset parameters and threshold values in the formula are set by the person skilled in the art according to the actual situation.

[0125] Although the embodiments of the present application have been shown and described, those skilled in the art can understand that various changes, modifications, replacements and variations can be made to these embodiments without departing from the principles and purposes of the present application, and the scope of the present application is defined by the claims and their equivalents.

Claims

1. A method for dynamically generating intelligent charts based on speech recognition and multimodal interaction, characterized in that: include: S1. Collect user voice data and perform voice data denoising to obtain accurate user voice data; S2. Perform semantic structure segmentation and combination on the precise user voice data, including: A dynamic sliding window is set and used to traverse the precise user speech data, and speech intervals whose interval time is greater than the dynamic sliding window size are marked as boundaries; the precise user speech data is structurally divided based on the marked speech intervals, and a set of structured speech segments is output; the two adjacent structured speech segments before and after each structured speech segment are extracted, and the structured speech segment and the adjacent structured speech segments are combined into a context speech segment group; Perform a coherence assessment on each contextual speech segment group and output a semantic coherence score. Set a coherence score threshold and combine the semantic coherence score and the coherence score threshold to screen for mergeable speech segment groups. Merge the mergeable speech segment groups and output complete user semantic data. Structurally divide the complete user semantic data based on reasonable speech intervals and calculate the comprehensive coherence score of the divided complete user semantic data. If the coherence score threshold is not met, adjust the segment merging strength; otherwise, do not adjust it. The methods of coherence assessment include: Perform speech-to-text recognition on any contextual speech segment group and output the original recognized text; collect user information, load the local accent reference database based on the user information, convert the original recognized text to standard text based on the local accent reference database, and output the standard recognized text; identify colloquial words in the standard recognized text, perform word form correction on the colloquial words based on a preset professional expression dictionary, and output the professional expression model text; Identify and label sub-words in professional expression pattern texts; count the number of sub-word switches in the target speech segment, the preceding speech segment, and the following speech segment in the professional expression pattern texts, and output the degree of semantic transition; if the degree of semantic transition is higher than a preset threshold, it is determined to be a broken structure and the fracture weight is output; otherwise, it is determined to be a coherent structure and the coherence weight is output; Semantic vector modeling is performed on professional expression pattern text to obtain the target speech segment semantic vector, pre-segment semantic vector, and post-segment semantic vector respectively; the semantic similarity between the target speech segment semantic vector and the pre-segment semantic vector is calculated, and a forward semantic coherence score is output; the semantic similarity between the target speech segment semantic vector and the post-segment semantic vector is calculated, and a backward semantic coherence score is output; the forward semantic coherence score and the backward semantic coherence score are weightedly fused based on the break weight and the coherence weight, and a semantic coherence score is output; The method of screening the mergeable speech segment groups includes: The coherence score threshold includes a merging threshold and an independent threshold. When the semantic coherence score is higher than or equal to the merging threshold, the corresponding context speech segment group is determined to be a mergible speech segment group; when the semantic coherence score is lower than the independent threshold, the structural speech segments corresponding to the context speech segment group are respectively determined to be independent semantic structures and are not involved in the merging; when the semantic coherence score is higher than the independent threshold and lower than the merging threshold, if the semantic coherence score is close to the merging threshold, the speech interval of the corresponding context speech segment group is adjusted and the coherence evaluation is re-performed until the semantic coherence score is higher than or equal to the merging threshold; if the semantic coherence score is close to the independent threshold, the semantic interval adjustment is first performed and the coherence evaluation is performed. If the difference between the adjusted output semantic coherence score and the merging threshold is greater than the preset distance threshold, the structural speech segment corresponding to the context speech segment is determined to be an independent semantic structure; S3. Perform semantic analysis on the complete user semantic data and output user query intent information; S4. Extracting a subset of query target data from a preset database based on user query intent information; S5. Construct a field structure portrait based on the query target data subset, generate a target chart format based on the field structure portrait using a preset language model, and output the target chart parameters; S6. real-time collection of multimodal interaction information, structural correction of target chart parameters based on the multimodal interaction information, and output of a dynamically corrected target chart; S7. Execute trend evolution display on the dynamically modified target chart to obtain an optimized target chart; and send the dynamically optimized target chart to a preset data large screen.

2. The method for dynamically generating intelligent charts based on speech recognition and multimodal interaction according to claim 1, characterized in that: The method of performing semantic parsing includes: Extract phrase and sentence structures from complete user semantic data, match them with the preset traffic term dictionary, filter phrases or sentence structures containing traffic terms, and obtain independent traffic term entries; detect whether the independent traffic term entries contain conjunctions, and classify the independent traffic term entries based on the conjunctions, including conjunction compound phrases and non-conjunction compound phrases; Using conjunctions as decomposition markers, the traffic term phrases adjacent to the decomposition markers in the connected compound phrases are extracted; the traffic term phrases are sorted according to the original word order and semantically concatenated to output a class of compound semantic fragments; The non-connected compound phrases are decomposed into lemmas to obtain preliminary split subsequences; a term-lemma relationship graph is constructed based on the preliminary split subsequences, with any lemma as a node, and the semantic relevance between lemmas is calculated and used as edges; the term-lemma relationship graph is clustered by neighbors to obtain the shortest relationship path for each lemma; the shortest relationship paths are reasonably screened and reasonable relationship paths are output; the lemmas in the reasonable relationship paths are semantically concatenated according to the original word order to obtain two types of compound semantic fragments; the first type of compound semantic fragments and the second type of compound semantic fragments are integrated to obtain user query intention information.

3. The method for dynamically generating intelligent charts based on speech recognition and multimodal interaction according to claim 2, characterized in that: The method of constructing the field structure portrait includes: Extract all relevant fields in the query target data subset, classify the fields based on the source of the relevant fields, and use the traffic business category to which the fields belong to mark the field type; count the basic attribute indicators of the query target data subset, extract the features of the basic attribute indicators, and output the target mathematical distribution characteristics; detect the time range corresponding to each relevant field, and perform trend change analysis on the corresponding values ​​of the relevant fields within the time range to obtain the trend change characteristics of the relevant fields; compare the values ​​of the relevant fields with the preset safety value range, and if they exceed the preset safety value range, determine them as alarm fields, and add alarm marks; match the field type with the target mathematical distribution characteristics, trend change characteristics and alarm marks respectively, and output the field structure portrait of each relevant field.

4. The method for dynamically generating intelligent charts based on speech recognition and multimodal interaction according to claim 3, characterized in that: The method of generating the target chart format includes: Obtain the speech text corresponding to the user's query intention information, use the pre-trained corpus model to perform semantic extraction on the speech text, and output the user intention vector; judge the user's semantic tendency based on the user intention vector; combine the field structure portrait with the corresponding user semantic tendency to generate demand target information; use the demand target information as the data basis, use the preset language model to construct the target chart format, calculate the matching fitness, and screen candidate charts based on the matching fitness; evaluate the expressive ability of the candidate charts, and output the target chart parameters based on the evaluation results.

5. The method for dynamically generating intelligent charts based on speech recognition and multimodal interaction according to claim 4, characterized in that: The method of performing structural correction includes: Construct a corresponding graph based on the target graph parameters; extract the interaction mode type and interaction area of ​​the multimodal interaction information after each interaction behavior is completed, and construct interaction association information; based on the interaction association information, divide the interaction behavior into simple interaction behavior and semantic interaction behavior; When the interactive behavior is a simple interactive behavior, the corresponding functional architecture of the corresponding chart is called to adaptively adjust the internal parameters of the structure; when the interactive behavior is a semantic interactive behavior, the semantic target of the semantic interactive behavior is extracted, and the preset language model is used to regenerate the appropriate chart format based on the semantic target, and the target chart parameters are reconfigured; the chart adjusted based on the interactive behavior is output as a dynamically corrected target chart.

6. The method for dynamically generating intelligent charts based on speech recognition and multimodal interaction according to claim 5, characterized in that: The method of performing trend evolution display includes: Construct an adjustable timeline; bind all fields in the dynamically corrected target chart to corresponding timestamps; and switch the value of each field in the dynamically corrected target chart frame by frame based on the timestamp by adjusting the adjustable timeline to obtain an optimized target chart.

7. An intelligent chart dynamic generation system based on speech recognition and multimodal interaction, which is used to implement the intelligent chart dynamic generation method based on speech recognition and multimodal interaction according to any one of claims 1 to 6, characterized in that: include: The data acquisition module is used to collect user voice data and perform voice data denoising to obtain accurate user voice data; Semantic segmentation module, used to segment and combine the precise user voice data into semantic structures and output complete user semantic data; Semantic parsing module, used to perform semantic parsing on complete user semantic data and output user query intention information; A data query module is used to extract a query target data subset from a preset database based on user query intention information; A chart generation module is used to construct a field structure portrait based on the query target data subset, generate a target chart format based on the field structure portrait using a preset language model, and output target chart parameters; The interactive optimization module is used to collect multimodal interaction information in real time, perform structural correction on the target chart parameters based on the multimodal interaction information, and output the corrected target chart; A chart visualization module is used to display the trend evolution of the modified target chart to obtain a dynamically optimized target chart; The dynamic optimization target chart is sent to the preset data screen; each module is connected via wired and / or wireless means.

Images