A large model multi-modal understanding and interaction method for a data large screen

Abstract

The application discloses a large model multi-modal understanding and interaction method for a data large screen, relates to the technical field of understanding and interaction, and comprises the following steps: collecting real-time state data, performing unified timestamp alignment, mapping user-selected pointing information to a twin three-dimensional coordinate system, obtaining a hit result, and mapping the hit result into a candidate twin entity set; extracting and aggregating entity correlation state data from the candidate twin entity set, encapsulating a multi-modal twin alignment package; performing end-side voice activity detection on a voice audio stream in the multi-modal twin alignment package, triggering streaming automatic speech recognition, obtaining a partial transcription result, and generating intent decision information through preliminary intent coarse classification; and the application enhances the safety and controllability of digital twin decision.

Description

Technical Field

[0001] This invention relates to the field of interactive understanding technology, and in particular to a method for understanding and interacting with large-scale models of data dashboards. Background Technology

[0002] With the advancement of the Industrial Internet, smart cities, and complex equipment operation and maintenance, digital twin technology is gradually evolving from a development stage primarily based on geometric modeling and offline simulation to a "virtual-real linkage" oriented towards operational states. To improve human-machine collaboration efficiency, the industry has introduced multimodal entry points such as voice interaction, pointing operations, text commands, and visual understanding, combined with the language reasoning and tool invocation capabilities of large models, to achieve querying, diagnosis, and control intent parsing of twin scenarios. Common technical paths include: processing multi-source state data with a unified time base; mapping AR / 3D interface clicks or pointing to the twin coordinate system to hit candidate entities; acquiring speech transcription and fusing it with text input; and using intent recognition and entity linking to assign user expressions to twin objects or alarm events, thereby driving visual feedback and action execution to form a closed-loop interactive experience.

[0003] In existing digital twin multimodal interaction systems, interaction decisions typically rely on fixed thresholds, empirical rules, or single model scores for triggering and follow-up questions, lacking a calibrable characterization of uncertainty. Consequently, the confidence level of interaction decisions is difficult to calibrate. Furthermore, under uncertain input conditions such as incomplete speech transcription stages, ambiguous directional choices, and delays and noise in state data, it is difficult to clearly define whether an action should be triggered and when it should be triggered. This results in unclear action trigger boundaries, making it difficult to balance real-time interaction with operational safety. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a large-scale model multimodal understanding and interaction method for data dashboards to solve the problems of difficulty in calibrating the confidence of interactive decisions and unclear action trigger boundaries under uncertain input conditions.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: This invention provides a method for multimodal understanding and interaction of large models for data dashboards. The method includes: collecting real-time state data and aligning it with a unified timestamp; mapping user-selected pointing information to a twin 3D coordinate system; obtaining hit results and mapping the hit results to a candidate twin entity set; extracting and aggregating entity-related state data from the candidate twin entity set and encapsulating it to generate a multimodal twin alignment package; performing edge-side voice activity detection on the speech audio stream in the multimodal twin alignment package and triggering streaming automatic speech recognition to obtain partial transcription results; generating intent decision information through preliminary intent coarse classification; extracting semantic cues from user-expressed text and intent decision information; fusing user perspective, topological adjacency, and anomaly alarms to obtain multi-source localization evidence features; inputting these features into a learnable fusion localization model to generate entity localization results; performing calibrable uncertainty assessment on the confidence level and action risk level of the entity localization results; and performing confidence judgment based on conformal confidence gating to generate interaction decision results.

[0007] As a preferred embodiment of the large-model multimodal understanding and interaction method for data dashboards described in this invention, the specific steps for obtaining the hit results and mapping them to a set of candidate twin entities are as follows: Collect real-time status data, establish a unified clock reference, attach a collection time mark to each piece of real-time status data, calculate the fixed offset of the real-time status data relative to the unified clock reference, and obtain the timestamp status sequence. Perform continuous reconstruction on the timestamp state sequence, obtain the state trajectory, and couple and bind the user-selected pointing information with the state trajectory according to the time interval of occurrence to obtain the pointing state association result; The pointing state association result is mapped to the twin 3D coordinate system, the pointing trajectory is obtained and the timestamp corresponding to the twin 3D coordinate is used to calculate the hit result, and the hit result is filtered to form a candidate twin entity set.

[0008] As a preferred embodiment of the large-model multimodal understanding and interaction method for data dashboards described in this invention, the specific steps for extracting and aggregating entity association state data from the candidate twin entity set and encapsulating it to generate a multimodal twin alignment package are as follows. Extract entity association state data from the candidate twin entity set according to the timestamp, perform aggregation calculation on the entity association state data under a unified clock reference, and obtain aggregated state data; The aggregated state data is mapped one-to-one with the timestamp and then associated and encapsulated to generate a multimodal twin alignment package.

[0009] As a preferred embodiment of the large-model multimodal understanding and interaction method for data dashboards described in this invention, the specific steps for performing end-side speech activity detection on the speech audio stream in the multimodal twin alignment package are as follows: On the device side, a continuous speech audio stream is parsed from the multimodal twin alignment packet, and the speech audio stream is uniformly sampled and framed to obtain audio frames. On the device side, the energy change, spectrum change and stability characteristics between the audio frame and the previous audio frame are calculated in real time to obtain the frame feature results. The frame feature results are comprehensively calculated to obtain the speech activity judgment value, and the judgment and recognition of speech activity frames and non-speech frames are integrated to generate speech activity detection results.

[0010] As a preferred embodiment of the large-model multimodal understanding and interaction method for data dashboards described in this invention, the specific steps for generating intent decision information are as follows: The speech activity detection results are divided into speech activity intervals, audio frame data is extracted from the speech activity intervals, and streaming automatic speech recognition is triggered to perform decoding processing to obtain transcription candidate sequences. Candidate sequences for transcription are filtered and confirmed in chronological order, and the decoded content in the candidate sequences is extracted to obtain partial transcription results; The partial transcription results are merged with the input text to obtain the user's expressed text, and a preliminary coarse classification of the user's expressed text is performed to obtain the preliminary intent category and confidence level. The initial intent category and confidence level are encapsulated and output to generate intent decision information.

[0011] As a preferred embodiment of the large-model multimodal understanding and interaction method for data dashboards described in this invention, the steps of extracting semantic cues from user-expressed text and intent decision information, fusing user perspective, topological adjacency, and anomaly alarms to obtain multi-source localization evidence features are as follows. Extract semantic cues related to objects, behaviors, and spaces from partial transcription results and align them with intent decision information to obtain structured semantic constraints; Based on structured semantic constraints, multi-source constraint fusion is performed by combining user perspective, topological adjacency and anomaly alarms to obtain multi-source evidence scores. The multi-source evidence scores and structured semantic constraints are then substituted into the integration formula, and multi-source localization evidence features are obtained through nonlinear fractional operations.

[0012] As a preferred embodiment of the large-model multimodal understanding and interaction method for data dashboards described in this invention, the input can learnable fusion positioning model to generate entity positioning results, and the specific steps are as follows. The multi-source localization evidence features are organized according to timestamps and corresponding candidate twin entities in the candidate twin entity set. Alignment feature sequences are obtained and a learnable fusion localization model is obtained through standardized input. The fusion localization score of each candidate twin entity in the candidate twin entity set is calculated. The fusion localization scores are sorted and filtered, and the final priority order of candidate twin entities is determined according to the fusion localization scores to generate entity localization results.

[0013] As a preferred embodiment of the large-model multimodal understanding and interaction method for data dashboards described in this invention, the construction steps of the learnable fusion positioning model are as follows: An input layer is established and the weights of each multimodal feature are initialized. A fusion layer is constructed, and different multimodal features are processed by weighting. A nonlinear activation function is used to extract the relationship between multimodal features. By using a spatiotemporal estimation layer to capture temporal and spatial dependencies, spatiotemporal positioning results are obtained and combined with risk indicators calculated from confidence and risk analysis to make decisions, thus constructing a learnable fusion positioning model.

[0014] As a preferred embodiment of the large-model multimodal understanding and interaction method for data dashboards described in this invention, the specific steps for assessing the calibrable uncertainty of the confidence level and action risk level of the entity localization results are as follows: The confidence level and action risk level of the positioning results are extracted from the candidate twin entities as evaluation objects, and the evaluation objects are integrated and standardized to obtain a standardized fusion feature vector. The standardized fusion feature vectors are weighted and aggregated, and then processed by a nonlinear mapping function to generate confidence risk values.

[0015] As a preferred embodiment of the large-model multimodal understanding and interaction method for data dashboards described in this invention, the specific steps for generating interactive decision results are as follows: The confidence risk value is interactively determined by Conformal confidence gating, which outputs clarification prompts to users and receives user feedback information, and generates a gating confidence score. When the gate confidence score is lower than the risk suppression threshold, a clarification question-and-answer mechanism is triggered, generating a highlighted mark. When the gate confidence score is higher than the risk suppression threshold but lower than the risk release threshold, the observation and confirmation mechanism is triggered, and a pending confirmation status is generated. When the gate confidence score is higher than the risk release threshold, the interactive decision generation mechanism is triggered, and an interactive decision value is generated through a non-linear fractional calculation method. The interactive decision values ​​are prioritized and sorted from high to low under the condition of the same risk level. High-risk tasks with high decision values ​​are processed first. The sorted tasks are formatted and displayed on the big screen in real time to generate interactive decision results.

[0016] The beneficial effects of this invention are as follows: By aligning with unified timestamps and mapping user pointers to the three-dimensional coordinates of the digital twin, candidate digital twin entities can be quickly located and spatiotemporally one-way fatality can be achieved, reducing ambiguity and improving interaction accuracy; furthermore, entity association state data is aggregated and encapsulated into a multimodal digital twin alignment package, which, combined with edge-side speech detection, streaming transcription, and coarse intent classification, forms a traceable contextual input, enhancing real-time understanding and robustness; in particular, semantic cues, user perspective, topological adjacency, and anomaly alarms are integrated to generate multi-source evidence and input into a learnable localization model, improving the reliability of digital twin entity localization; and clarification, confirmation, and decision-making are triggered by calibrable uncertainty assessment and conformal confidence gating, enhancing the security and controllability of digital twin decision-making. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a method for multimodal understanding and interaction of large models for data dashboards.

[0019] Figure 2 A flowchart for timestamp alignment and candidate entity generation.

[0020] Figure 3 A flowchart for generating intent decision information.

[0021] Figure 4 A flowchart for generating confidence risk values. Detailed Implementation

[0022] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0023] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0024] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0025] Reference Figures 1-4 This is one embodiment of the present invention, which provides a method for large-scale model multimodal understanding and interaction for data dashboards, including the following steps: S1. Collect real-time status data and perform unified timestamp alignment, map the user-selected pointing information to the twin 3D coordinate system, obtain the hit result, and map the hit result to a set of candidate twin entities.

[0026] S1.1 Collect real-time status data, establish a unified clock reference, attach a collection time mark to each piece of real-time status data, calculate the fixed offset of the real-time status data relative to the unified clock reference, and obtain the timestamp status sequence.

[0027] Furthermore, when collecting real-time status data, a collection time marker is written to the corresponding real-time status data at the same time as each collection action occurs. A unified clock reference is formed by continuous timing with the same time scale to provide a reference time point. The fixed offset is obtained by performing difference calculation on the collection time marker time point corresponding to the same collection action and the unified clock reference time point. The fixed offset is applied to the collection time marker as a constant correction in the subsequent processing of each real-time status data, converting the collection time marker into a time expression consistent with the unified clock reference, and arranging the real-time status data according to the converted time sequence to obtain the timestamp status sequence.

[0028] S1.2 Perform continuity reconstruction on the timestamp state sequence, obtain the state trajectory, and couple and bind the user-selected pointing information with the state trajectory according to the time interval of occurrence to obtain the pointing state association result.

[0029] Furthermore, the timestamp state sequence is read in timestamp order and continuous reconstruction is performed. By checking the timestamp intervals of adjacent states and splicing adjacent states in chronological order to form a state trajectory, the user-selected pointing information includes the pointing event record and the time interval of the pointing event. The coupling binding is achieved by extracting the start and end times of the time interval of the pointing event and retrieving state trajectory fragments that fall within the time interval in the state trajectory. The user-selected pointing information is then bound to the corresponding state trajectory fragments to obtain the pointing state association result.

[0030] It should be noted that the user-selected pointing information comes from the pointing event records generated by the user's interaction. The user-selected pointing information is used to characterize the user's selection and pointing behavior towards the target object or target location within a certain time interval. The user-selected pointing information includes at least the time interval of the pointing event and the target identifier.

[0031] S1.3 Map the pointing state association result to the twin 3D coordinate system, obtain the pointing trajectory and calculate the hit result with the timestamp corresponding to the twin 3D coordinates, and filter the hit result to form a candidate twin entity set.

[0032] Furthermore, the pointing state association results are mapped to the twin 3D coordinate system. The twin 3D coordinate system is a global spatial coordinate system established based on the digital twin 3D model of the target scene (pre-calibrated by the twin platform and its origin, axis, and scale are defined). By performing coordinate transformation on the spatial position of the state trajectory at each timestamp and converting it into a sequence of spatial trajectory points in the twin 3D coordinate system, the pointing trajectory is obtained. The hit result is calculated by performing spatial distance calculation or spatial inclusion relationship determination on the twin 3D coordinates whose spatial trajectory points at each timestamp are consistent with the timestamps of the twin 3D coordinates. The hit result is filtered to form a candidate twin entity set by retaining the twin entities with hit records and summarizing and outputting the candidate twin entity set.

[0033] S2. Extract and aggregate entity association state data from the candidate twin entity set, and encapsulate it to generate a multimodal twin alignment package.

[0034] S2.1 Extract entity association state data from the candidate twin entity set according to the timestamp, perform aggregation calculation on the entity association state data under a unified clock reference, and obtain aggregated state data.

[0035] Furthermore, the candidate twin entity set originates from the output results of the candidate twin entity set formed by the hit result screening. Entity-related state data is extracted from the candidate twin entity set according to the timestamp. This is achieved by reading the entity-related state data record corresponding to the timestamp for each twin entity in the candidate twin entity set, and arranging the entity-related state data records in chronological order according to the timestamp to form an entity-related state data sequence that can be indexed by timestamp. Aggregation calculation is performed on the entity-related state data under a unified clock reference by converting the timestamps in the entity-related state data sequence into a time expression under a unified clock reference. The corresponding entity-related state data records are aggregated using the same timestamp or adjacent time intervals under the unified clock reference as the aggregation unit. Within each aggregation unit, the aggregation calculation performs a summary operation on the same-named state fields of multiple entity-related state data records to obtain the aggregated state field value. The summary operation can use methods such as summation, mean, maximum or minimum value, and outputs the aggregated state data arranged according to the unified clock reference.

[0036] S2.2. Assign and encapsulate the aggregated state data with the timestamps to generate a multimodal twin alignment package.

[0037] Furthermore, the aggregated state data is read one by one according to the timestamp order under a unified clock reference. When each aggregated state data is read, the corresponding timestamp is extracted synchronously and a one-to-one correspondence between the timestamp and the aggregated state data is established. The associated encapsulation is achieved by writing the timestamp into the same encapsulation content as the aggregated state data and using the timestamp as an index to save the position of the aggregated state data. At the same time, the aggregated state data is completely written into the encapsulation content while ensuring that the correspondence between the timestamp and the aggregated state data does not overlap. All encapsulation content is collected sequentially according to the timestamp order to form a continuous encapsulation sequence. The continuous encapsulation sequence is output as a multimodal twin aligned packet.

[0038] It should be noted that the multimodal twin alignment packet includes a packet header and a sequence of records sorted by timestamp. The packet header contains a unified clock reference, timestamp unit / precision, and coordinate system identifier. Each record is indexed by timestamp t and contains eid, mid, payload (key / value / unit), q, and pose. Records under the same timestamp are aggregated and stored using sub-sequence number sub_idx to avoid overlapping correspondences.

[0039] S3. Perform end-side speech activity detection on the speech audio stream in the multimodal twin alignment package, trigger streaming automatic speech recognition, obtain partial transcription results, and generate intent decision information through preliminary intent coarse classification.

[0040] S3.1. On the device side, parse the continuous speech audio stream from the multimodal twin alignment packet, perform unified sampling and frame segmentation on the speech audio stream, and obtain audio frames.

[0041] Furthermore, on the edge side, a parsing operation is performed on the multimodal twin alignment packet. The parsing operation reads the encapsulated content one by one according to the encapsulation order of the multimodal twin alignment packet and locates the payload marked as speech audio in the encapsulated content. The speech audio payloads in each encapsulated content are spliced ​​in the order of timestamps to form a continuous speech audio stream. The speech audio stream is uniformly sampled by converting the speech audio stream into a discrete sampling sequence with the same sampling rate. When the sampling rate is inconsistent, a resampling operation is performed. The resampling operation generates a discrete sampling point sequence at the target sampling rate by interpolating or extracting the original discrete sampling point sequence. The speech audio stream is framed by extracting continuous segments in sequence from the discrete sampling point sequence based on a fixed time window and recording the corresponding start and end timestamps for each segment, outputting an audio frame composed of multiple segments.

[0042] S3.2 Calculate the energy change, spectrum change and stability characteristics between the audio frame and the previous audio frame in real time on the terminal side to obtain the frame feature results.

[0043] Furthermore, on the end side, audio frames are traversed in chronological order. Energy changes are obtained by calculating the energy of each audio frame and the energy of the previous audio frame separately and performing difference or ratio calculations. Spectral changes are obtained by performing Fast Fourier Transform on the audio frame and the previous audio frame and performing difference calculations on the corresponding frequency amplitudes, and then summing them up. Stability features are obtained by performing variance or mean square calculations on the amplitude fluctuations of the sampling points within the audio frame and performing difference calculations on the corresponding results of the previous audio frame. Energy changes, spectral changes, and stability features are associated, encapsulated, and output as frame feature results.

[0044] S3.3. Perform comprehensive calculation on the frame feature results to obtain the speech activity judgment value, and integrate the judgment and recognition of speech activity frames and non-speech frames to generate speech activity detection results.

[0045] Furthermore, the frame feature results are read frame by frame in the audio frame time sequence. The comprehensive calculation is performed by performing a dimensionless operation on energy change, spectrum change and stability features and performing weighted summation or weighted average operation to obtain the speech activity judgment value. The judgment and recognition is performed by comparing the speech activity judgment value with the boundary conditions to obtain the speech activity frame label or non-speech frame label. The judgment and recognition of speech activity frames and non-speech frames are integrated by merging adjacent speech activity frames with consistent labels and retaining the non-speech frame intervals, and the speech activity detection results are encapsulated in time sequence to generate speech activity detection results.

[0046] S3.4. Divide the speech activity detection results into speech activity intervals, extract audio frame data from the speech activity intervals, and trigger streaming automatic speech recognition for decoding processing through the audio frame data to obtain candidate sequences for transcription.

[0047] Furthermore, the speech activity detection results are divided into speech activity intervals by traversing the speech activity detection results by timestamp and recording the timestamp range of consecutive speech activity frames as speech activity intervals. The speech activity interval is written with a start timestamp and an end timestamp. Audio frame data is extracted from the speech activity interval by retrieving the corresponding audio frames in the audio frame segment using the start timestamp and end timestamp of the speech activity interval and concatenating them in chronological order to form audio frame data. The audio frame data triggers streaming automatic speech recognition for decoding processing by continuously inputting the audio frame data into streaming automatic speech recognition and outputting multiple candidate transcription text sequences. The multiple candidate transcription text sequences are then aggregated to generate a candidate transcription sequence.

[0048] S3.5. Filter and confirm the candidate sequences for transcription in chronological order, and extract the decoded content from the candidate sequences to obtain partial transcription results.

[0049] Furthermore, the candidate transcription sequences are read sequentially according to time and arranged according to the start and end timestamps of the speech activity interval. Filtering and confirmation are performed by comparing multiple candidate transcription sequences character by character at the same time position and retaining the candidate transcription sequence with the highest time sequence and the highest content consistency. The process of determining the highest content consistency is completed by counting the number of times multiple candidate transcription sequences appear at the same prefix position and selecting the prefix content with the most repetitions. The confirmation process is completed by marking the prefix content with a stable repetition count as confirmed content and retaining the tail content with a stable count as content to be confirmed. Decoded content extraction is performed by extracting text sequences from the confirmed content and splicing them in time sequence to form continuous text fragments, and outputting the continuous text fragments to obtain partial transcription results.

[0050] S3.6. Merge the partial transcription results with the input text to obtain the user's expressed text, and perform a preliminary coarse classification of the user's expressed text to obtain the preliminary intent category and confidence level.

[0051] Furthermore, by comparing the transcribed result with the input text, duplicate content is identified and removed, and only unique parts are retained. The remaining parts are then concatenated in sequence to form the user's expressed text, ensuring the integrity of the information. Word segmentation is then performed to extract keywords and phrases, and the similarity between the user's expressed text and predefined intent categories is calculated to determine the most matching intent category. Based on the similarity score, the intent category with the highest matching degree is selected, and the corresponding confidence score is recorded to reflect the reliability of the classification.

[0052] S3.7. Encapsulate and output the preliminary intent category and confidence level to generate intent decision information.

[0053] Furthermore, the initial intent category and corresponding confidence score are calculated and packaged into intent decision information according to a set format. The intent category field is mapped one-to-one with its corresponding confidence score to ensure the reliability of the classification results. These two fields are arranged in order to form complete decision information. The intent decision information also includes the ranking results of candidate coarse intents, slot filling values, and missing slots to ensure that further decisions can be made based on this information during subsequent processing. The packaged intent decision information is then output as the basis for subsequent interaction or action decisions.

[0054] S4. Extract semantic clues from user-expressed text and intent decision information, integrate user perspective, topological adjacency and anomaly alarms to obtain multi-source localization evidence features, and input them into a learnable fusion localization model to generate entity localization results.

[0055] S4.1 Extract semantic cues related to objects, behaviors, and spaces from the partial transcription results, and align them with intention decision information to obtain structured semantic constraints.

[0056] Furthermore, some transcription results are read in chronological order and subjected to Chinese word segmentation and part-of-speech tagging. Chinese word segmentation and part-of-speech tagging are performed by dividing the partial transcription results into word sequences and assigning a part-of-speech category to each word in the word sequence. Object-related semantic clues are obtained by locating nouns and noun phrases in the word sequence and extracting text fragments. Behavior-related semantic clues are obtained by locating verbs and verb phrases in the word sequence and extracting text fragments. Spatial-related semantic clues are obtained by locating locator words, location nouns, and combinations of locator words and location nouns in the word sequence and extracting text fragments. Intent decision information alignment is performed by matching string and inclusion relationships between object-related semantic clues, behavior-related semantic clues, and spatial-related semantic clues and intent decision information, and the matching results are output. The matching results are associated and encapsulated with object-related semantic clues, behavior-related semantic clues, and spatial-related semantic clues to generate structured semantic constraints.

[0057] S4.2 Based on structured semantic constraints, multi-source constraint fusion is performed by combining user perspective, topological adjacency and anomaly alarm to obtain multi-source evidence scores. The multi-source evidence scores and structured semantic constraints are then substituted into the integration formula, and multi-source localization evidence features are obtained through nonlinear fractional operations.

[0058] Furthermore, multi-source constraint fusion calculates the constraint matching degree of candidate twin entity sets based on structured semantic constraints. The user perspective constraint matching degree is obtained by performing visibility determination on the spatial relationship between the user perspective and the candidate twin entity sets. The topological adjacency constraint matching degree is obtained by retrieving the topological adjacency relationship of the candidate twin entity sets and scoring the cases where the topological adjacency relationship satisfies the structured semantic constraints. The anomaly alarm constraint matching degree is obtained by retrieving the anomaly alarm records in the corresponding time range of the candidate twin entity sets and scoring the correlation between the anomaly alarm records and the structured semantic constraint matching results. Multi-source constraint fusion normalizes the user perspective constraint matching degree, topological adjacency constraint matching degree, and anomaly alarm constraint matching degree and summarizes them to obtain a multi-source evidence score. The integration formula substitutes the multi-source evidence score and structured semantic constraints and performs nonlinear fraction operation. The nonlinear fraction operation generates numerator and denominator terms and performs division of the numerator term by the denominator term to output multi-source localization evidence features. The formula for calculating the fusion positioning score is:

[0059] in, This represents the fusion positioning score, with the value falling into... A larger value indicates that the candidate entity is more consistent with multi-source evidence; The index number indicating the type of evidence; Indicates the first The normalized scores of all pieces of evidence fell into the range of [missing information]. ; This represents a user-perspective evidence score. Indicates the topological adjacency evidence score; Indicates the score for abnormal alarm evidence; Indicates the first The weight coefficients of each piece of evidence are determined by calibration set grid search or cross-validation; This represents an exponential function used to non-linearly enhance evidence scoring.

[0060] S4.3 Organize the multi-source localization evidence features according to the timestamp and the corresponding candidate twin entities in the candidate twin entity set, obtain the aligned feature sequence, and calculate the fusion localization score of each candidate twin entity in the candidate twin entity set by using a standardized input learnable fusion localization model.

[0061] Furthermore, the multi-source localization evidence features are organized according to the timestamp and correspond to each candidate twin entity in the candidate twin entity set. At each timestamp position, a feature record is written with the timestamp and the candidate twin entity identifier as the index. The feature records of the same candidate twin entity are concatenated in the order of timestamp to form an aligned feature sequence. A training supervision correspondence is established between the target candidate twin entity label and the aligned feature sequence. The target candidate twin entity label comes from manually labeled records or hit result records in historical business logs. The standardized input is obtained by calculating the feature mean and feature standard deviation of the aligned feature sequence according to the feature dimension and performing a standardization transformation to obtain a standardized aligned feature sequence. The standardized aligned feature sequence is input into the learnable fusion localization model and outputs the fusion localization score of each candidate twin entity in the candidate twin entity set.

[0062] It should be noted that the construction steps of the learnable fusion localization model are as follows: An input layer is established and the weights of each multimodal feature are initialized. A fusion layer is constructed, and different multimodal features are processed by weighting. A nonlinear activation function is used to extract the relationship between multimodal features. By using a spatiotemporal estimation layer to capture temporal and spatial dependencies, spatiotemporal positioning results are obtained and combined with risk indicators calculated from confidence and risk analysis to make decisions, thus constructing a learnable fusion positioning model.

[0063] Multimodal positioning data is collected and aligned (timestamp synchronization, denoising and missing data handling, coordinate system unification). Each modal input is extracted into a fixed-length representation through a feature encoding network, and auxiliary features reflecting the observation quality are introduced. In the fusion layer, learnable static or dynamic weights (such as gating / attention) are set for different modal features. The fusion is completed through weighted summation, weighted concatenation, etc., and nonlinear transformation is used to characterize the complementarity and conflict between modalities. In the spatiotemporal estimation layer, temporal modeling and spatial constraints (kinematics / map / geometric consistency) are used to output positioning results at continuous time steps, and confidence or uncertainty is given at the same time. Decisions are made by combining the confidence and risk indicators obtained from risk analysis (threshold determination, degradation, reinitialization, or alarm output). Through joint loss training and deployment optimization with positioning error, confidence calibration, and risk constraints as the core, a learnable fusion positioning model with adaptive fusion and risk-controllable output capabilities is formed.

[0064] The training process of the learnable fusion localization model uses aligned feature sequences and target candidate twin entity labels from historical samples as training inputs and supervision information. During the training phase, the aligned feature sequences are input into the learnable fusion localization model to obtain fusion localization score prediction results. The fusion localization score prediction results and target candidate twin entity labels are used to calculate the loss value. The loss value is calculated by applying an increasing constraint to the fusion localization score prediction results corresponding to the target candidate twin entities and a decreasing constraint to the fusion localization score prediction results corresponding to the non-target candidate twin entities to form a differentiable loss function. Parameter updates are performed by backpropagating the loss function to calculate the gradient and using gradient descent-like algorithms to iteratively update the parameters of the learnable fusion localization model. The iterative update repeats the calculation of fusion localization score prediction results and loss value after each update until the loss value converges or the stopping condition is met.

[0065] S4.4 Sort and filter the fusion localization scores, determine the final priority order of candidate twin entities based on the fusion localization scores, and generate entity localization results.

[0066] Furthermore, the fused positioning score is read one-to-one with the candidate twin entities in the candidate twin entity set to form a pairing record of candidate twin entity identifier and fused positioning score. Sorting is performed by comparing the fused positioning scores in the pairing records and arranging them from high to low to obtain a sorted list. Filtering is performed by removing candidate twin entities whose fused positioning scores are lower than the filtering criteria in the sorted list and retaining candidate twin entities that meet the filtering criteria. The filtering criteria are that the fused positioning score is not lower than the adaptive filtering threshold determined by the score distribution. Priority order is determined by using the order in which the candidate twin entities are retained in the sorted list as the final priority order of the candidate twin entities. The candidate twin entity identifier, fused positioning score, and final priority order are associated and encapsulated, and the associated encapsulation output generates the entity positioning result.

[0067] It should be noted that the adaptive filtering threshold range for the score distribution is as follows: when the fusion positioning score itself is between 0 and 1, the adaptive filtering threshold range is between 0 and 1; to avoid filtering failure due to the threshold being too high or too low, the adaptive filtering threshold for the score distribution can be limited to between 0.1 and 0.9.

[0068] S5. Assess the calibrable uncertainty of the confidence level and action risk level of the entity positioning results, and perform confidence judgment based on Conformal confidence gating to generate interactive decision results.

[0069] S5.1 Extract the confidence level and action risk level of the positioning results from the candidate twin entities as evaluation objects, and integrate and standardize the evaluation objects to obtain standardized fusion feature vectors.

[0070] Furthermore, candidate twin entities are read one by one according to the final priority order in the entity positioning results. The confidence level and action risk level of the positioning results are extracted from the candidate twin entities as evaluation objects. The confidence level field and action risk level field of the positioning results in the associated record of each candidate twin entity are used to form an evaluation object set. The integration processing is performed by concatenating the confidence level value and action risk level value of the positioning results in a fixed field order to form a fused feature record. The standardization processing is performed by calculating the feature mean and feature standard deviation of the confidence level value and action risk level value of the positioning results respectively, and performing the numerical transformation of "feature value minus feature mean and then divided by feature standard deviation" to obtain standardized values. The standardized values ​​are concatenated in a fixed field order to output a standardized fused feature vector.

[0071] S5.2. The standardized fusion feature vectors are weighted and aggregated and processed by a nonlinear mapping function to generate confidence risk values.

[0072] Furthermore, the standardized fusion feature vector includes standardized confidence values ​​and standardized action risk level values ​​of the positioning results. Weighted aggregation assigns weights to the standardized confidence values ​​and standardized action risk level values ​​of the positioning results, performs multiplication operations on the standardized confidence values ​​and standardized action risk level values ​​of the positioning results with their corresponding weights, and then performs addition operations on the two multiplication results to obtain the aggregated scalar. Nonlinear mapping function processing inputs the aggregated scalar into a nonlinear mapping function and performs exponential and fractional operations on the aggregated scalar to obtain the output scalar, which serves as the confidence risk value.

[0073] It should be noted that the standardized values ​​of the confidence level and the standardized values ​​of the action risk level in the positioning results are assigned weights separately, and the sum of the weights is equal to one; the aggregate scalar is obtained by multiplying the two types of standardized values ​​by their corresponding weights and then adding them together; the nonlinear mapping function first performs an exponential operation on the aggregate scalar and then a fractional operation to obtain the output scalar, which serves as the confidence risk value and ranges from zero to one; the weights are determined by traversing the candidate values ​​on the calibration samples and selecting the value with the minimum misjudgment cost.

[0074] S5.3. The confidence risk value is interactively determined through Conformal confidence gating, clarification prompts are output to the user and user feedback information is received, and a gating confidence score is generated.

[0075] Furthermore, Conformal confidence gating interactively determines the confidence risk value. The Conformal calibration calculation expression is as follows: ; in, This represents the inconsistency score corresponding to the confidence risk value. express The non-consistency score of the calibration sample; This indicates the number of calibration samples in the historical interaction record. This represents the normalized denominator, which is used to map the counting result to a numerical range of zero to one. Indicates the gate confidence score; This indicates an indicator function. The indicator function takes the value 1 when the condition inside the parentheses is true, and takes the value 0 when the condition inside the parentheses is false.

[0076] The judgment result is obtained by comparing the confidence risk value with the Conformal confidence gating threshold. When the judgment result indicates that clarification is needed, a clarification prompt is output to the user and the user feedback information is received. The user feedback information and the corresponding content of the clarification prompt are matched and compared to obtain a consistency mark. The judgment result and the consistency mark are applied to the confidence risk value to complete the gating correction and output the gating confidence score.

[0077] It should be noted that the Conformal confidence gate threshold is determined by taking the corresponding quantile (i.e., taking the quantile) of the confidence risk (inconsistency) score calculated on the calibration set according to the preset confidence level. The value is adaptively changed according to the score definition used and the distribution of calibration data (if the p-value form is used, the Conformal confidence gate threshold is directly taken as the preset significance level).

[0078] S5.4 When the gate confidence score is lower than the risk suppression threshold, a clarification question-and-answer mechanism is triggered to generate a highlighted mark.

[0079] Furthermore, by comparing the gate confidence score with the risk suppression threshold, a result lower than the judgment result is obtained. If the result is lower than the judgment result, a clarification question and answer mechanism is triggered. The clarification question and answer mechanism generates the question content that needs clarification based on the result lower than the judgment result and outputs the option content to be confirmed by the user. At the same time, the clarification question and answer mechanism locates the scope of content that needs to be focused on corresponding to the question content and writes it into a highlight mark. The highlight mark is used to mark the correlation between the question content and the option content output by the clarification question and answer mechanism.

[0080] It should be noted that the clarification question-and-answer mechanism is used to automatically generate questions and candidate options that need clarification when the gating confidence score is lower than the risk suppression threshold, based on the uncertainty of entities, intentions or parameters in the current interaction. The questions and candidate options are sorted and output according to the hit distance, semantic matching, topological adjacency and alarm priority to guide users to confirm key information. At the same time, the twin objects or interface ranges corresponding to the questions and options are located and highlighted to indicate the associated location. After the user confirms, the evidence is filled back and the decision result is updated. The risk suppression threshold is represented by two scalars: a clarification trigger threshold and a release threshold. Both scalars fall between zero and one, and the clarification trigger threshold is less than the release threshold. When the gate confidence score is lower than the clarification trigger threshold, the clarification question-and-answer mechanism is triggered. When the gate confidence score is greater than or equal to the release threshold, the interactive decision is released. This setting can ensure timely intervention when the confidence is low, so as to avoid the risk of wrong decisions or low-quality results due to excessive uncertainty.

[0081] S5.5 When the gate confidence score is higher than the risk suppression threshold but lower than the risk release threshold, the observation and confirmation mechanism is triggered, and a pending confirmation status is generated.

[0082] Furthermore, by comparing the gating confidence score with the risk suppression threshold and the gating confidence score with the risk release threshold, an interval determination result is obtained. The interval determination result triggers a wait-and-see confirmation mechanism. Based on the interval determination result, the wait-and-see confirmation mechanism retains the entity location result corresponding to the current gating confidence score and generates a wait-and-see confirmation record. The wait-and-see confirmation record is written with the candidate twin entity identifier, fused location score, gating confidence score, and information items that need to be further confirmed. The wait-and-see confirmation mechanism marks the wait-and-see confirmation record as pending confirmation and outputs the pending confirmation status for subsequent confirmation processing.

[0083] It's important to note that the risk approval threshold is typically set relative to the risk suppression threshold. Its purpose is to ensure that decisions are approved when the confidence level is above a certain point, avoiding excessive intervention. The risk approval threshold is usually set above the risk suppression threshold to ensure that the system can make a decision directly when the confidence level is sufficiently high, without triggering a clarification mechanism. Its value range is usually within the [0,1] interval. A common setting is between 0.7 and 0.9, meaning that when the gate confidence score is above this threshold, the decision will be approved directly without further clarification or intervention.

[0084] S5.6 When the gate confidence score is higher than the risk release threshold, the interactive decision generation mechanism is triggered, and an interactive decision value is generated through a nonlinear fractional calculation method.

[0085] Furthermore, by comparing the gate confidence score with the risk release threshold, a result higher than the judgment result is obtained. This triggers an interactive decision generation mechanism. The interactive decision generation mechanism reads the gate confidence score and associates it with the candidate twin entity identifier and fused positioning score from the entity positioning result as decision inputs. The nonlinear fractional calculation method performs numerator and denominator calculations on the decision inputs. The numerator is obtained by performing multiplication or nonlinear transformation and addition on the gate confidence score and the fused positioning score. The denominator is obtained by performing addition on the gate confidence score and the fused positioning score and superimposing a stable term. The numerator is divided by the denominator to output the interactive decision value, which serves as the result generated by the interactive decision generation mechanism.

[0086] S5.7 Prioritize the interactive decision values. Under the condition of the same risk level, arrange them from high to low interactive decision values, and prioritize the processing of high-risk tasks with high decision values. Format the sorted tasks and display them on the large screen in real time to generate interactive decision results.

[0087] Furthermore, interactive decision values ​​are paired with task identifiers and risk levels to form a one-to-one correspondence. Priority is first sorted by risk level from high to low, and then, under the condition of the same risk level, by interactive decision values ​​from high to low. The task processing order is selected and processed sequentially according to the final sorted list. The sorted task data will be formatted, and the task identifier, risk level, and interactive decision value will be encapsulated into a standardized data format and updated to the large screen in real time through a real-time push system to generate interactive decision results, ensuring that the task sorting displayed on the large screen is always synchronized with the latest decision results.

[0088] In summary, this invention achieves rapid identification of candidate digital twin entities and spatiotemporal one-way hits by: unifying timestamp alignment and mapping user pointers to digital twin 3D coordinates, thereby reducing ambiguity and improving interaction accuracy; further, it aggregates entity-related state data to encapsulate a multimodal digital twin alignment package, combining edge-side speech detection, streaming transcription, and coarse intent classification to form traceable contextual input, enhancing real-time understanding and robustness; in particular, it integrates semantic cues, user perspective, topological adjacency, and anomaly alerts to generate multi-source evidence and input it into a learnable localization model, improving the reliability of digital twin entity localization; and it enhances the security and controllability of digital twin decision-making by triggering clarification, confirmation, and decision-making through calibrable uncertainty assessment and conformal confidence gating.

[0089] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for multimodal understanding and interaction of large models for data dashboards, characterized by: include, Collect real-time status data and perform unified timestamp alignment, map the user-selected pointing information to the twin 3D coordinate system, obtain the hit results, and map the hit results to a set of candidate twin entities; Extract and aggregate entity association state data from the candidate twin entity set, and encapsulate it to generate a multimodal twin alignment package; End-side speech activity detection is performed on the speech audio stream in the multimodal twin alignment package, and streaming automatic speech recognition is triggered to obtain partial transcription results. Intent decision information is generated through preliminary intent coarse classification. Semantic cues are extracted from user-expressed text and intent decision information, and user perspective, topological adjacency and anomaly alarms are integrated to obtain multi-source localization evidence features. These features are then input into a learnable fusion localization model to generate entity localization results. The confidence level and action risk level of the entity positioning results are assessed for calibrable uncertainty, and confidence judgment is performed based on Conformal confidence gating to generate interactive decision results.

2. The method for large-scale model multimodal understanding and interaction for data dashboards as described in claim 1, characterized in that: The specific steps for obtaining the hit results and mapping them to a set of candidate twin entities are as follows. Collect real-time status data, establish a unified clock reference, attach a collection time mark to each piece of real-time status data, calculate the fixed offset of the real-time status data relative to the unified clock reference, and obtain the timestamp status sequence. Perform continuous reconstruction on the timestamp state sequence, obtain the state trajectory, and couple and bind the user-selected pointing information with the state trajectory according to the time interval of occurrence to obtain the pointing state association result; The pointing state association result is mapped to the twin 3D coordinate system, the pointing trajectory is obtained and the timestamp corresponding to the twin 3D coordinate is used to calculate the hit result, and the hit result is filtered to form a candidate twin entity set.

3. The method for large-scale model multimodal understanding and interaction for data dashboards as described in claim 1, characterized in that: The encapsulation process generates a multimodal twin-aligned package, and the specific steps are as follows. Extract entity association state data from the candidate twin entity set according to the timestamp, perform aggregation calculation on the entity association state data under a unified clock reference, and obtain aggregated state data; The aggregated state data is mapped one-to-one with the timestamp and then associated and encapsulated to generate a multimodal twin alignment package.

4. The method for large-scale model multimodal understanding and interaction for data dashboards as described in claim 3, characterized in that: The specific steps for performing end-side speech activity detection on the speech audio stream in the multimodal twin alignment package are as follows: On the device side, a continuous speech audio stream is parsed from the multimodal twin alignment packet, and the speech audio stream is uniformly sampled and framed to obtain audio frames. On the device side, the energy change, spectrum change and stability characteristics between the audio frame and the previous audio frame are calculated in real time to obtain the frame feature results. The frame feature results are comprehensively calculated to obtain the speech activity judgment value, and the judgment and recognition of speech activity frames and non-speech frames are integrated to generate speech activity detection results.

5. The method for large-scale model multimodal understanding and interaction for data dashboards as described in claim 1, characterized in that: The specific steps for generating the intent decision information are as follows: The speech activity detection results are divided into speech activity intervals, audio frame data is extracted from the speech activity intervals, and streaming automatic speech recognition is triggered to perform decoding processing to obtain transcription candidate sequences. Candidate sequences for transcription are filtered and confirmed in chronological order, and the decoded content in the candidate sequences is extracted to obtain partial transcription results; The partial transcription results are merged with the input text to obtain the user's expressed text, and a preliminary coarse classification of the user's expressed text is performed to obtain the preliminary intent category and confidence level. The initial intent category and confidence level are encapsulated and output to generate intent decision information.

6. The method for large-scale model multimodal understanding and interaction for data dashboards as described in claim 1, characterized in that: The specific steps for obtaining multi-source localization evidence features are as follows. Extract semantic cues related to objects, behaviors, and spaces from partial transcription results and align them with intent decision information to obtain structured semantic constraints; Based on structured semantic constraints, multi-source constraint fusion is performed by combining user perspective, topological adjacency and anomaly alarms to obtain multi-source evidence scores. The multi-source evidence scores and structured semantic constraints are then substituted into the integration formula, and multi-source localization evidence features are obtained through nonlinear fractional operations.

7. The method for large-scale model multimodal understanding and interaction for data dashboards as described in claim 1, characterized in that: The specific steps for generating entity location results are as follows: The multi-source localization evidence features are organized according to timestamps and corresponding candidate twin entities in the candidate twin entity set. Alignment feature sequences are obtained and a learnable fusion localization model is obtained through standardized input. The fusion localization score of each candidate twin entity in the candidate twin entity set is calculated. The fusion localization scores are sorted and filtered, and the final priority order of candidate twin entities is determined according to the fusion localization scores to generate entity localization results.

8. The method for large-scale model multimodal understanding and interaction for data dashboards as described in claim 7, characterized in that: The construction steps of the learnable fusion localization model are as follows: An input layer is established and the weights of each multimodal feature are initialized. A fusion layer is constructed, and different multimodal features are processed by weighting. A nonlinear activation function is used to extract the relationship between multimodal features. By using a spatiotemporal estimation layer to capture temporal and spatial dependencies, spatiotemporal positioning results are obtained and combined with risk indicators calculated from confidence and risk analysis to make decisions, thus constructing a learnable fusion positioning model.

9. The method for large-scale model multimodal understanding and interaction for data dashboards as described in claim 7, characterized in that: The specific steps for assessing the calibrable uncertainty of the confidence level and action risk level of the entity positioning results are as follows. The confidence level and action risk level of the positioning results are extracted from the candidate twin entities as evaluation objects, and the evaluation objects are integrated and standardized to obtain a standardized fusion feature vector. The standardized fusion feature vectors are weighted and aggregated, and then processed by a nonlinear mapping function to generate confidence risk values.

10. The method for large-scale model multimodal understanding and interaction for data dashboards as described in claim 1, characterized in that: The specific steps for generating interactive decision results are as follows: The confidence risk value is interactively determined by Conformal confidence gating, which outputs clarification prompts to users and receives user feedback information, and generates a gating confidence score. When the gate confidence score is lower than the risk suppression threshold, a clarification question-and-answer mechanism is triggered, generating a highlighted mark. When the gate confidence score is higher than the risk suppression threshold but lower than the risk release threshold, the observation and confirmation mechanism is triggered, and a pending confirmation status is generated. When the gate confidence score is higher than the risk release threshold, the interactive decision generation mechanism is triggered, and an interactive decision value is generated through a non-linear fractional calculation method. The interactive decision values ​​are prioritized and sorted from high to low under the condition of the same risk level. High-risk tasks with high decision values ​​are processed first. The sorted tasks are formatted and displayed on the big screen in real time to generate interactive decision results.

Images