Multi-modal index interpretability enhancement method and system

Abstract

The invention relates to the technical field of artificial intelligence and man-machine interaction, and discloses a multi-modal index interpretability enhancement method and system. The method comprises the following steps: acquiring a voice instruction, a DOM (Document Object Model) structure snapshot and front-end burying point data of a user in a Web system operation process; executing cross-modal attention fusion by adopting a modal guide residual error alignment mechanism; constructing an explaining causal chain graph and carrying out deviation detection; semantic perturbation mapping and structured repair generation are executed; and outputting the multi-modal interpretation content set. Compared with the prior art, the technical problem that cross-modal causal chain construction and structured code repair suggestions cannot be realized in abnormal scenes of single-modal indexes, especially multi-modal user behavior data including click streams, form filling and voice command driving is solved. Due to the fact that the modal guide residual error alignment mechanism is introduced, the interpretation reachability and the code repair performability of the model prediction result in the Web system are improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and human-computer interaction technology, and in particular to a method and system for enhancing the interpretability of multimodal metrics. Background Technology

[0002] Currently, in contemporary web application systems, the interpretability of user behavior data is increasingly becoming a core requirement for intelligent interaction systems, anomaly detection systems, and model operation and maintenance platforms. Especially in scenarios focused on business optimization, user behavior analysis, and anomaly tracking, building mechanisms capable of interpreting model-predicted behavior and tracing user operational intentions has become an important research direction.

[0003] However, current mainstream interpretability methods mainly rely on single-modal input data, such as front-end event logs, page DOM structure, or speech recognition text, and generate static index attribution results through shallow model interpretation methods such as SHAP and LIME. These methods have the following significant shortcomings: (1) Strong modal isolation: In actual operation, users often complete tasks through multimodal means such as voice control, mouse clicks, and form input. The interpretation results from a single modal perspective often cannot restore the real causal behavior chain. (2) Difficulty in time alignment: The data collection frequency of each modality is different (for example, voice frame sampling is at the ms level, while front-end event sampling is at the event level), resulting in a lack of alignment mechanism in the time dimension and misalignment of the interpretation path. (3) Severe context fragmentation: Existing interpretation methods have difficulty maintaining the consistency of contextual logic between behaviors, especially in complex interaction paths such as user page jumps, nested component triggers, and semantically ambiguous command inputs, and cannot construct a complete "semantic path". For example, in an e-commerce system, a user searches for "red office chair with USB interface" by voice and then clicks on multiple irrelevant results. Traditional tracking cannot identify the semantic gap between voice and clicks. In a medical decision support platform, when doctors use natural language to query patient history records and operate complex nested components, it is impossible to infer the causal relationship between interface hierarchy and semantic intent. Therefore, there is an urgent need to propose a multimodal indicator interpretability enhancement method that can integrate speech, front-end tracking, and DOM structure multimodal interpretability enhancement mechanisms. While maintaining temporal consistency and semantic integrity, it can establish a cross-modal causal chain graph and, on this basis, realize counterfactual reasoning and code-level structured repair generation for abnormal prediction behavior, so as to improve the interpretability of Web systems, the efficiency of anomaly debugging, and the closed-loop capability of development and operation feedback. Summary of the Invention

[0004] To address the aforementioned technical shortcomings, the purpose of this invention is to propose a method for enhancing the interpretability of multimodal metrics. This method aims to solve the technical problem that existing technologies rely solely on single-modal metrics, particularly in abnormal scenarios involving multimodal user behavior data such as clickstreams, form filling, and voice command-driven data, where cross-modal causal chain construction and structured code repair suggestions are impossible.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: The present invention provides a method for enhancing the interpretability of multimodal indicators. The methods for enhancing the interpretability of multimodal metrics include: Step S10: Obtain the voice command text, front-end tracking data, and page DOM snapshot data generated by the user during the web application interaction at time t; based on the voice command text, front-end tracking data, and page DOM snapshot data, perform a multimodal event synchronization mapping task using an adaptive time-weighted sliding window alignment mechanism, and output the multimodal synchronization event window. ; Step S20: Based on multimodal synchronization event window A multi-semantic residual alignment mechanism based on modality-guided weights is used to perform a cross-modal attention aggregation task, generating a unified cross-modal representation vector X; Step S30: Based on the unified cross-modal representation vector X, a path reasoning and deviation detection mechanism based on semantic graph structure is used to perform causal chain analysis task and output cross-modal interpretation chain graph; Step S40: Based on the cross-modal interpretation chain graph, perform the counterfactual reasoning task of the abnormal path using the information entropy inversion principle based on semantic constraints, and output the main path of the abnormal interpretation; Step S50: Based on the main path of anomaly interpretation, a semantic perturbation mapping mechanism is used to perform a structured code modification task and output the final set of interpretation content.

[0006] Preferably, in step S10, the voice command text, front-end tracking data, and page DOM snapshot data generated by the user during the interaction with the Web application at time t are acquired; based on the voice command text, front-end tracking data, and page DOM snapshot data, a sliding window alignment mechanism with adaptive time weights is used to perform a multimodal event synchronization mapping task, and a multimodal synchronization event window is output. The steps specifically include: Step S101: Obtain the voice command text, front-end tracking data, and page DOM snapshot data generated by the user during the interaction with the web application at time t; construct a voice modality set based on the voice command text using a BERT-embedded temporal semantic clustering method. A click modality set is constructed based on front-end event tracking data using a sequence time window segmentation method based on event type tags. A DOM state set is constructed based on the node change extraction method using structural difference measurement based on the page DOM snapshot data. ; Among them, the speech modal set Including voice modal timestamps Click on the modal set Including click modal timestamps DOM state collection Includes DOM state timestamps ; Step S102: Construct adaptive time constraints, which satisfy the following: ,in, Uses the default time window as the reference. Step S103: For the speech modality set Click modal set and DOM state collection A multimodal synchronization event window is constructed using a sliding window alignment mechanism based on adaptive time constraints. .

[0007] Preferably, in step S20, based on the multimodal synchronization event window The steps for performing a cross-modal attention aggregation task and generating a unified cross-modal representation vector X using a multi-semantic residual alignment mechanism based on modality-guided weights specifically include: Step S201: Use the BERT model to synchronize event windows from multiple modalities The speech embedding vector is extracted and used... Model from multimodal synchronization event window The click embedding vector is extracted and used... Model from multimodal synchronization event window Extract the DOM structure embedding vector; and set the first modality guiding factor for the speech embedding vector. Set a second modality guiding factor for the click embedding vector. Set a third modality guiding factor for the DOM structure embedding vector. ;in, ; Step S202: Based on the speech embedding vector, click embedding vector, and DOM structure embedding vector, perform a cross-modal attention semantic fusion task using the scaling dot product attention principle, and output the cross-modal semantic fusion result; Step S203: Based on the first mode guiding factor Second mode guiding factor With the third mode guiding factor A modally guided residual representation vector is constructed using a combined mechanism of weighted residual fusion and gated suppression. And modally guided residual representation vector The cross-modal semantic fusion result is subjected to dynamic harmonic processing based on bidirectional gated nonlinear transformation to generate a unified cross-modal representation vector X.

[0008] Preferably, step S30, which involves performing causal chain analysis based on a path reasoning and deviation detection mechanism using a semantic graph structure based on a unified cross-modal representation vector X, and outputting a cross-modal explanation chain graph, specifically includes: Step S301: Based on the unified cross-modal representation vector X, construct a heterogeneous semantic graph G using Python's networkx and geometric libraries. Each node in the heterogeneous semantic graph G represents a semantic unit under a different modality. Semantic units include text entities, image regions, and code snippets. The weight of each edge in the heterogeneous semantic graph G represents semantic similarity. Based on the heterogeneous semantic graph G, combine the offset node2vec random walk method to sample the set of semantic paths within the modality and output the initial path set P. Step S302: Calculate the bias propagation strength score of each path in the initial path set P using a path dependency scoring method based on graph attention mechanism, and remove paths in the initial path set P whose bias propagation strength scores are less than a preset bias propagation strength score threshold, outputting an optimized local subgraph. ; Step S303: Based on the optimized local subgraph A shortest path-based counterfactual analysis method is used to perform semantic bias chain reconstruction tasks, and outputs a cross-modal explanation chain graph.

[0009] Preferably, in step S303, based on the optimized local subgraph The steps for performing semantic bias chain reconstruction using shortest path-based counterfactual analysis and outputting a cross-modal explanatory chain graph specifically include: optimizing local subgraphs. The node corresponding to the path with the highest bias propagation strength score is taken as the starting node. Based on the starting node and semantic similarity, the counterfactual path modeling principle is used to search for the shortest causal path based on graph structure and semantic coupling, and a set of shortest causal paths is constructed. Finally, a cross-modal explanatory chain graph is generated based on the set of shortest causal paths. The cross-modal explanatory chain graph is used to represent the path propagation mode, the main triggering nodes and the causal jump relationship.

[0010] Preferably, step S40, which involves performing counterfactual reasoning on anomaly paths based on the information entropy inversion principle using semantic constraints according to the cross-modal interpretation chain graph, and outputting the main path of anomaly interpretation, specifically includes: Step S401: Select the k-th cross-modal interpretation chain from the cross-modal interpretation chain diagram. For the k-th cross-modal interpretation chain Semantic information entropy is calculated using the weighted normalized probability modeling method. Semantic information entropy Used to indicate the uneven distribution of the degree of abnormality among nodes in a path under semantic consistency constraints; Step S402: Based on semantic information entropy The path local entropy mutation rate was calculated using the window difference analysis method. Select the path local entropy mutation rate in the cross-modal interpretation chain graph. The longest path is used as the exception interpretation path; Step S403: If the anomaly explanation path and the optimized local subgraph are... If there are no overlapping nodes, the semantic deviation chain reconstruction task is performed using the shortest path-based counterfactual analysis method based on the anomaly interpretation path, and the main anomaly interpretation path is output.

[0011] Preferably, step S50, which involves performing a structured code modification task based on the semantic perturbation mapping mechanism using the main path of anomaly interpretation and outputting the final set of interpretation content, specifically includes: Step S501: Obtain the semantic node sequence based on the main path of anomaly interpretation, construct a semantic context embedding graph based on the semantic node sequence, identify anomalous mutation points using the transformers library in Python based on the semantic context embedding graph, and construct a semantic offset perturbation vector field based on the anomalous mutation points; Step S502: Perform clustering analysis and attribution in the semantic offset perturbation vector field using K-nearest neighbor clustering to form candidate repair paths; Step S503: Obtain code modality context information, which includes source code semantic graph information, AST node chain information, and function call graph information; sort the candidate repair paths based on the code modality context information and output a structured set of code modification suggestions; Step S504: Apply the code modification suggestion set and obtain the code modification result set; based on the code modification suggestion set and the code modification result set, use Python's libcst library to perform syntax-preserving abstract syntax tree interpretation processing, and output the final interpretation content set.

[0012] The present invention also provides a multimodal index interpretability enhancement system comprising: The event alignment module is used to acquire voice command text, front-end tracking data, and page DOM snapshot data generated by the user during interaction with the web application at time t. Based on the voice command text, front-end tracking data, and page DOM snapshot data, it performs a multimodal event synchronization mapping task using an adaptive time-weighted sliding window alignment mechanism, and outputs a multimodal synchronization event window. ; The cross-modal fusion module is used for multimodal synchronization event windows. A multi-semantic residual alignment mechanism based on modality-guided weights is used to perform a cross-modal attention aggregation task, generating a unified cross-modal representation vector X; The causal chain construction module is used to perform causal chain analysis tasks based on a path reasoning and deviation detection mechanism based on a semantic graph structure, using a unified cross-modal representation vector X, and outputs a cross-modal interpretation chain graph. The counterfactual reasoning module is used to perform counterfactual reasoning tasks for abnormal paths based on the information entropy inversion principle of semantic constraints according to the cross-modal interpretation chain graph, and output the main path of abnormal interpretation. The semantically driven repair module is used to perform structured code modification tasks based on the main path of anomaly explanation using a semantic perturbation mapping mechanism, and outputs the final set of explanation content.

[0013] The present invention also provides a multimodal indicator interpretability enhancement device, comprising: a memory, a processor, and a multimodal indicator interpretability enhancement program stored in the memory and executable on the processor, wherein the multimodal indicator interpretability enhancement program implements a multimodal indicator interpretability enhancement method when executed by the processor.

[0014] The present invention also provides a computer program product, including a multimodal indicator interpretability enhancement program, which, when executed by a processor, implements the multimodal indicator interpretability enhancement method.

[0015] The beneficial effects of this invention are as follows: This invention introduces a modality-guided residual alignment mechanism and a path reasoning method based on semantic graph structure, which can establish a unified cross-modal semantic representation among multi-source heterogeneous modalities such as voice commands, page structures and embedded data, and construct an explanatory causal chain graph with contextual coherence, which significantly improves the model's ability to understand and explain complex user operation behaviors.

[0016] This invention uses semantic perturbation mapping and information entropy inversion mechanisms to controllably reconstruct abnormal paths, and combines DOM snapshots and front-end state context to generate structured code modification suggestions, significantly improving the practicality of Web system exception prompts and development assistance efficiency. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the first embodiment of a method for enhancing the interpretability of multimodal indices according to the present invention.

[0019] Figure 2 This is a schematic diagram of the device for a method to enhance the interpretability of multimodal indicators according to the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Example 1: As Figure 1 The diagram shown is a flowchart of the first embodiment of the multimodal index interpretability enhancement method of the present invention, which presents the first embodiment of the multimodal index interpretability enhancement method of the present invention.

[0022] In the first embodiment, the method for enhancing the interpretability of multimodal metrics includes: Step S10: Obtain the voice command text, front-end tracking data, and page DOM snapshot data generated by the user during the web application interaction at time t; based on the voice command text, front-end tracking data, and page DOM snapshot data, perform a multimodal event synchronization mapping task using an adaptive time-weighted sliding window alignment mechanism, and output the multimodal synchronization event window. ; It should be noted that the "adaptive temporal weighting sliding window alignment mechanism" refers to first calculating the temporal resolution weight coefficients between modalities based on the sampling granularity of the multimodal data source (e.g., 20ms frame level for speech, event level for event tracking, and state level for DOM snapshots). Then, the sliding window step size and window size are set, and modal data of different time sequences are resampled or interpolated and aligned within the sliding window to form an event aggregation view under a unified time base. The sliding window size and step size are dynamically adjusted according to the complexity of the current interaction context, ensuring strong temporal consistency and semantic coupling of multimodal events.

[0023] Understandably, by incorporating voice input, user clicks, and page structure status into the same event window, not only is cross-modal event analysis achieved, but a high-quality data foundation is also provided for subsequent causal path extraction and abnormal behavior attribution. Compared to traditional static modal stitching methods, this mechanism can more accurately capture the user's true intent and operation path.

[0024] Step S20: Based on multimodal synchronization event window A multi-semantic residual alignment mechanism based on modality-guided weights is used to perform a cross-modal attention aggregation task, generating a unified cross-modal representation vector X; It should be noted that the "multi-semantic residual alignment mechanism based on modality-guided weights" refers to the following: after acquiring the three types of embedding vectors—voice, click, and DOM structure—the mechanism first performs weight-guided residual modeling on the semantic vectors of each modality based on a preset or dynamically learned modality guidance factor, constructing modality difference compensation terms and semantic contribution enhancement terms. The modality guidance factor includes not only static weight parameters but also dynamically updated weight adjustment amounts based on attention response intensity, semantic saliency score, event trigger priority, and user intent relevance, used to adaptively adjust the contribution ratio of each modality in the representation learning process under different interaction scenarios.

[0025] Understandably, by introducing modality-guided residual modeling into cross-modal semantic fusion, this step can significantly improve the expressive completeness and semantic alignment of the cross-modal semantic fusion results, enabling the generated unified cross-modal representation vector X to more accurately reflect the user's true operational intent and interface state changes.

[0026] For example, in a typical form submission error repair scenario, the user first issues a voice command saying "I want to fill in a Shanghai address," then clicks on the "Shipping Address" field on the page and attempts to fill it in, but an error message pops up. In traditional methods, the click sequence and DOM snapshot might misinterpret the user's intent as an incorrect path selection. However, in this method, the semantic intent of "address modification" is reinforced through voice guidance, and the attribute hierarchy of the "province / city input box" embedded in the DOM is aligned with the click path residual. Ultimately, the unified cross-modal vector X successfully points to the root cause node of the "form structure configuration error."

[0027] Step S30: Based on the unified cross-modal representation vector X, a path reasoning and deviation detection mechanism based on semantic graph structure is used to perform causal chain analysis task and output cross-modal interpretation chain graph; It should be noted that the "path reasoning and deviation detection mechanism based on semantic graph structure" refers to first mapping the unified cross-modal representation vector X constructed in step S20 to a preset multimodal semantic graph G. The nodes of graph G represent various interface elements, user intent words, operation events, and their implicit semantic labels, while the edges represent the semantic, causal, or structural relationships between them. Subsequently, multi-hop path expansion is performed on the subgraph regions most relevant to X in the graph using a graph attention mechanism and path aggregation algorithm to construct a set of causal paths from user input to anomalies. Based on this, a path deviation detection algorithm based on information gain and cross-entropy residuals is introduced to identify anomalous nodes in the path (such as semantic drift, operation offset, component response mismatch, etc.), and finally outputs a cross-modal explanation chain graph.

[0028] Understandably, through this path reasoning and deviation detection mechanism, the original modal fusion representation can be transformed into an interpretation path with a graph structure, thereby achieving structured visual tracking of abnormal operation flows. It can not only clearly point out which page node or operation semantics has deviated, but also deduce how the deviation evolved step by step from the user input, providing a precise logical starting point and contextual support for the generation of subsequent repair suggestions.

[0029] It should be understood that, compared to traditional anomaly tracking methods that rely solely on single-modality or planar causal chains, this mechanism has three advantages: First, graph neural reasoning based on semantic graph structures supports deep correlation modeling of cross-modal semantics, which can more accurately express dynamic causal relationships such as "voice commands, UI operations, and DOM structure changes"; Second, the residual detection mechanism introduces probability statistics (such as local path entropy fluctuations) for deviation judgment, improving the ability to identify fuzzy anomalies; Third, the generated explanation chain graph has structural stability, which can maintain consistency in state tracking in multiple rounds of operations and avoid explanation drift problems.

[0030] For example, in a scenario where "a user fails to submit after using voice search + clicking a page button + entering information," traditional methods might only capture the surface symptom of "submission failure" and fail to trace the root cause. Using the explanatory chain graph mechanism of this invention, the path between "voice keyword: reimbursement address," "click node: edit address button," and "input node: address input box" is first mapped to graph G. Then, discrepancies are identified between the voice target intent node and the DOM node in their address type encoding (e.g., one points to "company address," while the other is bound to "home address"). Finally, a complete causal chain is output, from voice semantic offset → error page click → form configuration mismatch → submission failure, and key offset nodes are marked.

[0031] Step S40: Based on the cross-modal interpretation chain graph, perform the counterfactual reasoning task of the abnormal path using the information entropy inversion principle based on semantic constraints, and output the main path of the abnormal interpretation; It should be noted that the "execution of anomaly path counterfactual reasoning task based on the semantic constraint information entropy inversion principle" in this step refers to, based on the cross-modal interpretation chain graph output in step S30, firstly extracting all candidate path subgraphs marked as anomaly nodes (deviation residual scores exceeding a threshold), and constructing a semantic constraint set L, which includes user intent categories, operation context labels, semantic mapping tables, etc. Subsequently, an improved information entropy inversion algorithm is used to calculate the path entropy of each node in the path, and path counterfactual reconstruction is performed based on the "minimum entropy change" principle. That is, by minimizing the variation of causal information entropy in the path, the shortest corrected path that does not trigger anomaly nodes in the alternative paths is found. Finally, an anomaly explanation main path with minimum entropy mutation magnitude and meeting semantic constraints is output, used to identify the most critical causal relationship sequence that causes the deviation.

[0032] Understandably, this mechanism constructs a "counterfactual substitution space" in the explanation chain graph to achieve reverse reconstruction and interpretable optimization of abnormal paths. It can not only identify "what error occurred" but also deduce "how the user path would have evolved if this error had not occurred." The resulting main path of abnormal explanation has high causal compactness and semantic coherence, serving as the basic structural unit for generating subsequent repair suggestions or feedback prompts.

[0033] For example, in an exception chain diagram where "the page becomes unresponsive after a user clicks the payment button," the path marked in the explanation chain diagram is "click button" → "trigger DOM event" → "request encapsulation module" → "payment API returns 404." By performing counterfactual reasoning in this step, it is found that if the "request encapsulation module" node is replaced with "backup module interface," the information entropy in the path decreases significantly (from 1.45 to 0.76), and the backup interface is consistent with the user's payment intention (matching the semantic tag "payment"). The final output exception explanation main path is "click button" → "trigger DOM event" → "backup module interface" → "payment API returns 200," and this main path is marked as the counterfactual alternative suggestion path.

[0034] Step S50: Based on the main path of anomaly interpretation, a semantic perturbation mapping mechanism is used to perform a structured code modification task and output the final set of interpretation content.

[0035] It should be noted that the "execution of structured code modification tasks based on semantic perturbation mapping mechanism" in this step refers to: for the main path of anomaly explanation identified in step S40, combining the operation semantics, data dependencies, and context code positions corresponding to the nodes in the path, an "internal perturbation mapping matrix" is constructed to achieve intelligent recommendation of candidate modification operations. This mechanism transforms the key trigger nodes in the main path of anomalies into semantic embedding vectors and compares them with the repair fragment library in the pre-trained operation semantic model in vector space to generate a set of structured modification suggestions with confidence scores, including but not limited to function replacement, parameter correction, and anomaly branch injection. The final output set of explanation content not only includes the visualized main path of anomalies but also includes structured code patch suggestions, modification reasons, semantic consistency scores, etc., supporting subsequent developer verification and automatic repair processes.

[0036] It should be understood that traditional automated code repair methods mostly rely on syntax-level comparison based on static rules or historical patch migration, which struggles to combine the specific semantic context of the exception occurrence with the user's actual operation path, often resulting in repairs that are "syntactically feasible but semantically incompatible." This step, however, uses a semantic perturbation mechanism to simulate a local modification candidate set for the exception node in the path, selects the optimal repair suggestion based on the principle of minimum semantic drift, and provides explanatory labels and contextual scores while generating the modified fragment, thereby improving the understandability and engineering applicability of the modification suggestions.

[0037] For example, in the main path of the anomaly, a key anomaly node was identified: "The event binding function triggered after clicking the submit button is missing debouncing logic." A semantic perturbation mapping mechanism generated the following structured suggestions: 1) Replace the original event binding function with the version that includes debouncing; 2) Simultaneously, a natural language explanation was generated: "To avoid frequent clicks triggering repeated submissions, it is recommended to add a debouncing function"; 3) After comparing multiple repair candidates, the operation "using the lodash.debounce function to wrap the submit event" was given a confidence score of 92%. This structured suggestion can not only be directly used for code replacement but also automatically populated as an explanatory reason in a pull request.

[0038] Example 2: Furthermore, the multimodal indicator interpretability enhancement system provided by this invention employs a multimodal indicator interpretability enhancement method from the above embodiments, and can solve a technical problem related to multimodal indicator interpretability enhancement. Compared with the prior art, the beneficial effects of the multimodal indicator interpretability enhancement system provided by this invention are the same as those of the multimodal indicator interpretability enhancement method provided in the above embodiments, and other technical features of the multimodal indicator interpretability enhancement system are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0039] Example 3: This invention provides a device for enhancing the interpretability of multimodal indicators. Please refer to... Figure 2 A multimodal indicator interpretability enhancement device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to perform a multimodal indicator interpretability enhancement method as described in Embodiment 1 above. The multimodal indicator interpretability enhancement device in this embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. This multimodal indicator interpretability enhancement device is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this invention. A multimodal indicator interpretability enhancement device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. Random access memory 1004 also stores various programs and data required for the operation of a multimodal indicator interpretability enhancement device. Processing device 1001, read-only memory 1002, and random access memory 1004 are interconnected via bus 1005. I / O interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows a multimodal indicator interpretability enhancement device to communicate wirelessly or wiredly with other devices to exchange data. Although a multimodal indicator interpretability enhancement device with various systems is shown in the figures, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems may be implemented or possessed alternatively.

[0040] Example 4: This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the multimodal indicator interpretability enhancement method described above. The computer program product provided by this invention can solve a technical problem related to multimodal indicator interpretability enhancement. Compared with the prior art, the beneficial effects of the computer program product provided by this invention are the same as those of the multimodal indicator interpretability enhancement method provided in the above embodiments, and will not be repeated here.

[0041] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this invention.

[0042] It should be understood that the various parts disclosed in this invention can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0043] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for enhancing the interpretability of multimodal indices, characterized in that, The methods include: Step S10: Obtain the voice command text, front-end tracking data, and page DOM snapshot data generated by the user during the web application interaction at time t; based on the voice command text, front-end tracking data, and page DOM snapshot data, perform a multimodal event synchronization mapping task using an adaptive time-weighted sliding window alignment mechanism, and output the multimodal synchronization event window. ; Step S20: Based on multimodal synchronization event window A multi-semantic residual alignment mechanism based on modality-guided weights is used to perform a cross-modal attention aggregation task, generating a unified cross-modal representation vector X; Step S30: Based on the unified cross-modal representation vector X, a path reasoning and deviation detection mechanism based on semantic graph structure is used to perform causal chain analysis task and output cross-modal interpretation chain graph; Step S40: Based on the cross-modal interpretation chain graph, perform the counterfactual reasoning task of the abnormal path using the information entropy inversion principle based on semantic constraints, and output the main path of the abnormal interpretation; Step S50: Based on the main path of anomaly interpretation, a semantic perturbation mapping mechanism is used to perform a structured code modification task and output the final set of interpretation content.

2. The method for enhancing the interpretability of multimodal indicators as described in claim 1, characterized in that, In step S10, the voice command text, front-end tracking data, and page DOM snapshot data generated by the user during the web application interaction at time t are obtained; based on the voice command text, front-end tracking data, and page DOM snapshot data, a sliding window alignment mechanism with adaptive time weights is used to perform a multimodal event synchronization mapping task, and the multimodal synchronization event window is output. The steps specifically include: Step S101: Obtain the voice command text, front-end tracking data, and page DOM snapshot data generated by the user during the interaction with the web application at time t; construct a voice modality set based on the voice command text using a BERT-embedded temporal semantic clustering method. A click modality set is constructed based on front-end event tracking data using a sequence time window segmentation method based on event type tags. A DOM state set is constructed based on the node change extraction method using structural difference measurement based on the page DOM snapshot data. ; Among them, speech modal set Including voice modal timestamps Click on the modal set Including click modal timestamps DOM state collection Includes DOM state timestamps ; Step S102: Construct adaptive time constraints, which satisfy the following: ,in, Uses the default time window as the reference. Step S103: For the speech modality set Click modal set and DOM state collection A multimodal synchronization event window is constructed using a sliding window alignment mechanism based on adaptive time constraints. .

3. The method for enhancing the interpretability of multimodal indicators as described in claim 1, characterized in that, In step S20, based on the multimodal synchronization event window The steps for performing a cross-modal attention aggregation task and generating a unified cross-modal representation vector X using a multi-semantic residual alignment mechanism based on modality-guided weights specifically include: Step S201: Use the BERT model to synchronize event windows from multiple modalities The speech embedding vector is extracted and used... Model from multimodal synchronization event window The click embedding vector is extracted and used... Model from multimodal synchronization event window Extract the DOM structure embedding vector; and set the first modality guiding factor for the speech embedding vector. Set a second modality guiding factor for the click embedding vector. Set a third modality guiding factor for the DOM structure embedding vector. ;in, ; Step S202: Based on the speech embedding vector, click embedding vector, and DOM structure embedding vector, perform a cross-modal attention semantic fusion task using the scaling dot product attention principle, and output the cross-modal semantic fusion result; Step S203: Based on the first mode guiding factor Second mode guiding factor With the third mode guiding factor A modally guided residual representation vector is constructed using a combined mechanism of weighted residual fusion and gated suppression. And modally guided residual representation vector The cross-modal semantic fusion result is subjected to dynamic harmonic processing based on bidirectional gated nonlinear transformation to generate a unified cross-modal representation vector X.

4. The method for enhancing the interpretability of multimodal indicators as described in claim 1, characterized in that, Step S30, which involves performing causal chain analysis based on a path reasoning and deviation detection mechanism using a semantic graph structure based on a unified cross-modal representation vector X, and outputting a cross-modal explanation chain graph, specifically includes: Step S301: Based on the unified cross-modal representation vector X, construct a heterogeneous semantic graph G using Python's networkx and geometric libraries. Each node in the heterogeneous semantic graph G represents a semantic unit under a different modality. Semantic units include text entities, image regions, and code snippets. The weight of each edge in the heterogeneous semantic graph G represents semantic similarity. Based on the heterogeneous semantic graph G, combine the offset node2vec random walk method to sample the set of semantic paths within the modality and output the initial path set P. Step S302: Calculate the bias propagation strength score of each path in the initial path set P using a path dependency scoring method based on graph attention mechanism, and remove paths in the initial path set P whose bias propagation strength scores are less than a preset bias propagation strength score threshold, outputting an optimized local subgraph. ; Step S303: Based on the optimized local subgraph A shortest path-based counterfactual analysis method is used to perform semantic bias chain reconstruction tasks, and outputs a cross-modal explanation chain graph.

5. The method for enhancing the interpretability of multimodal indicators as described in claim 4, characterized in that, In step S303, based on the optimized local subgraph The steps for performing semantic bias chain reconstruction using shortest path-based counterfactual analysis and outputting a cross-modal explanatory chain graph specifically include: optimizing local subgraphs. The node corresponding to the path with the highest bias propagation strength score is taken as the starting node. Based on the starting node and semantic similarity, the counterfactual path modeling principle is used to search for the shortest causal path based on graph structure and semantic coupling, and a set of shortest causal paths is constructed. Finally, a cross-modal explanatory chain graph is generated based on the set of shortest causal paths. The cross-modal explanatory chain graph is used to represent the path propagation mode, the main triggering nodes and the causal jump relationship.

6. The method for enhancing the interpretability of multimodal indicators as described in claim 1, characterized in that, In step S40, the steps of performing counterfactual reasoning on anomaly paths based on the information entropy inversion principle using semantic constraints according to the cross-modal interpretation chain graph, and outputting the main path of anomaly interpretation, specifically include: Step S401: Select the k-th cross-modal interpretation chain from the cross-modal interpretation chain diagram. For the k-th cross-modal interpretation chain Semantic information entropy is calculated using the weighted normalized probability modeling method. Semantic information entropy Used to indicate the uneven distribution of the degree of abnormality among nodes in a path under semantic consistency constraints; Step S402: Based on semantic information entropy The path local entropy mutation rate was calculated using the window difference analysis method. Select the path local entropy mutation rate in the cross-modal interpretation chain graph. The longest path is used as the exception interpretation path; Step S403: If the anomaly explanation path and the optimized local subgraph are... If there are no overlapping nodes, the semantic deviation chain reconstruction task is performed using the shortest path-based counterfactual analysis method based on the anomaly interpretation path, and the main anomaly interpretation path is output.

7. The method for enhancing the interpretability of multimodal indicators as described in claim 1, characterized in that, Step S50, which involves performing a structured code modification task based on the semantic perturbation mapping mechanism using the main path of anomaly interpretation and outputting the final set of interpretation content, specifically includes: Step S501: Obtain the semantic node sequence based on the main path of anomaly interpretation, construct a semantic context embedding graph based on the semantic node sequence, identify anomalous mutation points using the transformers library in Python based on the semantic context embedding graph, and construct a semantic offset perturbation vector field based on the anomalous mutation points; Step S502: Perform clustering analysis and attribution in the semantic offset perturbation vector field using K-nearest neighbor clustering to form candidate repair paths; Step S503: Obtain code modality context information, which includes source code semantic graph information, AST node chain information, and function call graph information; sort the candidate repair paths based on the code modality context information and output a structured set of code modification suggestions; Step S504: Apply the code modification suggestion set and obtain the code modification result set; based on the code modification suggestion set and the code modification result set, use Python's libcst library to perform syntax-preserving abstract syntax tree interpretation processing, and output the final interpretation content set.

8. A multimodal index interpretability enhancement system, applied to the multimodal index interpretability enhancement method according to any one of claims 1 to 7, characterized in that, The multimodal index interpretability enhancement system includes: The event alignment module is used to acquire voice command text, front-end tracking data, and page DOM snapshot data generated by the user during interaction with the web application at time t. Based on the voice command text, front-end tracking data, and page DOM snapshot data, it performs a multimodal event synchronization mapping task using an adaptive time-weighted sliding window alignment mechanism, and outputs a multimodal synchronization event window. ; The cross-modal fusion module is used for multimodal synchronization event windows. A multi-semantic residual alignment mechanism based on modality-guided weights is used to perform a cross-modal attention aggregation task, generating a unified cross-modal representation vector X; The causal chain construction module is used to perform causal chain analysis tasks based on a path reasoning and deviation detection mechanism based on a semantic graph structure, using a unified cross-modal representation vector X, and outputs a cross-modal interpretation chain graph. The counterfactual reasoning module is used to perform counterfactual reasoning tasks for abnormal paths based on the information entropy inversion principle of semantic constraints according to the cross-modal interpretation chain graph, and output the main path of abnormal interpretation. The semantically driven repair module is used to perform structured code modification tasks based on the main path of anomaly explanation using a semantic perturbation mapping mechanism, and outputs the final set of explanation content.

9. A device for enhancing the interpretability of multimodal indicators, characterized in that, The multimodal indicator interpretability enhancement device includes: a memory, a processor, and a multimodal indicator interpretability enhancement program stored in the memory and executable on the processor. When the multimodal indicator interpretability enhancement program is executed by the processor, it implements a multimodal indicator interpretability enhancement method according to any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a multimodal indicator interpretability enhancement program, which, when executed by a processor, implements a multimodal indicator interpretability enhancement method according to any one of claims 1 to 7.

Images