A method and system for evaluating the complexity of multimodal information interfaces

Abstract

This invention discloses a method and system for evaluating the complexity of multimodal information interfaces, comprising: collecting multimodal interface elements, including visual data, auditory data, and interaction data; decomposing the interface complexity of the multimodal interface into information complexity, visual complexity, auditory complexity, and cognitive complexity; calculating the indices for information complexity, visual complexity, auditory complexity, and cognitive complexity respectively; normalizing all calculated indices; assigning weights to each indices; and calculating a comprehensive score by combining the normalized index values ​​with the corresponding weights. This method decomposes multimodal complexity into four dimensions: information, visual, auditory, and cognitive, providing a comprehensive analysis of multimodal information fusion and embedding a dynamic optimization process, significantly improving evaluation efficiency and interface usability, and offering better practicality.

Description

Technical Field

[0001] This invention belongs to the field of human-computer interaction technology, specifically a method and system for evaluating the complexity of multimodal information interfaces based on a multidimensional framework. It is applicable to the interface complexity optimization design in scenarios such as intelligent driving systems, industrial monitoring platforms, smart city interfaces, and complex equipment control systems. Background Technology

[0002] With the widespread application of multimodal information interfaces, complexity management has become a core challenge in interface design. Traditional complexity assessment methods often focus on a single modality (such as visual or auditory), lack comprehensive analysis of multimodal information fusion, and have fragmented assessment metrics, resulting in limited practicality.

[0003] For example:

[0004] 1. Visual complexity assessment: Traditional methods calculate complexity by the number of interface elements or information entropy, but cannot reflect the timeliness impact of dynamic information (such as real-time video and dynamic maps).

[0005] 2. Auditory complexity assessment: Existing studies mostly focus on the length of voice commands or the intensity of background noise, but lack analysis of the synergy between auditory and visual information.

[0006] 3. Lack of cross-modal coordination: Existing technologies do not quantify the compatibility, integration degree, and smoothness of cross-modal transitions of multimodal information, leading to an increase in user cognitive load.

[0007] Furthermore, existing methods lack a unified evaluation framework, making it difficult to balance information capacity and user cognitive ability.

[0008] Therefore, a systematic evaluation method is needed to address the shortcomings of current multimodal interface complexity evaluation methods that cannot take into account multiple factors. Summary of the Invention

[0009] The technical problem to be solved by the present invention is to provide a method and system for evaluating the complexity of multimodal information interfaces in order to address the shortcomings of the prior art. The evaluation method and system decompose multimodal complexity into four dimensions: information, vision, hearing, and cognition. It performs comprehensive analysis of multimodal information fusion and embeds a dynamic optimization process, which significantly improves evaluation efficiency and interface usability, and has better practicality.

[0010] To achieve the above-mentioned technical objectives, the technical solution adopted by the present invention is as follows:

[0011] A method for evaluating the complexity of a multimodal information interface includes the following steps:

[0012] Step 1: Collect multimodal interface elements, which include visual data, auditory data, and interaction data;

[0013] Step 2: Decompose the interface complexity of the multimodal interface into information complexity, visual complexity, auditory complexity, and cognitive complexity; calculate the metrics for information complexity, visual complexity, auditory complexity, and cognitive complexity respectively. The metrics for information complexity include the number of sensory modalities, the type of symbolic modality, and cross-modal fusion. The metrics for visual complexity include static information density, dynamic information complexity, and visual consistency. The metrics for auditory complexity include semantic clarity, background noise interference index, and cross-modal synchronization. The metrics for cognitive complexity include the rationality of the layout order, the depth of information hierarchy, and the interactive control logic.

[0014] Step 3: Normalization: Normalize all the indicators calculated in Step 2 respectively;

[0015] Step 4: Assign weights to indicators: Assign weights to each indicator;

[0016] Step 5; Comprehensive score calculation: Calculate the comprehensive score by combining the normalized index values ​​with the weights corresponding to each index.

[0017] As a further improvement of the present invention, in step 1, the visual data includes static text, static icons, static graphics, dynamic videos, animations, and real-time monitoring screens; the auditory data includes voice commands and background noise intensity; and the interactive data includes user operation paths, response times, and error rates.

[0018] As a further improvement to the present invention, the number of sensory modalities N s The calculation formula is:

[0019]

[0020] Where m is the total number of preset sensory modalities in the interface; δ k This indicates whether the k-th sensory modality is used in the interface; sensory modalities include visual, auditory, tactile, and olfactory.

[0021] The symbolic modal type T m The calculation formula is:

[0022]

[0023] Where U represents the total number of preset symbol types in the interface, γ μ Indicates whether the μ-th symbol type appears in the interface; symbol types include static text, static icons, and static graphics;

[0024] The cross-modal fusion degree F c The calculation formula is:

[0025] F c =α·C s +β·S t , (α+β=1);

[0026] Among them, C s Compatibility scores or task completion rates are used to measure the synergistic effect between different sensory modalities;

[0027] When C s When rating compatibility, C s The calculation process is as follows: Multiple representative user samples are recruited, and information retrieval and decision-making tasks are conducted. Users are required to rate the consistency and synergistic effects of intermodal information, and the average score A of all user ratings is calculated. s The compatibility score is calculated using the following formula:

[0028] When C s When C is the task completion rate, s The calculation process is as follows: recruit multiple representative user samples, conduct multimodal collaborative task completion rate test tasks, require users to complete the task correctly and successfully when the multimodal information collaboration is correct, record the task completion rate, and the task completion rate is the number of times the task was completed correctly and successfully divided by the total number of tasks.

[0029] S t For smoothness, it is used to measure how smoothly a user switches between different modes;

[0030] Smoothness of transition t The calculation formula is:

[0031] or

[0032] Among them, T avg T represents the average time taken by multiple users to complete a test task involving switching between multiple modalities. base E represents the set baseline time; E is the number of erroneous operations caused by user errors due to poor mode switching during the modal switching test task. max The maximum number of errors allowed is set. Error rate;

[0033] α is C s The weighting coefficient, β is S t The weighting coefficients.

[0034] As a further improvement of the present invention, the static information density D static The calculation formula is:

[0035]

[0036] Where, N y w represents the number of static elements of type y. y Let N be the weight coefficient of the static element of class y, and T be the total number of static elements; text N represents the number of static text elements in the interface. icon This represents the number of static icon elements in the interface.

[0037] N graphic The number of static graphic elements in the interface; w text The weight coefficient for static text elements; w icon This refers to the weight coefficient of static icon elements; w graphic A represents the weighting coefficients for static graphic elements. interface The total area of ​​the interface;

[0038] The dynamic information complexity D dynamic The calculation formula is:

[0039]

[0040] Among them, Vr f Let Fr be the refresh rate of the f-th type of dynamic element. f For the frame rate or effective visual update frequency of the f-th type of dynamic element, Area f Let A be the display area of ​​the f-th type of dynamic element in the interface. total V represents the total area of ​​the interface; max ×F max This is a reference value for the product of the preset maximum refresh rate and maximum frame rate; the types of dynamic elements include dynamic video, animation, and real-time monitoring screens;

[0041] The aforementioned visual consistency V c The calculation formula is:

[0042] V c =γ·M d +(1-γ)·L c , (0≤γ≤1);

[0043] Where M d The layout matching degree is calculated as follows:

[0044]

[0045] The average offset distance refers to the average spatial misalignment distance between all dynamic elements and their associated static elements in the interface; the maximum allowable offset distance is a preset, tolerable upper limit value for visual offset.

[0046] Lc Logical consistency is scored, and the calculation method is as follows:

[0047]

[0048] The user task completion rate ranges from 0 to 1. Specifically, a representative user sample is recruited and asked to complete a set of standardized test tasks. The effective completion of these tasks depends on the logical connection between dynamic and static elements in the interface. The user task completion rate is recorded as the number of times the user successfully completed the task divided by the total number of tasks. The expert score ranges from 0 to 1. Specifically, experts in human-computer interaction or interface design independently evaluate the logical rationality of dynamic and static elements in the interface in terms of information expression, causal relationships, and process coordination. Experts use a continuous scale from 0 to 1 to score, and the average of all expert scores is calculated as the final expert score.

[0049] γ is the weighting coefficient.

[0050] As a further improvement to the present invention, the semantic clarity S atotal The calculation formula is:

[0051]

[0052] Where B is the total number of voice commands in the interface, and b is the sequence number of the voice command; S a,b The average semantic clarity of the b-th voice command in the interface is given by this value.

[0053] S a,b The calculation formula is:

[0054]

[0055] Where H represents the number of times the speech recognition system in the interface recognizes the b-th speech command, that is, the number of times the speech recognition system in the interface repeatedly recognizes the b-th speech command; h represents the sequence number of times the speech recognition system in the interface recognizes the b-th speech command, h = 1, 2, ..., H; T ref,b T is the reference text for the b-th voice command. rec,b,h The text obtained by the speech recognition system in the interface after recognizing the b-th speech command for the h-th time;

[0056] Similarity(T ref,b ,T rec,b,h ) is a semantic similarity calculation function. The specific method is as follows: The reference text T... ref,b and recognize text T rec,b,hInput the data into the BERT fine-tuning model, extract the output vector corresponding to the CLS marker as the sentence vector to obtain the corresponding sentence vector representation, then calculate the cosine similarity between the two sentence vectors, and then normalize the cosine similarity to obtain Similarity(T). ref,b ,T rec,b,h The value of );

[0057] The background noise interference index N interference The calculation formula is:

[0058]

[0059] Where dB norm =min(dB,100), where dB is the background noise level in decibels, P n This represents the proportion of background noise in the total signal.

[0060] The cross-modal synchronization S av The calculation formula is:

[0061]

[0062] Where Δt is the average time delay of the auditory and visual signals, calculated using timestamp alignment; T max The maximum allowable delay threshold is set to 50ms.

[0063] As a further improvement to the present invention, the rationality of the layout order is L o The calculation formula is:

[0064]

[0065] Where C represents the total number of attention regions divided in the interface;

[0066] p c The reference text for the c-th attention region is calculated as follows:

[0067]

[0068] Q c Q represents the number of fixations in the c-th attention region. 总 Total number of fixations;

[0069] The information hierarchy depth H d The calculation formula is:

[0070] H d = L·log2(B+1);

[0071] L represents the total number of levels in the user's operation path; B is the branching factor, i.e., the average number of branches per level, calculated using the following formula: Among them, b l This indicates the number of branches at each level in the user's operation path;

[0072] The interactive control logic I r The calculation formula is:

[0073]

[0074] Among them, R s Redundancy of user operation steps, calculated as: R s = Actual number of steps - Theoretical minimum number of steps; R max R is the preset maximum redundancy threshold. max =5.

[0075] As a further improvement to the present invention, step 3 specifically comprises:

[0076] Step 3 specifically includes:

[0077] The values ​​of each indicator in step 2 are normalized, and the calculation formula is as follows:

[0078]

[0079] Among them, X j X is the value of the j-th indicator calculated according to the corresponding formula in step 2. min X is the minimum value of the j-th indicator. max x is the maximum value of the j-th indicator. j Let be the normalized value of the j-th indicator.

[0080] As a further improvement to the present invention, step 4 specifically comprises:

[0081] The entropy weight method or the CRITIC method can be used to assign weights to each indicator.

[0082] The entropy weight method is used to assign weights to each indicator as follows:

[0083] Calculate the probability:

[0084]

[0085] Where p ij Let x be the probability distribution of the j-th indicator of the i-th interface sample. ij is the normalized value of the j-th indicator of the i-th interface sample, and n is the total number of interface samples;

[0086] Calculate information entropy:

[0087]

[0088] e j Let ln(n) be the information entropy of the j-th index, and ln(n) be the normalization factor.

[0089] Calculate the weights:

[0090]

[0091] Among them, w j e represents the weight of the j-th indicator; g Let G be the information entropy of the g-th indicator, and G be the total number of indicators.

[0092] The CRITIC method is used to assign weights to each indicator as follows:

[0093] Calculate the standard deviation:

[0094]

[0095] Where σ j Let x be the standard deviation of the j-th indicator. ij It is the normalized value of the j-th indicator for the i-th interface sample;

[0096] This represents the mean of the j-th indicator across all n interface samples; the calculation formula is:

[0097]

[0098] n is the total number of interface samples;

[0099] Calculate the correlation coefficient:

[0100]

[0101] Where i represents the interface sample index, i = 1, 2, ..., n; x ip Let represent the normalized value of the p-th metric for the i-th interface sample. It is the mean of the p-th indicator across all n interface samples; r pj The value range of r is [-1, 1]. The larger its absolute value, the stronger the linear correlation and the lower the conflict between the p-th indicator and the j-th indicator; pj This represents the Pearson correlation coefficient between the p-th and j-th indicators;

[0102] Calculate the total information content:

[0103]

[0104] Where C j G represents the total amount of information; G represents the total number of indicators.

[0105] Calculate the weights:

[0106]

[0107] Among them, w j C represents the weight of the j-th indicator; g This represents the total information content of the g-th indicator.

[0108] As a further improvement to the present invention, step 5 specifically comprises:

[0109] Calculate the overall score R:

[0110]

[0111] Where w j Let x be the weight of the j-th indicator; j Let be the normalized value of the j-th indicator.

[0112] To achieve the above-mentioned technical objectives, another technical solution adopted by the present invention is as follows:

[0113] A multimodal information interface complexity evaluation system, comprising:

[0114] Multimodal interface element acquisition module: used to acquire multimodal interface elements, including visual data, auditory data and interaction data;

[0115] The Interface Complexity Analysis module categorizes interface complexity into information complexity, visual complexity, auditory complexity, and cognitive complexity. It calculates metrics for each of these categories. Information complexity metrics include the number of sensory modalities, the type of symbolic modality, and cross-modal fusion. Visual complexity metrics include static information density, dynamic information complexity, and visual consistency. Auditory complexity metrics include semantic clarity, background noise interference index, and cross-modal synchronization. Cognitive complexity metrics include layout order rationality, information hierarchy depth, and interactive control logic.

[0116] Normalization module: Used to normalize all indicators separately;

[0117] Indicator weight allocation module: used to assign weights to each indicator;

[0118] Comprehensive scoring and analysis module: This module calculates the comprehensive score by combining the normalized index values ​​with the weights corresponding to each index.

[0119] The beneficial effects of this invention are as follows:

[0120] This invention designs a four-dimensional dynamic evaluation framework: for the first time, it integrates information, visual, auditory, and cognitive complexity, covering all elements of a multimodal interface. This invention also features a method-coupled design: with the process at its core and algorithms as auxiliary tools, it ensures that the evaluation results are interpretable and practical. Furthermore, this invention possesses real-time optimization capabilities: through dynamic data feedback and algorithm iteration, it supports real-time adjustments to the interface design.

[0121] This invention integrates algorithm design into a dynamic evaluation model and process to achieve complexity quantification, weight allocation, comprehensive scoring, and optimized selection. Its core innovation lies in decomposing multimodal complexity into four dimensions: information, vision, hearing, and cognition, and embedding a dynamic optimization process, significantly improving evaluation efficiency and interface usability. Attached Figure Description

[0122] Figure 1 This is a flowchart for evaluating the complexity of a multimodal information interface. Detailed Implementation

[0123] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings:

[0124] A method for evaluating the complexity of a multimodal information interface, such as Figure 1 As shown, Figure 1 This outlines the complete process from steps 1 to 5, highlighting the logical relationships between data collection, analysis, weight allocation, scoring, and optimization. Specifically, it includes the following steps:

[0125] Step 1: Collect multimodal interface elements using a camera, microphone, and interaction logger. These multimodal interface elements include visual data, auditory data, and interaction data.

[0126] Visual data includes static text, static icons, static graphics (including charts, maps, etc.), dynamic videos, animations, and real-time monitoring footage. Auditory data includes voice commands and background noise intensity. Interaction data includes user operation paths, response times, and error rates.

[0127] Data preprocessing: Time-series alignment of dynamic information and synchronous labeling of cross-modal data (such as voice and visual cues).

[0128] Step 2: Decompose the interface complexity of the multimodal interface into information complexity, visual complexity, auditory complexity, and cognitive complexity. Calculate the primary indicators for information complexity, visual complexity, auditory complexity, and cognitive complexity respectively. The primary indicators for information complexity include the number of sensory modalities, the type of symbolic modality, and cross-modal fusion. The primary indicators for visual complexity include static information density, dynamic information complexity, and visual consistency. The primary indicators for auditory complexity include semantic clarity, background noise interference index, and cross-modal synchronization. The primary indicators for cognitive complexity include the rationality of layout order, the depth of information hierarchy, and interactive control logic.

[0129] Among them, the number of sensory modalities N s : The number of sensory channels (such as vision, hearing, and touch) involved in the quantification interface.

[0130] Number of sensory modalities N s The calculation formula is:

[0131]

[0132] Where m is the total number of preset sensory modalities in the interface (such as visual, auditory, tactile, olfactory, etc.); δ k This indicates whether the k-th sensory modality is used in the interface; sensory modalities include visual, auditory, tactile, and olfactory senses.

[0133] Calculation method: Iterate through all m sensory modalities, marking them as 1 if they exist and 0 otherwise, and finally sum all values. If the interface includes both visual and tactile senses, then N s =1+1=2; if only hearing is included, then N s =1.

[0134] Among them, the symbolic modal type T m : Classify statistical symbol element types (such as static text, static icons, and static graphics).

[0135] Symbolic modal type T m The calculation formula is:

[0136]

[0137] Where U represents the total number of preset symbol types in the interface (such as static text, static icons, and static graphics (including charts, maps, etc.)), γ μ Indicates whether the μth symbol type appears in the interface; symbol types include static text, static icons, and static graphics.

[0138] Calculation method: Count the number of symbol types in the interface, incrementing by 1 for each new type. If the interface contains static text and static graphics, then T... m =1+1=2; if only static icons are included, then T m =1.

[0139] Among them, the cross-modal fusion degree F c Compatibility score (C) s ) and transition smoothness (S t Weighted calculation:

[0140] F c =α·C s +β·S t , (α+β=1);

[0141] Among them, C s It serves as a compatibility score or task completion rate, used to measure the synergistic effect between different sensory modalities (such as visual and auditory consistency).

[0142] When C s For compatibility scoring, the quantification method is as follows: A representative user sample (e.g., n≥15 people) is recruited based on subjective user ratings. An information retrieval and decision-making task experiment is conducted. In the experiment, the interface simultaneously presents visual text and auditory speech. Users need to locate the target element indicated by the auditory command within the visual interface and complete the marking or operation. The detailed experimental procedure is as follows:

[0143] Preliminary experiment: Familiarize yourself with interface operation and modal mapping rules;

[0144] Experimental materials: Select multiple interfaces involving several core functions in the evaluation interface to complete multiple sets of tests;

[0145] Experimental content: After each task, a 5-point Likert scale was used for scoring (1 = highly incompatible, 5 = highly compatible). The scoring dimensions included: consistency of modal information meaning (e.g., whether the speech and text descriptions contradict each other); coordination of modal presentation timing (e.g., whether the speech plays within 0.5-2 seconds after the visual element appears); and complementarity of information (e.g., whether the auditory information supplements key details not presented visually). The scoring dimensions can also be modified according to the actual needs of different product interfaces.

[0146] After completing the experimental task, users are asked to rate the consistency and synergistic effect of intermodal information on a Likert scale ranging from 1 (extremely incompatible) to 5 (extremely compatible).

[0147] Calculate the average A of all user ratings. s The compatibility score is calculated using the following formula:

[0148] When Cs When calculating task completion rate, the quantification method is as follows: A representative user sample (e.g., n≥15 people) is recruited using objective user ratings. A multimodal collaborative task completion rate test is conducted, requiring users to successfully complete the task only when multimodal information collaboration is correct. The detailed experimental procedure is as follows:

[0149] Preliminary experiment: Familiarize yourself with the experimental rules for interface operation and modal collaboration;

[0150] Experimental Materials: Multiple modalities involving several core functions in the evaluation interface were selected to form the test materials required for the experiment. These included three groups: a full-modal group; a single-modal missing group (where any one of the visual, auditory, or tactile senses was disabled, and the decrease in task completion rate was observed); and a modal conflict group (where contradictory information was intentionally set (e.g., a visual display of "normal" but a voice message of "malfunction"), and the error rate was recorded).

[0151] Experimental content: 1. Group testing: Each subject needs to complete three tasks: "full modality", "single modality missing", and "modal conflict"; 2. Task timing: Set a time limit for the task, and record the task as not completed if the time limit is exceeded; 3. Data recording: Record the completion status (success / failure), completion time, and error operation type of each group of tasks.

[0152] Record the task completion rate (number of successful tasks / total number of tasks). Use the completion rate directly as C. s The value C s =Task completion rate.

[0153] S t To assess modal transition fluency, a measure of how smoothly users switch between different modalities (e.g., switching time, error rate) is used. Quantification methods include: recruiting a representative user sample (e.g., n≥15 people) for modal transition fluency assessment. Multiple modal transition tasks involving vision, hearing, and touch are conducted to quantify the cognitive load and operational efficiency of users switching between different modalities. Detailed experimental results are as follows:

[0154] Preliminary experiment: Familiarize yourself with the interface modal switching rules;

[0155] Experimental materials: Different switching materials were designed according to different modalities (such as switching from visual modality to auditory modality. Visual modality stage: find the button marked in red on the interface; auditory modality stage: hear the voice command "press and hold for 3 seconds" and then perform the operation; this completes the switching from visual modality to auditory modality).

[0156] Experimental content: 1. Set the reference time T for mode switching through expert group testing. base(For reference: Invite 5 UI design experts to complete 20 tasks, and take the average of the best 10); 2. Complete multiple sets of switching tasks, each set containing 3 rounds of modal switching (visual → auditory → touch → visual loop). Repeat multiple sets of tasks and record the average time Tavg (seconds) for each user to complete each set.

[0157] The smoothness of transition is calculated as follows:

[0158]

[0159] The closer the value is to 1, the smoother the switching.

[0160] Based on the number of erroneous operations: In modal switching tests, record the number of erroneous operations (E) caused by user errors due to poor modal switching (such as incorrect clicks, incorrect voice commands). Set a maximum allowed number of errors, E. max (e.g., total number of task steps × 0.2). Transition fluency is calculated as follows:

[0161]

[0162] α is C s The weighting coefficient, β is S t The weighting coefficients.

[0163] Among them, the static information density D static : Quantify the density of static visual elements per unit area (referring to visual elements that do not change over time at a specific point in time, such as text, icons, graphics, etc.), reflecting the visual complexity of static information on the interface.

[0164] Static information density D static The calculation formula is:

[0165]

[0166] Where, N y w represents the number of static elements of type y. y Let N be the weight coefficient of the static element of class y, and T be the total number of static elements; text N represents the number of static text elements in the interface (unit: number), referring to individual text blocks or tags; icon N represents the number of static icon elements in the interface (unit: number), referring to graphic symbols with specific meanings; graphic The number of static graphic elements in the interface (unit: number), referring to basic visual components such as lines, shapes, and non-dynamic chart elements;

[0167] w text w icon w graphicThese are the weighting coefficients for static text elements, static icon elements, and static graphic elements (reflecting the relative impact of different element types on visual complexity; these can be set through experiments or expert evaluation when applied to different interfaces); A interface The total area of ​​the interface (unit: square pixels or square meters).

[0168] Calculation method: Sum the elements by weighted type and divide by the area. A larger value indicates denser static information. If the interface has 10 text elements (weight 1.0), 5 icons (weight 1.2), and 3 graphics (weight 0.8), with an area of ​​2 square meters, then:

[0169]

[0170] Among them, the dynamic information complexity D dynamic : Measure dynamic elements (referring to visual elements that change continuously over time, such as:

[0171] The contribution of dynamic videos, animations, and real-time monitoring footage to visual complexity.

[0172] Dynamic information complexity D dynamic The calculation formula is:

[0173]

[0174] Among them, Vr f Fr is the refresh rate (in Hz) of the f-th type of dynamic element. f For the frame rate or effective visual update frequency (in FPS) of the f-th type of dynamic element, Area f Let A be the display area of ​​the f-th type of dynamic element in the interface. total V represents the total area of ​​the interface; max ×F max This is a preset reference value for the product of the maximum refresh rate and the maximum frame rate (e.g., 100Hz × 120FPS = 12000). This value is used to normalize the area-weighted sum of the products of the refresh rate and frame rate of dynamic elements, mapping it to a relative scale and serving as a reference point when calculating logarithmic complexity. +1 is added to avoid the logarithm being negative infinity when the product is 0. The base of the logarithm is chosen based on the fact that a base of 2 can compress high dynamic range and conforms to the non-linear characteristics of human visual perception. Types of dynamic elements include dynamic video, animation, and real-time monitoring footage.

[0175] Calculation method: Calculate the product of refresh rate and frame rate for all dynamic elements (such as dynamic videos, animations, and real-time monitoring screens), and then perform a weighted sum based on their area proportion on the screen. Finally, normalize this weighted sum and take its logarithm. A larger result value indicates higher dynamic complexity.

[0176] Example: If the interface contains two types of dynamic elements. Element A: Vr A =30Hz,Fr A =60FPS, Area A Area A total 30%; Element B: Vr B =20Hz,Fr B =40FPS, Area B Area A total 20% of; VmaxFmax = 12000. Therefore:

[0177]

[0178] Among them, visual consistency V c : By matching the layout of dynamic and static elements (M) d ) and logical consistency (L c A comprehensive evaluation of the synergy and logical rationality of dynamic and static elements is conducted.

[0179] V c =γ·M d +(1-γ)·L c , (0≤γ≤1).

[0180] Where M d The layout matching score (0-1) measures the degree of alignment and coordination between dynamic elements and their associated static elements in visual space. The calculation method is as follows:

[0181]

[0182] The average offset distance refers to the average spatial misalignment distance between all dynamic elements and their associated static elements in the interface (for example, the pixel distance between their visual center point or key reference point can be calculated). The maximum allowable offset distance is a preset, tolerable upper limit of visual offset; exceeding this value indicates extremely poor layout matching (which can be determined through expert evaluation or based on the interface size).

[0183] L c Logical consistency is scored (0-1), calculated as follows:

[0184]

[0185] User Task Completion Rate (0-1): Recruit a representative user sample (e.g., n≥10 people) and require them to complete a set of standardized test tasks. Effective completion of these tasks depends on the logical connection between dynamic and static elements in the interface (e.g., interpreting the meaning of corresponding static labels based on changes in dynamic charts). Record the percentage of users who successfully complete the tasks (number of successful completions / total number of tasks). Expert Rating (0-1) (i.e., Expert Logical Consistency Rating): Experts in human-computer interaction or interface design (e.g., ≥3 people) independently evaluate the logical rationality of dynamic and static elements in the interface in terms of information expression, causal relationships, and process coordination. Experts use a continuous scale from 0 (completely illogical) to 1 (logically perfect and self-consistent) for rating. Calculate the average of all expert ratings as the expert logical consistency rating.

[0186] γ is a weighting coefficient that is dynamically adjusted according to the scenario (e.g., γ = 0.8 for game interfaces and γ = 0.5 for utility software).

[0187] Calculation method: Weighted summation followed by normalization; the closer the value is to 1, the higher the consistency. If M d =0.75,L c =0.85, γ=0.6, then V c =0.6×0.75+0.4×0.85=0.79.

[0188] Among them, semantic clarity S atotal The calculation process is as follows:

[0189] First, calculate S a,b , representing the semantic similarity score (0-1) based on the output of the natural language processing model, is used to evaluate the average semantic matching degree of the same voice command in multiple tests.

[0190]

[0191] Among them, S a,b This represents the average semantic clarity of the b-th command in a single test. H is the number of times the speech recognition system in the interface recognizes the b-th speech command, i.e., the number of repeated tests for the b-th speech command; h represents the test number of the b-th speech command, h = 1, 2, ..., H, used to iterate through each test data; T ref,b The reference text (original semantic content) for the b-th speech instruction; only one original semantic content is used for reference per instruction; T rec,b,h The text obtained after the speech recognition system in the interface performs the h-th recognition and conversion of the b-th speech command. Similarity(T) ref,b ,T rec,b,hThe semantic similarity calculation function is implemented using sentence embedding technology based on pre-trained language models (such as BERT). Specifically, the method involves embedding the reference text T... ref,b and recognize text T rec,b,h Input the data into the BERT fine-tuning model, extract the output vector corresponding to the CLS marker as the sentence vector to obtain the corresponding sentence vector representation, then calculate the cosine similarity between the two sentence vectors, and then normalize the cosine similarity to obtain Similarity(T). ref,b ,T rec,b,h The cosine similarity value ranges from -1 to 1, but in this type of model, semantically similar sentence pairs usually produce a positive value close to 1. The similarity value output by this function has been normalized to a suitable range by (cosine similarity + 1) / 2.

[0192] Calculation method: The similarity between multiple sets of speech recognition results and the reference text is averaged. The closer the value is to 1, the clearer the semantics. If the voice command "adjust air conditioning temperature" is tested 3 times, and the recognition results are ① "adjust air conditioning temperature" (similarity 1.0) ② "adjust air conditioning humidity" (similarity 0.7) ③ "adjust car interior temperature" (similarity 0.8), then:

[0193]

[0194] Secondly, after evaluating the average semantic matching degree of the same voice command across multiple tests, a comprehensive evaluation of multiple commands can be further achieved, namely semantic clarity S. atotal :

[0195]

[0196] S atotal This reflects the overall semantic recognition accuracy of all voice commands in the interface. A higher value indicates higher overall clarity. Here, B is the total number of voice commands in the interface (e.g., 5 different commands); b is the command number. Iterates through the b-th command and sequentially calls its single command semantic clarity S. a,b Accumulate. Assume the interface contains 3 commands: Command 1: S a,1 =0.9 (average similarity 0.9), Instruction 2: S a,2 =0.7, Instruction 3: S a,3 =0.8, then the overall sharpness is:

[0197]

[0198] Among them, the background noise interference index N interference This is used to quantify the degree to which ambient noise (i.e., background noise) interferes with the transmission of auditory information. The calculation formula is:

[0199] Where dB norm =min(dB,100);

[0200] dB is the decibel value of background noise (unit: dB), and the measurement range is 0-120 dB. n The percentage of background noise in the total signal (0-1, e.g., 0.3 represents 30%); dB norm This is the normalized decibel value, with a maximum limit of 100dB to avoid numerical overflow.

[0201] Calculation method: Normalize the decibel value and multiply it by the noise ratio; the larger the value, the stronger the interference. If dB = 85, P n =0.6, then:

[0202]

[0203] Among them, cross-modal synchronization S av This is used to assess the temporal alignment of auditory and visual signals, ensuring a consistent user experience. The calculation formula is:

[0204]

[0205] Where Δt is the average time delay (in milliseconds) of the auditory and visual signals, calculated using timestamp alignment; T max The maximum allowable latency threshold is set to 50ms (a common standard based on real-time interactive scenarios).

[0206] Calculation method: The smaller the delay, the higher the score; exceeding the threshold results in a score of 0. If Δt = 30ms, then:

[0207]

[0208] Among them, the rationality of the layout order L o : Analyze user attention distribution using eye-tracking heatmaps and calculate information entropy:

[0209] Layout order rationality L o The calculation formula is:

[0210]

[0211] Where C represents the total number of attention regions divided in the interface; p c The reference text (original semantic content) for the c-th attention region is calculated as follows:

[0212]

[0213] Q c Q represents the number of fixations in the c-th attention region.总 This represents the total number of fixations.

[0214] Calculation method: The probability of fixation for each region is calculated using eye-tracking data, and the degree of concentration of attention distribution is quantified using information entropy. A higher entropy value indicates more dispersed user attention and lower layout rationality; therefore, L... o This is a positive indicator of complexity (i.e., the larger the original value, the higher the complexity). If the interface is divided into 3 regions, with 50, 30, and 20 fixations respectively, and a total of 100 fixations, then:

[0215] p1 = 0.5, p2 = 0.3, p3 = 0.2, L o = -(0.5log20.5+0.3log20.3+0.2log20.2)≈1.49 (bits).

[0216] Among them, the information hierarchy depth H d This quantifies the depth and breadth of a navigation path, reflecting the difficulty users face in finding information. The calculation formula is:

[0217] H d = L·log2(B+1);

[0218] L represents the total number of levels in the user's navigation path (e.g., Homepage → Submenu → Details Page, with 3 levels); B represents the branching factor, i.e., the average number of branches (entry points / options) per level, calculated as follows: The number of direct child nodes (branches) at each level (excluding the final leaf node) in the navigation structure is counted. l

[0219] (l represents the hierarchical index), representing the number of branches at each level in the user's operation path; then, the average value of all branching levels is calculated:

[0220]

[0221] The +1 in log2(B+1) is used to avoid the value being meaningless when the branch factor is 0.

[0222] Example: If level 1 has 5 branches, level 2 has 3 branches, and level 3 is a leaf node with no branches, then

[0223] Among them, interactive control logic I r Used to evaluate the simplicity of the interaction flow:

[0224] The calculation formula is:

[0225]

[0226] Among them, R sRedundancy of user operation steps, calculated as: R s = Actual number of steps - Theoretical minimum number of steps; R max R is the preset maximum redundancy threshold. max =5.

[0227] Calculation method: The ratio of redundancy to the maximum threshold, with the result limited to between 0 and 1. A larger value indicates more redundant interaction logic, hence I r R is a positive indicator of complexity (i.e., the larger the original value, the higher the complexity). If the task requires at least 3 steps to complete, but the user actually used 7 steps, R... max =5, then R s =7-3=4,I r =min(4 / 5,1)=0.8.

[0228] Step 3: Normalization: Normalize all the indicators calculated in Step 2, that is, eliminate the differences in the dimensions between the indicators and unify the directionality (all indicators are converted to positive, and the larger the value, the better).

[0229] Step 3 specifically includes:

[0230] The values ​​of each indicator in step 2 are normalized, and the calculation formula is as follows:

[0231]

[0232] Among them, X j For the j-th index, X is the value calculated according to the corresponding formula in step 2 (e.g., the number of sensory modalities Ns, static information density Dstatic, information hierarchy depth Hd, etc. calculated in step 2). min X is the minimum value of the j-th indicator (historical data or preset range). max x is the maximum value of the j-th indicator (historical data or preset range). j This is the normalized value (range 0-1) of the j-th indicator. A larger value indicates better performance. If we assume the number of sensory modalities (Ns) is a negative indicator, the original value is 3, and the preset range is 1-5:

[0233]

[0234] Step 4: Assign weights to indicators: Assign weights to each indicator;

[0235] Weights are assigned to indicators based on their information content or conflict level to differentiate their importance. The entropy weighting method or the CRITIC method can be used to assign weights to each indicator.

[0236] Method 1: Entropy weight method (suitable for high-dimensional data where the correlation between indicators is weak).

[0237] 1. Calculate the probability:

[0238]

[0239] Where p ij Let x be the probability distribution of the j-th indicator in the i-th interface sample, reflecting the proportion of indicator j in different samples. ij is the normalized value of the j-th indicator of the i-th interface sample, and n is the total number of interface samples;

[0240] 2. Calculate information entropy:

[0241]

[0242] e j Let be the information entropy of the j-th indicator. A larger entropy value indicates that the data is more dispersed and the information content is lower. ln(n) is a normalization factor to ensure that the entropy value is within a reasonable range.

[0243] 3. Calculate the weights:

[0244]

[0245] Among them, w j e represents the weight of the j-th indicator; g Let w be the information entropy of the g-th indicator, where G is the total number of indicators; the smaller the entropy (the greater the information content), the higher the weight w. j The higher.

[0246] Method 2: CRITIC method (strong correlation between indicators).

[0247] 1. Calculate the standard deviation, which measures the volatility of the data. Indicators with higher volatility contain more independent information and have higher weights.

[0248]

[0249] Where σ j Let x be the standard deviation of the j-th indicator. ij is the normalized value of the j-th indicator for the i-th interface sample; n is the number of interface samples (e.g., the number of interfaces being evaluated).

[0250] This represents the mean of the j-th indicator across all n interface samples; the calculation formula is:

[0251]

[0252] n is the total number of interface samples;

[0253] 2. Calculate the correlation coefficient: Calculate the Pearson correlation coefficient between any two indicators:

[0254]

[0255] Where i represents the interface sample index, i = 1, 2, ..., n; x ip Let represent the normalized value of the p-th metric for the i-th interface sample. It is the mean of the p-th indicator across all n interface samples; r pj The value range of r is [-1, 1]. The larger its absolute value, the stronger the linear correlation between the p-th indicator and the j-th indicator, and the lower the conflict (independence); pj This represents the Pearson correlation coefficient between the p-th and j-th indicators;

[0256] 3. Calculate the total information content:

[0257]

[0258] Where C j To maximize the overall information content, indicators with high volatility and strong conflict are given higher weights. G represents the total number of indicators; σ j It is an independent piece of information that measures the volatility of the indicator itself.

[0259] 4. Calculate the weight: Calculate the total information content C. j Normalization is then applied, transforming the result into weights. Larger values ​​indicate greater importance of the metric.

[0260]

[0261] Among them, w j C represents the weight of the j-th indicator; g This represents the total information content of the g-th indicator.

[0262] Step 5; Overall Score Calculation: Calculate the overall score by combining the normalized index values ​​with the weights corresponding to each index. Weighted summation of normalized indicators quantifies the overall complexity of the interface.

[0263] Step 5 specifically includes:

[0264] Calculate the overall score R:

[0265]

[0266] Where w j Let x be the weight of the j-th indicator; j Let be the normalized value of the j-th indicator. The overall score R ranges from 0 to 1; a larger value indicates higher interface usability or optimization, i.e., lower interface complexity.

[0267] Table 1 illustrates the four-dimensional framework and indicator hierarchy, showcasing the primary and secondary indicator classifications for the four dimensions: information, visual, auditory, and cognitive. Table 1:

[0268]

[0269]

[0270] This embodiment also provides a multimodal information interface complexity evaluation system, including:

[0271] Multimodal interface element acquisition module: used to acquire multimodal interface elements, including visual data, auditory data and interaction data;

[0272] The Interface Complexity Analysis module categorizes interface complexity into information complexity, visual complexity, auditory complexity, and cognitive complexity. It calculates metrics for each of these categories. Information complexity metrics include the number of sensory modalities, the type of symbolic modality, and cross-modal fusion. Visual complexity metrics include static information density, dynamic information complexity, and visual consistency. Auditory complexity metrics include semantic clarity, background noise interference index, and cross-modal synchronization. Cognitive complexity metrics include layout order rationality, information hierarchy depth, and interactive control logic.

[0273] Normalization module: Used to normalize all indicators separately;

[0274] Indicator weight allocation module: used to assign weights to each indicator;

[0275] Comprehensive scoring and analysis module: This module calculates the comprehensive score by combining the normalized index values ​​with the weights corresponding to each index.

[0276] The scope of protection of this invention includes, but is not limited to, the above embodiments. The scope of protection of this invention is defined by the claims. Any substitutions, modifications, or improvements to this technology that are easily conceived by those skilled in the art fall within the scope of protection of this invention.

Claims

1. A method for evaluating the complexity of a multimodal information interface, characterized in that: Includes the following steps: Step 1: Collect multimodal interface elements, which include visual data, auditory data, and interaction data; Step 2: Decompose the interface complexity of the multimodal interface into information complexity, visual complexity, auditory complexity, and cognitive complexity; calculate the metrics for information complexity, visual complexity, auditory complexity, and cognitive complexity respectively. The metrics for information complexity include the number of sensory modalities, the type of symbolic modality, and cross-modal fusion. The metrics for visual complexity include static information density, dynamic information complexity, and visual consistency. The metrics for auditory complexity include semantic clarity, background noise interference index, and cross-modal synchronization. The metrics for cognitive complexity include the rationality of the layout order, the depth of information hierarchy, and the interactive control logic. The semantic clarity mentioned The calculation formula is: ； Where B is the total number of voice commands in the interface, and b is the sequence number of the voice command. The average semantic clarity of the b-th voice command in the interface is given by this value. The calculation formula is: ； Where H represents the number of times the speech recognition system in the interface recognizes the b-th speech command, that is, the number of times the speech recognition system in the interface repeatedly recognizes the b-th speech command. This indicates the sequence number of times the speech recognition system in the interface recognizes the b-th speech command. =1,2,..., H; The reference text for the b-th voice command is as follows: The speech recognition system in the interface performs the first... The text obtained after the second recognition; This is a semantic similarity calculation function. The specific method is as follows: [The text then abruptly shifts to a different topic:] ...reference text... and text recognition Input the data into the BERT fine-tuning model, extract the output vectors corresponding to the CLS tags as sentence vectors to obtain the corresponding sentence vector representations, then calculate the cosine similarity between the two sentence vectors, and finally normalize the cosine similarity to obtain the final sentence vector representation. The value; The background noise interference index The calculation formula is: ； , The background noise level in decibels. This represents the proportion of background noise in the total signal. The aforementioned cross-modal synchronization The calculation formula is: ； in The average time delay of auditory and visual signals is calculated by aligning with timestamps. The maximum allowable latency threshold is set to 50 ms; Step 3: Normalization: Normalize all the indicators calculated in Step 2 respectively; Step 4: Assign weights to indicators; Step 5; Comprehensive score calculation: Calculate the comprehensive score by combining the normalized index values ​​with the weights corresponding to each index.

2. The method for evaluating the complexity of a multimodal information interface according to claim 1, characterized in that: In step 1, visual data includes static text, static icons, static graphics, dynamic videos, animations, and real-time monitoring screens; auditory data includes voice commands and background noise intensity; and interactive data includes user operation paths, response times, and error rates.

3. The method for evaluating the complexity of a multimodal information interface according to claim 1, characterized in that: The number of sensory modalities The calculation formula is: ； in, The total number of preset sensory modalities in the interface; This indicates whether the k-th sensory modality is used in the interface; sensory modalities include visual, auditory, tactile, and olfactory. The symbolic modal type The calculation formula is: ； in The total number of preset symbol types in the interface. Indicates the first Whether a symbol type appears in the interface; symbol types include static text, static icons, and static graphics; The cross-modal fusion degree The calculation formula is: ； in, Compatibility scores or task completion rates are used to measure the synergistic effect between different sensory modalities; when When scoring compatibility, The calculation process is as follows: Multiple representative user samples are recruited, and information retrieval and decision-making tasks are conducted. Users are required to rate the consistency and synergistic effects of intermodal information, and the average of all user ratings is calculated. The compatibility score is calculated using the following formula: ; when When the task completion rate is... The calculation process is as follows: recruit multiple representative user samples, conduct multimodal collaborative task completion rate test tasks, require users to complete the task correctly and successfully when the multimodal information collaboration is correct, record the task completion rate, and the task completion rate is the number of times the task was completed correctly and successfully divided by the total number of tasks. For smoothness, it is used to measure how smoothly a user switches between different modes; Smoothness of transition The calculation formula is: or ; in, This represents the average time a user takes to complete a test task that switches between multiple modalities. The set reference time; This represents the number of erroneous user actions caused by poor mode switching during a test task involving mode switching. The maximum number of errors allowed is set. Error rate; for The weighting coefficients, for The weighting coefficients.

4. The method for evaluating the complexity of a multimodal information interface according to claim 1, characterized in that: The static information density The calculation formula is: ； in, This represents the number of static elements of type y. Let be the weight coefficient of the y-th static element, and T be the total number of static elements; The number of static text elements in the interface; This represents the number of static icon elements in the interface. This represents the number of static graphic elements in the interface. This represents the weighting coefficient for static text elements; This is the weight coefficient for static icon elements; Weight coefficients for static graphic elements The total area of ​​the interface; The dynamic information complexity The calculation formula is: ； in, For the first Refresh rate of dynamic elements For the first The frame rate or effective visual update frequency of dynamic elements. For the first The display area of ​​dynamic elements in the interface. The total area of ​​the interface; This is a reference value for the product of the preset maximum refresh rate and maximum frame rate; the types of dynamic elements include dynamic video, animation, and real-time monitoring screens; The aforementioned visual consistency The calculation formula is: ； in The layout matching degree is calculated as follows: ； The average offset distance refers to the average spatial misalignment distance between all dynamic elements and their associated static elements in the interface; the maximum allowable offset distance is a preset, tolerable upper limit value for visual offset. Logical consistency is scored, and the calculation method is as follows: ； The user task completion rate ranges from 0 to 1. Specifically, a representative user sample is recruited and asked to complete a set of standardized test tasks. The effective completion of the task depends on the reasonable logical association between dynamic and static elements of the interface. The user task completion rate is recorded as the number of times the user successfully completed the task divided by the total number of tasks. The value range is 0-1. Specifically, experts in human-computer interaction or interface design independently evaluate the logical rationality of dynamic and static elements in the interface in terms of information expression, causal relationships, and process coordination. Experts use a continuous scale from 0 to 1 to score, and the average of all expert scores is calculated as the final value. ; These are the weighting coefficients.

5. The method for evaluating the complexity of a multimodal information interface according to claim 1, characterized in that: The rationality of the layout order The calculation formula is: ； Where C represents the total number of attention regions divided in the interface; The reference text for the c-th attention region is calculated as follows: ； Let c be the number of fixations in the c-th attention region. Total number of fixations; The information hierarchy depth The calculation formula is: ； L represents the total number of levels in the user's operation path; B is the branching factor, i.e., the average number of branches per level, calculated using the following formula: ;in, express ; The aforementioned interactive control logic The calculation formula is: ； in, The redundancy of user operation steps is calculated as follows: = Actual number of steps - Theoretical minimum number of steps; The preset maximum redundancy threshold, =5.

6. The method for evaluating the complexity of a multimodal information interface according to claim 1, characterized in that: Step 3 specifically includes: Step 3 specifically includes: The values ​​of each indicator in step 2 are normalized, and the calculation formula is as follows: ； in, For the j-th indicator , Let j be the minimum value of the j-th indicator. The maximum value of the j-th indicator. Let be the normalized value of the j-th indicator.

7. The method for evaluating the complexity of a multimodal information interface according to claim 1, characterized in that: Step 4 specifically includes: The entropy weight method or the CRITIC method can be used to assign weights to each indicator. The entropy weight method is used to assign weights to each indicator as follows: Calculate the probability: ； in Let be the probability distribution of the j-th indicator for the i-th interface sample. is the normalized value of the j-th indicator of the i-th interface sample, and n is the total number of interface samples; Calculate information entropy: ； Let the information entropy of the j-th indicator be , Normalization factor; Calculate the weights: ； in, Let be the weight of the j-th indicator; G is ; The CRITIC method is used to assign weights to each indicator as follows: Calculate the standard deviation: ； in Let j be the standard deviation of the j-th indicator. Let be the normalized value of the j-th indicator for the i-th interface sample; This represents the mean of the j-th indicator across all n interface samples; the calculation formula is: ； n is the total number of interface samples; Calculate the correlation coefficient: ； Where i represents the interface sample index, i=1, 2, ..., n; Let represent the normalized value of the p-th metric for the i-th interface sample. It is the mean of the p-th indicator across all n interface samples; The value range of is [-1, 1]. The larger the absolute value, the stronger the linear correlation between the p-th indicator and the j-th indicator, and the lower the conflict. This represents the Pearson correlation coefficient between the p-th and j-th indicators; Calculate the total information content: ； in G represents the total amount of information; G is... ; Calculate the weights: ； in, Let be the weight of the j-th indicator; for Total amount of information.

8. The method for evaluating the complexity of a multimodal information interface according to claim 1, characterized in that: Step 5 specifically includes: Calculate the overall score R: in Let be the weight of the j-th indicator; Let be the normalized value of the j-th indicator.

9. A system for evaluating the complexity of a multimodal information interface, implementing the multimodal information interface complexity evaluation method of claim 1, characterized in that: include: Multimodal interface element acquisition module: used to acquire multimodal interface elements, including visual data, auditory data and interaction data; Interface complexity analysis module: used to classify interface complexity into information complexity, visual complexity, auditory complexity, and cognitive complexity; Indicators for calculating information complexity, visual complexity, auditory complexity, and cognitive complexity are used respectively. Indicators for information complexity include the number of sensory modalities, the type of symbolic modality, and cross-modal fusion. Indicators for visual complexity include static information density, dynamic information complexity, and visual consistency. Indicators for auditory complexity include semantic clarity, background noise interference index, and cross-modal synchronization. Indicators for cognitive complexity include layout order rationality, information hierarchy depth, and interactive control logic. Normalization module: Used to normalize all indicators separately; Indicator weight allocation module: used to assign weights to each indicator; Comprehensive scoring and analysis module: This module calculates the comprehensive score by combining the normalized index values ​​with the weights corresponding to each index.

Images