Multi-modal information interface complexity evaluation method and evaluation system

Abstract

The invention discloses a multi-modal information interface complexity evaluation method and system, and the method comprises the steps: collecting multi-modal interface elements which comprise visual data, auditory data and interaction data; decomposing the interface complexity of the multi-modal interface, and dividing the interface complexity into information complexity, visual complexity, auditory complexity and cognitive complexity; respectively calculating an index of information complexity, an index of visual complexity, an index of auditory complexity and an index of cognitive complexity; respectively carrying out normalization processing on all the calculated indexes; allocating a weight to each index; and comprehensive score calculation: calculating a comprehensive score through the normalized index value and the weight corresponding to each index. According to the method, the multi-modal complexity is decomposed into four dimensions of information, vision, auditory sense and cognition, comprehensive analysis is carried out on multi-modal information fusion, a dynamic optimization process is embedded, the evaluation efficiency and the interface availability are remarkably improved, and the practicability is better.

Description

Technical Field

[0001] The present invention belongs to the field of human-computer interaction technology, specifically a multimodal information interface complexity evaluation method and evaluation system based on a multidimensional framework, which is suitable for the interface complexity optimization design of scenarios such as intelligent driving systems, industrial monitoring platforms, smart city interfaces, and complex equipment control systems. Background Art

[0002] With the widespread adoption of multimodal information interfaces, managing their complexity has become a core challenge in interface design. Traditional complexity assessment methods often focus on a single modality (such as vision or hearing), lack comprehensive analysis of multimodal information fusion, and employ fragmented evaluation metrics, limiting their practicality.

[0003] For example:

[0004] 1. Visual complexity assessment: Traditional methods calculate complexity by the number of interface elements or information entropy, but they cannot reflect the timeliness impact of dynamic information (such as real-time video and dynamic maps).

[0005] 2. Auditory complexity assessment: Existing studies mostly focus on the length of voice commands or the intensity of background noise, but lack the synergistic analysis of auditory and visual information.

[0006] 3. Lack of cross-modal coordination: Existing technologies do not quantify the compatibility, degree of fusion, and smoothness of cross-modal conversion of multimodal information, resulting in increased cognitive load for users.

[0007] In addition, existing methods lack a unified evaluation framework and find it difficult to balance information capacity and user cognitive ability.

[0008] Therefore, a systematic evaluation method is needed to address the defect of current multimodal interface complexity evaluation that cannot take multiple factors into account. Summary of the Invention

[0009] The technical problem to be solved by the present invention is to provide a multimodal information interface complexity evaluation method and evaluation system in response to the above-mentioned deficiencies in the existing technology. This evaluation method and evaluation system decompose multimodal complexity into four dimensions: information, vision, hearing, and cognition. They conduct a comprehensive analysis of multimodal information fusion and embed a dynamic optimization process, which significantly improves the evaluation efficiency and interface usability, and is more practical.

[0010] In order to achieve the above technical objectives, the technical solution adopted by the present invention is:

[0011] A method for evaluating the complexity of a multimodal information interface comprises the following steps:

[0012] Step 1: Collect multimodal interface elements, which include visual data, auditory data, and interaction data;

[0013] Step 2: Decompose the interface complexity of the multimodal interface and divide it into information complexity, visual complexity, auditory complexity, and cognitive complexity; calculate the indicators of information complexity, visual complexity, auditory complexity, and cognitive complexity respectively; the indicators of information complexity include the number of sensory modalities, the type of symbolic modality, and the degree of cross-modal fusion; the indicators of visual complexity include static information density, dynamic information complexity, and visual consistency; the indicators of auditory complexity include semantic clarity, background noise interference index, and cross-modal synchronization; the indicators of cognitive complexity include layout order rationality, information hierarchy depth, and interactive control logic;

[0014] Step 3: Normalization: Normalize all the indicators calculated in step 2 separately;

[0015] Step 4: Indicator weight assignment: assign weights to each indicator;

[0016] Step 5: Calculate the comprehensive score: Calculate the comprehensive score by combining the normalized indicator values ​​with the weight corresponding to each indicator.

[0017] As a further improved technical solution of the present invention, in step 1, the visual data includes static text, static icons, static graphics, dynamic videos, animations and real-time monitoring images; the auditory data includes voice commands and background noise intensity; and the interactive data includes user operation paths, response times and error rates.

[0018] As a further improved technical solution of the present invention, the number of sensory modalities N s The calculation formula is:

[0019]

[0020] Where m is the total number of sensory modalities preset in the interface; δ k Indicates whether the kth sensory modality is used in the interface; sensory modalities include vision, hearing, touch, and smell;

[0021] The symbol modal type T m The calculation formula is:

[0022]

[0023] Where U is the total number of symbol types preset in the interface, γ μ Indicates whether the μth symbol type appears in the interface; symbol types include static text, static icon and static graphic;

[0024] The cross-modal fusion degree F c The calculation formula is:

[0025] F c =α·C s +β·S t , (α+β=1);

[0026] Among them, C s Compatibility scores or task completion rates are used to measure the synergy between different sensory modalities;

[0027] When C s When scoring compatibility, C s The calculation process is as follows: recruit multiple representative user samples, conduct information retrieval and decision-making task experiments, require users to rate based on the consistency and synergy of information between modalities, and calculate the average value A of all user ratings. s , and the compatibility score is calculated by the following formula:

[0028] When C s When C is the task completion rate, s The calculation process is as follows: recruit multiple representative user samples, conduct a multimodal collaborative task completion rate test task, require users to successfully complete the task when the multimodal information is coordinated correctly, record the task completion rate, and the task completion rate is the number of times the task is successfully completed divided by the total number of tasks;

[0029] S t Transition fluency is used to measure the smoothness of users switching between different modes;

[0030] Transition Fluency S t The calculation formula is:

[0031] or

[0032] Among them, T avg T represents the average time for multiple users to complete the test task of switching between multiple modes. base is the set benchmark time; E is the number of incorrect operations caused by the user due to poor mode switching in the test task of switching between modes, E max The maximum number of allowed errors is set; is the error rate;

[0033] α is C s The weight coefficient of S t The weight coefficient of .

[0034] As a further improved technical solution of the present invention, the static information density D static The calculation formula is:

[0035]

[0036] Among them, N y represents the number of static elements of type y, w y is the weight coefficient of the y-th type of static elements, T is the total number of static elements; N text is the number of static text elements in the interface; N icon is the number of static icon elements in the interface;

[0037] N graphic is the number of static graphic elements in the interface; w text is the weight coefficient of the static text element; w icon is the weight coefficient of the static icon element; w graphic is the weight coefficient of the static graphic element; A interface is the total area of ​​the interface;

[0038] The dynamic information complexity D dynamic The calculation formula is:

[0039]

[0040] Among them, Vr f is the refresh rate of the f-th type of dynamic elements, Fr f is the frame rate or effective visual update frequency of the f-th type of dynamic element, Area f is the display area of ​​the f-th type of dynamic element in the interface, A total is the total area of ​​the interface; V max ×F max It is a reference value of the product of the preset maximum refresh rate and the maximum frame rate; the types of dynamic elements include dynamic video, animation and real-time monitoring screen;

[0041] The visual consistency V c The calculation formula is:

[0042] V c =γ·M d +(1-γ)·L c , (0≤γ≤1);

[0043] Among them, M d is the layout matching degree, which is calculated as follows:

[0044]

[0045] The average offset distance refers to the average visual spatial misalignment distance between all dynamic elements and their associated static elements in the interface; the maximum allowable offset distance is a preset, tolerable upper limit of visual offset;

[0046] Lc Score logical consistency, calculated as:

[0047]

[0048] The user task completion rate ranges from 0 to 1. Specifically, a representative sample of users is recruited and asked to complete a set of standardized test tasks. The effective completion of these tasks depends on the reasonable information logic connection between the dynamic and static elements of the interface. The user task completion rate is recorded, which is equal to the number of times the user successfully completes the task divided by the total number of tasks. The expert score ranges from 0 to 1. Specifically, experts in the field of human-computer interaction or interface design independently evaluate the logical rationality of the dynamic and static elements in the interface in terms of information expression, causal relationships, and process coordination. The experts use a continuous scale of 0 to 1 to score, and the average of all expert scores is calculated as the final expert score.

[0049] γ is the weight coefficient.

[0050] As a further improved technical solution of the present invention, the semantic clarity S atotal The calculation formula is:

[0051]

[0052] Where B is the total number of voice commands in the interface, b is the sequence number of the voice command; S a,b is the single average semantic clarity of the bth voice command in the interface;

[0053] S a,b The calculation formula is:

[0054]

[0055] Where H is the number of times the voice recognition system in the interface recognizes the bth voice command, that is, the number of times the voice recognition system in the interface repeatedly recognizes the bth voice command; h represents the number of times the voice recognition system in the interface recognizes the bth voice command, h = 1, 2, ..., H; T ref,b is the reference text of the bth voice instruction, T rec,b,h The text obtained by the speech recognition system in the interface after the bth voice command is recognized for the hth time;

[0056] Similarity(T ref,b ,T rec,b,h ) is the semantic similarity calculation function, the specific method is: the reference text T ref,b and recognized text T rec,b,hThe BERT fine-tuning model is input respectively, and the output vector corresponding to the CLS tag is extracted as the sentence vector to obtain the corresponding sentence vector representation. Then the cosine similarity of the two sentence vectors is calculated, and the cosine similarity is normalized to obtain Similarity(T ref,b ,T rec,b,h ) value;

[0057] The background noise interference index N interference The calculation formula is:

[0058]

[0059] where dB norm =min(dB,100), dB is the decibel value of background noise, P n is the ratio of background noise signal to the total signal;

[0060] The cross-modal synchronization S av The calculation formula is:

[0061]

[0062] Where Δt is the average time delay between auditory and visual signals, calculated by timestamp alignment; T max The maximum allowed delay threshold is set to 50ms.

[0063] As a further improved technical solution of the present invention, the layout order rationality L o The calculation formula is:

[0064]

[0065] Where C is the total number of attention areas divided in the interface;

[0066] p c is the reference text of the c-th attention region, which is calculated as:

[0067]

[0068] Q c is the number of fixations of the c-th attention region, Q 总 is the total number of fixations;

[0069] The information level depth H d The calculation formula is:

[0070] H d =L·log2(B+1);

[0071] L is the total number of levels in the user operation path; B is the branching factor, that is, the average number of branches at each level, calculated as: Among them, b l Indicates the number of branches at each level in the user operation path;

[0072] The interactive control logic I r The calculation formula is:

[0073]

[0074] Among them, R s The redundancy of user operation steps is calculated as: R s = Actual number of steps - theoretical minimum number of steps; R max is the preset maximum redundancy threshold, R max =5.

[0075] As a further improved technical solution of the present invention, the step 3 is specifically as follows:

[0076] The step 3 is specifically as follows:

[0077] Normalize the values ​​of each indicator in step 2 separately, and the calculation formula is:

[0078]

[0079] Among them, X j is the value of the jth indicator calculated according to the corresponding formula in step 2, X min is the minimum value of the jth index, X max is the maximum value of the jth index, x j is the normalized value of the j-th indicator.

[0080] As a further improved technical solution of the present invention, the step 4 is specifically as follows:

[0081] Use the entropy weight method to assign weights to each indicator or use the CRITIC method to assign weights to each indicator;

[0082] The entropy weight method is used to assign weights to each indicator as follows:

[0083] Calculate the probability:

[0084]

[0085] where p ij is the probability distribution of the jth indicator of the i-th interface sample, x ij is the normalized value of the jth index of the i-th interface sample, and n is the total number of interface samples;

[0086] Calculate information entropy:

[0087]

[0088] e j is the information entropy of the jth indicator, ln(n) is the normalization factor;

[0089] Calculate weights:

[0090]

[0091] Among them, w j is the weight of the jth indicator; e g is the information entropy of the g-th indicator, G is the total number of indicators;

[0092] The CRITIC method is used to assign weights to each indicator as follows:

[0093] Calculate the standard deviation:

[0094]

[0095] where σ j is the standard deviation of the jth indicator, x ij is the normalized value of the jth indicator of the i-th interface sample;

[0096] It represents the mean value of the jth indicator on all n interface samples; the calculation formula is:

[0097]

[0098] n is the total number of interface samples;

[0099] Calculate the correlation coefficient:

[0100]

[0101] Where i represents the interface sample index, i = 1, 2, ..., n; x ip represents the normalized value of the pth indicator of the i-th interface sample, is the mean of the pth indicator on all n interface samples; r pj The value range of is [-1,1]. The larger its absolute value is, the stronger the linear correlation between the pth indicator and the jth indicator is, and the lower the conflict is. pj represents the Pearson correlation coefficient between the pth indicator and the jth indicator;

[0102] Calculate the comprehensive information volume:

[0103]

[0104] Among them C j is the comprehensive information volume; G is the total number of indicators;

[0105] Calculate weights:

[0106]

[0107] Among them, w j is the weight of the jth indicator; C g is the comprehensive information amount of the g-th indicator.

[0108] As a further improved technical solution of the present invention, the step 5 is specifically as follows:

[0109] Calculate the comprehensive score R:

[0110]

[0111] where w j is the weight of the jth indicator; x j is the normalized value of the j-th indicator.

[0112] In order to achieve the above technical objectives, another technical solution adopted by the present invention is:

[0113] A multimodal information interface complexity evaluation system, comprising:

[0114] Multimodal interface element acquisition module: used to acquire multimodal interface elements, including visual data, auditory data and interaction data;

[0115] Interface complexity analysis module: used to classify interface complexity into information complexity, visual complexity, auditory complexity, and cognitive complexity; used to calculate the indicators of information complexity, visual complexity, auditory complexity, and cognitive complexity respectively; the indicators of information complexity include the number of sensory modalities, symbolic modality types, and cross-modal fusion; the indicators of visual complexity include static information density, dynamic information complexity, and visual consistency; the indicators of auditory complexity include semantic clarity, background noise interference index, and cross-modal synchronization; the indicators of cognitive complexity include layout order rationality, information hierarchy depth, and interactive control logic;

[0116] Normalization processing module: used to normalize all indicators separately;

[0117] Indicator weight allocation module: used to assign weights to each indicator;

[0118] Comprehensive score analysis module: used to calculate the comprehensive score through the normalized indicator values ​​and the weight corresponding to each indicator.

[0119] The beneficial effects of the present invention are:

[0120] This paper designs a four-dimensional dynamic evaluation framework: for the first time, it integrates information, vision, hearing, and cognitive complexity, covering all elements of multimodal interfaces. It also employs a method-coupled design approach: with processes at the core and algorithms as supporting tools, it ensures that evaluation results are interpretable and implementable. It also possesses real-time optimization capabilities: through dynamic data feedback and algorithmic iteration, it supports real-time adjustments to interface designs.

[0121] This invention integrates dynamic evaluation models and processes into algorithmic design to achieve complexity quantification, weight assignment, comprehensive scoring, and optimal selection. Its core innovation lies in decomposing multimodal complexity into four dimensions: information, vision, hearing, and cognition. These dimensions are then embedded in a dynamic optimization process, significantly improving evaluation efficiency and interface usability. BRIEF DESCRIPTION OF THE DRAWINGS

[0122] Figure 1 Flowchart for complexity assessment of multimodal information interfaces. DETAILED DESCRIPTION

[0123] The specific embodiments of the present invention are further described below with reference to the accompanying drawings:

[0124] A method for evaluating the complexity of multimodal information interfaces, such as Figure 1 As shown, Figure 1 This is the complete process from step 1 to step 5, highlighting the logical relationship between data collection, analysis, weight allocation, scoring and optimization. Specifically, it includes the following steps:

[0125] Step 1: Collect multimodal interface elements through the camera, microphone, and interaction log recorder. Multimodal interface elements include visual data, auditory data, and interaction data.

[0126] Visual data includes static text, static icons, static graphics (including charts, maps, etc.), dynamic videos, animations, and real-time monitoring images. Auditory data includes voice commands and background noise intensity. Interaction data includes user operation paths, response times, and error rates.

[0127] Data preprocessing: Temporal alignment of dynamic information and synchronous tagging of cross-modal data (such as speech and visual cues).

[0128] Step 2: Decompose the interface complexity of the multimodal interface, dividing it into information complexity, visual complexity, auditory complexity, and cognitive complexity. Calculate the first-level indicators of information complexity, visual complexity, auditory complexity, and cognitive complexity. Information complexity indicators include the number of sensory modalities, symbolic modality types, and cross-modal fusion. Visual complexity indicators include static information density, dynamic information complexity, and visual consistency. Auditory complexity indicators include semantic clarity, background noise interference index, and cross-modal synchronization. Cognitive complexity indicators include layout order rationality, information hierarchy depth, and interactive control logic.

[0129] Among them, the number of sensory modalities N s : Quantify the number of sensory channels involved in the interface (e.g., vision, hearing, touch).

[0130] Number of sensory modalities N s The calculation formula is:

[0131]

[0132] Where m is the total number of sensory modalities preset in the interface (such as vision, hearing, touch, smell, etc.); δ k Indicates whether the kth sensory modality is used in the interface; sensory modalities include vision, hearing, touch, and smell.

[0133] Operation method: traverse all m sensory modalities, record them as 1 if they exist, otherwise 0, and finally accumulate all values. If the interface includes vision and touch, then N s =1+1=2; if only hearing is included, then N s =1.

[0134] Among them, the symbolic modal type T m : Classify statistical symbol element types (such as static text, static icon and static graphic).

[0135] Symbolic modal type T m The calculation formula is:

[0136]

[0137] Where U is the total number of symbol types preset in the interface (such as static text, static icons and static graphics (including charts, maps, etc.), γ μ Indicates whether the μth symbol type appears in the interface; symbol types include static text, static icon, and static graphic.

[0138] Operation method: Count the number of symbol types in the interface, and add 1 for each type. If the interface contains static text and static graphics, then T m =1+1=2; if only static icons are included, then T m =1.

[0139] Among them, the cross-modal fusion degree F c :Pass the compatibility score (C s ) and transition fluency (S t )Weighted calculation:

[0140] F c =α·C s +β·S t , (α+β=1);

[0141] Among them, C s Compatibility score or task completion rate is used to measure the synergy between different sensory modalities (such as consistency between vision and hearing).

[0142] When C s When scoring compatibility, the quantitative method is: user subjective scoring. Recruit a representative user sample (e.g., n ≥ 15 people). Conduct information retrieval and decision-making task experiments. In the experiment, the interface will simultaneously present visual text and auditory speech. Users need to locate the target element pointed to by the auditory instruction in the visual interface and complete the mark or perform the operation. The detailed experimental process is as follows:

[0143] Pre-experiment: familiarize yourself with interface operations and modal mapping rules;

[0144] Experimental materials: Select interfaces involving multiple core functions in the evaluation interface to complete multiple groups of tests;

[0145] Experimental Content: After each task, participants were rated on a 5-point Likert scale (1 = extremely incompatible, 5 = extremely compatible). Scoring criteria included: consistency of modal information (e.g., whether the audio and text descriptions contradict each other); timing coordination of modal presentation (e.g., whether the audio played within 0.5-2 seconds of the visual element); and information complementarity (e.g., whether the auditory component supplemented key details not presented visually). Scoring criteria can be modified to meet the specific needs of different product interfaces.

[0146] After completing the experimental task, users were asked to rate the consistency and synergy of the information between the modalities on a Likert scale ranging from 1 (extremely incompatible) to 5 (extremely compatible).

[0147] Calculate the average A of all user ratings s , and the compatibility score is calculated by the following formula:

[0148] When Cs When it comes to task completion rate, the quantitative method is: objective user scoring. Recruit a representative user sample (e.g., n ≥ 15 people). Conduct a multimodal collaborative task completion rate test task, requiring users to successfully complete the task only when the multimodal information is correctly coordinated. The detailed experimental process is as follows:

[0149] Pre-experiment: familiarize yourself with the experimental rules of interface operation and modal collaboration;

[0150] Experimental Materials: Multiple modalities related to multiple core functions in the evaluation interface were selected to form the multiple test materials required for the experiment. This included three groups: a full-modality group; a single-modality loss group: blocking any one of the visual / auditory / tactile senses and observing the decrease in task completion rate; and a modality conflict group: intentionally setting contradictory information (e.g., visual display "normal" but voice says "fault") and recording the error rate.

[0151] Experimental content: 1. Group testing: Each subject is required to complete three tasks: "full modality", "single modality missing", and "modal conflict"; 2. Task timing: Set a task time limit, and timeout is recorded as incomplete; 3. Data recording: Record the completion status (success / failure), completion time, and error operation type of each group of tasks.

[0152] Record the task completion rate (number of successes / total number of tasks). Use the completion rate directly as C s The value of C s =Task completion rate.

[0153] S t Transition fluency is used to measure the smoothness of users switching between different modes (e.g., switching time, error rate). Quantification method: To evaluate modal switching fluency, a representative sample of users (e.g., n ≥ 15 people) was recruited. Multiple modal switching tasks involving vision, hearing, and touch were conducted. This was used to quantify the cognitive load and operational efficiency of users switching between different modes. The detailed experimental flow is as follows:

[0154] Pre-experiment: familiarize yourself with the interface mode switching rules;

[0155] Experimental materials: Different switching materials are formulated according to different modalities (such as switching from visual mode to auditory mode. Visual mode stage: find the red button in the interface; auditory mode stage: perform the operation after hearing the voice command "press and hold for 3 seconds"; this completes the switch from visual mode to auditory mode).

[0156] Experimental content: 1. Setting the benchmark time T for mode switching through expert group testing base(For reference: Invite five interface design experts to complete 20 tasks, and take the average of the best 10 times.) 2. Complete multiple sets of switching tasks, where each set includes three rounds of modal switching (visual → auditory → touch → visual cycle). Repeat these multiple sets of tasks, and record the average time (Tavg) for each set.

[0157] Transition fluency is calculated as:

[0158]

[0159] Values ​​closer to 1 indicate smoother switching.

[0160] Based on the number of wrong operations: In the modal switching test task, record the number of wrong operations E caused by the user due to poor modal switching (such as wrong clicks, wrong voice commands). Set a maximum allowable number of errors E max (e.g. total number of steps in a task × 0.2). Transition fluency is calculated as:

[0161]

[0162] α is C s The weight coefficient of S t The weight coefficient of .

[0163] Among them, the static information density D static : Quantify the density of static visual elements per unit area (visual elements that do not change over time at a specific time point, such as text, icons, graphics, etc.), reflecting the visual complexity of static information on the interface.

[0164] Static information density D static The calculation formula is:

[0165]

[0166] Among them, N y represents the number of static elements of type y, w y is the weight coefficient of the y-th type of static elements, T is the total number of static elements; N text N is the number of static text elements in the interface (unit: pieces), which refers to independent text blocks or labels; icon N is the number of static icon elements in the interface (unit: pieces), which refers to graphic symbols with specific meanings; graphic The number of static graphic elements in the interface (unit: pieces), which refers to basic visual components such as lines, shapes, and non-dynamic chart bodies;

[0167] w text 、w icon 、w graphicare weight coefficients for static text elements, static icon elements, and static graphic elements (reflecting the relative impact of different types of elements on visual complexity, which can be set through experiments or expert evaluation when applied to different interfaces); A interface is the total area of ​​the interface (unit: square pixels or square meters).

[0168] Calculation method: weighted sum of element types and then divided by area. The larger the value, the denser the static information. If the interface has 10 texts (weight 1.0), 5 icons (weight 1.2), and 3 graphics (weight 0.8), and the area is 2 square meters, then:

[0169]

[0170] Among them, the dynamic information complexity D dynamic : Measures dynamic elements (visual elements that change continuously over time, such as:

[0171] The contribution of dynamic videos, animations, and real-time monitoring images to visual complexity.

[0172] Dynamic Information Complexity D dynamic The calculation formula is:

[0173]

[0174] Among them, Vr f is the refresh rate of the f-th dynamic element (unit: Hz), Fr f is the frame rate or effective visual update frequency of the f-th type of dynamic element (unit: FPS), Area f is the display area of ​​the f-th type of dynamic element in the interface, A total is the total area of ​​the interface; V max ×F max This is the reference value for the product of the preset maximum refresh rate and maximum frame rate (e.g., 100 Hz × 120 FPS = 12000). This value is used to normalize the area-weighted sum of the product of the refresh rate and frame rate of dynamic elements, mapping it to a relative scale and serving as a reference point for calculating logarithmic complexity. A value of +1 is used to avoid negative infinity when the logarithm is zero. The logarithmic base is chosen based on the fact that a base of 2 compresses high dynamic range and conforms to the nonlinear characteristics of human visual perception. Dynamic elements include dynamic video, animation, and real-time surveillance footage.

[0175] Calculation method: Calculate the product of the refresh rate and frame rate of all dynamic elements (such as dynamic video, animation, and real-time monitoring images), and then weight the sum based on their screen area. The weighted sum is then normalized and the logarithm is taken. A larger result indicates greater dynamic complexity.

[0176] Example: If the interface contains two types of dynamic elements. Element A: Vr A =30Hz,Fr A =60FPS,Area A Total area A total 30% of element B:Vr B =20Hz,Fr B =40FPS,Area B Total area A total 20% of VmaxFmax = 12000. Then:

[0177]

[0178] Among them, visual consistency V c :Through the matching degree of dynamic and static element layout (M d ) and logical consistency (L c ) Comprehensively evaluate the synergy and logical rationality of dynamic and static elements:

[0179] V c =γ·M d +(1-γ)·L c , (0≤γ≤1).

[0180] Among them, M d It is the layout matching degree (0-1), which is used to measure the alignment and coordination degree of dynamic elements and their associated static elements in the visual space. The calculation method is:

[0181]

[0182] The average offset distance refers to the average visual spatial misalignment between all dynamic elements and their associated static elements in the interface (for example, the pixel distance between their visual centers or key reference points can be calculated). The maximum allowable offset distance is a preset, tolerable upper limit for visual misalignment. Exceeding this limit is considered an extremely poor layout match (which can be determined through expert evaluation or based on interface dimensions).

[0183] L c Score the logical consistency (0-1) as follows:

[0184]

[0185] User task completion rate (0-1): Recruit a representative sample of users (e.g., n≥10 people) and ask them to complete a set of standardized test tasks. The effective completion of these tasks depends on the reasonable information logic association between the dynamic and static elements of the interface (for example, interpreting the meaning of the corresponding static labels based on the changes in dynamic charts); record the rate at which users successfully complete the tasks (number of successes / total number of tasks). Expert rating (0-1) (also known as expert logical consistency rating): Experts in the field of human-computer interaction or interface design (e.g., ≥3 people) independently evaluate the logical rationality of the dynamic and static elements in the interface in terms of information expression, cause-and-effect relationships, and process coordination. Experts use a continuous scale from 0 (completely illogical) to 1 (perfectly logically self-consistent) to rate. The average of all expert ratings is calculated as the expert logical consistency rating.

[0186] γ is the weight coefficient, which is adjusted dynamically according to the scenario (e.g., γ = 0.8 for game interface and γ = 0.5 for tool software).

[0187] Operation method: weighted sum and then normalized. The closer the value is to 1, the higher the consistency. d =0.75,L c =0.85,γ=0.6, then V c =0.6×0.75+0.4×0.85=0.79.

[0188] Among them, semantic clarity S atotal The calculation process is as follows:

[0189] First, calculate S a,b , represents the semantic similarity score (0-1) based on the output of the natural language processing model, which is used to evaluate the average semantic matching degree of the same voice command in multiple tests.

[0190]

[0191] Among them, S a,b represents the single average semantic clarity of the bth instruction. H is the number of times the speech recognition system in the interface recognizes the bth voice instruction, that is, the number of repeated tests of the bth voice instruction; h represents the test sequence number of the bth voice instruction, h = 1, 2, ..., H, used to traverse each test data; T ref,b is the reference text (original semantic content) of the bth voice instruction. One instruction has only one original semantic content for reference; T rec,b,h The text obtained by the speech recognition system in the interface after performing the h-th recognition conversion on the b-th speech instruction. Similarity(T ref,b ,T rec,b,h) is the semantic similarity calculation function, which is implemented using sentence embedding technology based on pre-trained language models (such as BERT). The specific method is: ref,b and recognized text T rec,b,h The BERT fine-tuning model is input respectively, and the output vector corresponding to the CLS tag is extracted as the sentence vector to obtain the corresponding sentence vector representation. Then the cosine similarity of the two sentence vectors is calculated, and the cosine similarity is normalized to obtain Similarity(T ref,b ,T rec,b,h ). Cosine similarity ranges from -1 to 1, but in this type of model, semantically similar sentence pairs often produce positive values ​​close to 1. The similarity value output by this function has been normalized to the appropriate range using (cosine similarity + 1) / 2.

[0192] Calculation method: The similarity between multiple sets of speech recognition results and reference text is averaged. The closer the value is to 1, the clearer the meaning. If the voice command "adjust air conditioning temperature" is tested three times, and the recognition results are ① "adjust air conditioning temperature" (similarity 1.0) ② "adjust air conditioning humidity" (similarity 0.7) ③ "adjust car temperature" (similarity 0.8), then:

[0193]

[0194] Secondly, after completing the evaluation of the average semantic matching of the same voice instruction in multiple tests, we can further realize the comprehensive evaluation of multiple instructions, that is, the semantic clarity S atotal :

[0195]

[0196] S atotal Reflects the overall semantic recognition accuracy of all voice commands in the interface. A larger value indicates a higher overall clarity. B is the total number of voice commands in the interface (e.g., 5 different commands); b is the command number. Traverse the bth command and call its single command semantic clarity S in turn. a,b To accumulate. Assume that the interface contains 3 instructions: Instruction 1: S a,1 =0.9 (average similarity 0.9), instruction 2: S a,2 =0.7, instruction 3: S a,3 =0.8, then the comprehensive clarity is:

[0197]

[0198] Among them, the background noise interference index N interference : It is used to quantify the degree of interference of environmental noise (i.e. background noise) on the transmission of auditory information. The calculation formula is:

[0199] where dB norm =min(dB,100);

[0200] dB is the decibel value of background noise (unit: dB), and the measurement range is 0-120dB. n The ratio of background noise signal to the total signal (0-1, such as 0.3 means 30%); dB norm This is the normalized decibel value, limited to a maximum of 100dB to avoid overflow.

[0201] Calculation method: Normalize the decibel value and multiply it by the noise ratio. The larger the value, the stronger the interference. If dB=85, P n =0.6, then:

[0202]

[0203] Among them, cross-modal synchronization S av , used to evaluate the temporal alignment of auditory and visual signals to ensure the continuity of user experience. The calculation formula is:

[0204]

[0205] Where Δt is the average time delay between auditory and visual signals (unit: milliseconds), calculated by timestamp alignment; T max The maximum allowable delay threshold is set to 50ms (based on the common standard of real-time interaction scenarios).

[0206] Calculation method: The smaller the delay, the higher the score. If it exceeds the threshold, the score is 0. If Δt = 30ms, then:

[0207]

[0208] Among them, the layout order rationality L o : Analyze user attention distribution through eye tracking heat map and calculate information entropy:

[0209] Layout order rationality L o The calculation formula is:

[0210]

[0211] Where C is the total number of attention areas divided in the interface; p c is the reference text (original semantic content) of the cth attention region, which is calculated as:

[0212]

[0213] Q c is the number of fixations of the c-th attention region, Q总 is the total number of fixations.

[0214] Calculation method: The gaze probability of each area is calculated through eye tracking data, and the degree of concentration of attention distribution is quantified using information entropy. The higher the entropy value, the more scattered the user's attention is and the lower the rationality of the layout is. Therefore, L o is a positive indicator of complexity (i.e., a larger original value indicates a higher complexity). If the interface is divided into three areas, with gaze counts of 50, 30, and 20 respectively, and a total gaze count of 100, then:

[0215] p1=0.5,p2=0.3,p3=0.2,L o =-(0.5log20.5+0.3log20.3+0.2log20.2)≈1.49 (bits).

[0216] Among them, the information level depth H d : It is used to quantify the depth and breadth of the navigation path, reflecting the difficulty for users to find information. The calculation formula is:

[0217] H d =L·log2(B+1);

[0218] L is the total number of levels in the user's operation path (e.g., homepage → submenu → details page, the number of levels is 3); B is the branching factor, which is the average number of branches (entry / option number) at each level. It is calculated by counting the number of direct child nodes (branches) b at each level (except the last leaf node) in the navigation structure. l

[0219] (l represents the level index), which indicates the number of branches at each level in the user operation path; then the average value of all levels with branches is calculated:

[0220]

[0221] The +1 in log2(B+1) is used to avoid the meaningless logarithm when the branching factor is 0. (

[0222] Example: If level 1 has 5 branches, level 2 has 3 branches, and level 3 is a leaf node with no branches, then

[0223] Among them, the interactive control logic I r :Used to evaluate the simplicity of the interaction process:

[0224] The calculation formula is:

[0225]

[0226] Among them, R sThe redundancy of user operation steps is calculated as: R s = Actual number of steps - theoretical minimum number of steps; R max is the preset maximum redundancy threshold, R max =5.

[0227] Operation method: the ratio of redundancy to the maximum threshold. The result is limited to 0-1. The larger the value, the more redundant the interaction logic. r It is a positive indicator of complexity (i.e., the larger the original value, the higher the complexity). If the task requires at least 3 steps to complete, and the user actually takes 7 steps, R max =5, then R s =7-3=4,I r =min(4 / 5,1)=0.8.

[0228] Step 3: Normalization: Normalize all the indicators calculated in step 2 separately, that is, eliminate the dimensional differences between the indicators and unify the direction (all indicators are converted to positive, and the larger the value, the better).

[0229] The step 3 is specifically as follows:

[0230] Normalize the values ​​of each indicator in step 2 separately, and the calculation formula is:

[0231]

[0232] Among them, X j is the value of the jth indicator calculated according to the corresponding formula in step 2 (for example, the number of sensory modalities Ns, static information density Dstatic, information hierarchy depth Hd, etc. calculated in step 2), X min is the minimum value of the jth indicator (historical data or preset range), X max is the maximum value of the jth indicator (historical data or preset range), x j is the normalized value of the jth indicator (range 0-1). A larger value indicates better performance. If the number of sensory modalities (Ns) is assumed to be a negative indicator, the original value = 3, and the preset range is 1-5:

[0233]

[0234] Step 4: Indicator weight assignment: assign weights to each indicator;

[0235] Assign weights to indicators based on their information content or conflict, and distinguish their importance. Use the entropy weight method or the CRITIC method to assign weights to each indicator.

[0236] Method 1: Entropy Weight Method (suitable for high-dimensional data with weak correlation between indicators.)

[0237] 1. Calculate probability:

[0238]

[0239] where p ij is the probability distribution of the jth indicator of the i-th interface sample, reflecting the proportion of indicator j in different samples, x ij is the normalized value of the jth index of the i-th interface sample, and n is the total number of interface samples;

[0240] 2. Calculate information entropy:

[0241]

[0242] e j is the information entropy of the jth indicator. A larger entropy value indicates more dispersed data and lower information content. ln(n) is the normalization factor to ensure that the entropy value is within a reasonable range.

[0243] 3. Calculate weight:

[0244]

[0245] Among them, w j is the weight of the jth indicator; e g is the information entropy of the g-th indicator, G is the total number of indicators; the smaller the entropy value (the greater the amount of information), the greater the weight w j The higher.

[0246] Method 2: CRITIC method (strong correlation between indicators).

[0247] 1. Calculate the standard deviation to measure the volatility of the data. Indicators with high volatility contain more independent information and have higher weights:

[0248]

[0249] where σ j is the standard deviation of the jth indicator, x ij is the normalized value of the jth indicator of the i-th interface sample; n is the number of interface samples (such as the number of interfaces evaluated).

[0250] It represents the mean value of the jth indicator on all n interface samples; the calculation formula is:

[0251]

[0252] n is the total number of interface samples;

[0253] 2. Calculate the correlation coefficient: Calculate the Pearson correlation coefficient between any two indicators:

[0254]

[0255] Where i represents the interface sample index, i = 1, 2, ..., n; x ip represents the normalized value of the pth indicator of the i-th interface sample, is the mean of the pth indicator on all n interface samples; r pj The value range of is [-1,1]. The larger its absolute value is, the stronger the linear correlation between the pth indicator and the jth indicator is, and the lower the conflict (independence) is. pj represents the Pearson correlation coefficient between the pth indicator and the jth indicator;

[0256] 3. Calculate the comprehensive information volume:

[0257]

[0258] Among them C j is the comprehensive information content, and the indicators with high volatility and strong conflict have higher weights. G is the total number of indicators; σ j It is the independent information, which measures the volatility of the indicator itself.

[0259] 4. Calculate weight: The comprehensive information C j Normalized and converted into weights. The larger the value, the more important the indicator:

[0260]

[0261] Among them, w j is the weight of the jth indicator; C g is the comprehensive information amount of the g-th indicator.

[0262] Step 5: Calculate the overall score: Calculate the overall score by combining the normalized indicator values ​​with the corresponding weights for each indicator. Weighted summation of the normalized indicators quantifies the overall complexity of the interface.

[0263] The step 5 is specifically as follows:

[0264] Calculate the comprehensive score R:

[0265]

[0266] where w j is the weight of the jth indicator; x j is the normalized value of the jth indicator. The comprehensive score R ranges from 0 to 1. A larger value indicates a higher degree of interface usability or optimization, that is, a lower degree of interface complexity.

[0267] Table 1 shows the four-dimensional framework and indicator hierarchy diagram, showing the classification of the first-level indicators and second-level indicators in the four dimensions of information, vision, hearing, and cognition. Table 1:

[0268]

[0269]

[0270] This embodiment also provides a multimodal information interface complexity assessment system, including:

[0271] Multimodal interface element acquisition module: used to acquire multimodal interface elements, including visual data, auditory data and interaction data;

[0272] Interface complexity analysis module: used to classify interface complexity into information complexity, visual complexity, auditory complexity, and cognitive complexity; used to calculate the indicators of information complexity, visual complexity, auditory complexity, and cognitive complexity respectively; the indicators of information complexity include the number of sensory modalities, symbolic modality types, and cross-modal fusion; the indicators of visual complexity include static information density, dynamic information complexity, and visual consistency; the indicators of auditory complexity include semantic clarity, background noise interference index, and cross-modal synchronization; the indicators of cognitive complexity include layout order rationality, information hierarchy depth, and interactive control logic;

[0273] Normalization processing module: used to normalize all indicators separately;

[0274] Indicator weight allocation module: used to assign weights to each indicator;

[0275] Comprehensive score analysis module: used to calculate the comprehensive score through the normalized indicator values ​​and the weight corresponding to each indicator.

[0276] The protection scope of the present invention includes but is not limited to the above embodiments. The protection scope of the present invention is based on the claims. Any replacement, deformation, and improvement of the technology that can be easily thought of by those skilled in the art fall within the protection scope of the present invention.

Claims

1. A method for evaluating the complexity of a multimodal information interface, characterized by: The following steps are involved: Step 1: Collect multimodal interface elements, which include visual data, auditory data, and interaction data; Step 2: Decompose the interface complexity of the multimodal interface and divide it into information complexity, visual complexity, auditory complexity, and cognitive complexity; calculate the indicators of information complexity, visual complexity, auditory complexity, and cognitive complexity respectively; the indicators of information complexity include the number of sensory modalities, the type of symbolic modality, and the degree of cross-modal fusion; the indicators of visual complexity include static information density, dynamic information complexity, and visual consistency; the indicators of auditory complexity include semantic clarity, background noise interference index, and cross-modal synchronization; the indicators of cognitive complexity include layout order rationality, information hierarchy depth, and interactive control logic; Step 3: Normalization: Normalize all the indicators calculated in step 2 separately; Step 4: Indicator weight assignment: assign weights to each indicator; Step 5: Calculate the comprehensive score: Calculate the comprehensive score by combining the normalized indicator values ​​with the weight corresponding to each indicator.

2. The method for evaluating the complexity of a multimodal information interface according to claim 1, wherein: In step 1, the visual data includes static text, static icons, static graphics, dynamic videos, animations and real-time monitoring images; the auditory data includes voice commands and background noise intensity; and the interactive data includes user operation paths, response times and error rates.

3. The method for evaluating the complexity of a multimodal information interface according to claim 1, wherein: The number of sensory modalities N s The calculation formula is: Where m is the total number of sensory modalities preset in the interface; δ k Indicates whether the kth sensory modality is used in the interface; sensory modalities include vision, hearing, touch, and smell; The symbol modal type T m The calculation formula is: Where U is the total number of symbol types preset in the interface, γ μ Indicates whether the μth symbol type appears in the interface; symbol types include static text, static icon and static graphic; The cross-modal fusion degree F c The calculation formula is: F c =α·C s +β·S t ,(α+β=1); Among them, C s Compatibility scores or task completion rates are used to measure the synergy between different sensory modalities; When C s When scoring compatibility, C s The calculation process is as follows: recruit multiple representative user samples, conduct information retrieval and decision-making task experiments, require users to rate based on the consistency and synergy of information between modalities, and calculate the average value A of all user ratings. s , and the compatibility score is calculated by the following formula: When C s When C is the task completion rate, s The calculation process is as follows: recruit multiple representative user samples, conduct a multimodal collaborative task completion rate test task, require users to successfully complete the task when the multimodal information is coordinated correctly, record the task completion rate, and the task completion rate is the number of times the task is successfully completed divided by the total number of tasks; S t Transition fluency is used to measure the smoothness of users switching between different modes; Transition Fluency S t The calculation formula is: or Among them, T avg T represents the average time it takes for users to complete the test task of switching between multiple modes. base is the set benchmark time; E is the number of incorrect operations caused by the user due to poor mode switching in the test task of switching between modes, E max The maximum number of allowed errors is set; is the error rate; α is C s The weight coefficient of S t The weight coefficient of .

4. The method for evaluating the complexity of a multimodal information interface according to claim 1, wherein: The static information density D static The calculation formula is: Among them, N y represents the number of static elements of type y, w y is the weight coefficient of the y-th type of static elements, T is the total number of static elements; N text is the number of static text elements in the interface; N icon is the number of static icon elements in the interface; N graphic is the number of static graphic elements in the interface; w text is the weight coefficient of the static text element; w icon is the weight coefficient of the static icon element; w graphic is the weight coefficient of the static graphic element; A interface is the total area of ​​the interface; The dynamic information complexity D dynamic The calculation formula is: Among them, Vr f is the refresh rate of the f-th type of dynamic elements, Fr f is the frame rate or effective visual update frequency of the f-th type of dynamic element, Area f is the display area of ​​the f-th type of dynamic element in the interface, A total is the total area of ​​the interface; V max ×F max It is a reference value of the product of the preset maximum refresh rate and the maximum frame rate; the types of dynamic elements include dynamic video, animation and real-time monitoring screen; The visual consistency V c The calculation formula is: V c =γ·M d +(1-γ)·L c ,(0≤γ≤1); Among them, M d is the layout matching degree, which is calculated as follows: The average offset distance refers to the average visual spatial misalignment distance between all dynamic elements and their associated static elements in the interface; the maximum allowable offset distance is a preset, tolerable upper limit of visual offset; L c Score logical consistency, calculated as: The user task completion rate ranges from 0 to 1. Specifically, a representative sample of users is recruited and asked to complete a set of standardized test tasks. The effective completion of these tasks depends on the reasonable information logic connection between the dynamic and static elements of the interface. The user task completion rate is recorded, which is equal to the number of times the user successfully completes the task divided by the total number of tasks. The expert score ranges from 0 to 1. Specifically, experts in the field of human-computer interaction or interface design independently evaluate the logical rationality of the dynamic and static elements in the interface in terms of information expression, causal relationships, and process coordination. The experts use a continuous scale of 0 to 1 to score, and the average of all expert scores is calculated as the final expert score. γ is the weight coefficient.

5. The method for evaluating the complexity of a multimodal information interface according to claim 1, wherein: The semantic clarity S atotal The calculation formula is: Where B is the total number of voice commands in the interface, b is the sequence number of the voice command; S a,b is the single average semantic clarity of the bth voice command in the interface; S a,b The calculation formula is: Where H is the number of times the voice recognition system in the interface recognizes the bth voice command, that is, the number of times the voice recognition system in the interface repeatedly recognizes the bth voice command; h represents the number of times the voice recognition system in the interface recognizes the bth voice command, h = 1, 2, ..., H; T ref,b is the reference text of the bth voice instruction, T rec,b,h The text obtained by the speech recognition system in the interface after the bth voice command is recognized for the hth time; Similarity(T ref,b ,T rec,b,h ) is the semantic similarity calculation function, the specific method is: the reference text T ref,b and recognized text T rec,b,h The BERT fine-tuning model is input respectively, and the output vector corresponding to the CLS tag is extracted as the sentence vector to obtain the corresponding sentence vector representation. Then the cosine similarity of the two sentence vectors is calculated, and the cosine similarity is normalized to obtain Similarity(T ref,b ,T rec,b,h ) value; The background noise interference index N interference The calculation formula is: where dB norm =min(dB,100), dB is the decibel value of background noise, P n is the ratio of background noise signal to the total signal; The cross-modal synchronization S av The calculation formula is: Where Δt is the average time delay between auditory and visual signals, calculated by timestamp alignment; T max The maximum allowed delay threshold is set to 50ms.

6. The method for evaluating the complexity of a multimodal information interface according to claim 1, wherein: The rationality of the layout order L o The calculation formula is: Where C is the total number of attention areas divided in the interface; p c is the reference text of the cth attention region, which is calculated as: Q c is the number of fixations of the c-th attention region, Q 总 is the total number of fixations; The information level depth H d The calculation formula is: H d =L·log2(B+1); L is the total number of levels in the user operation path; B is the branching factor, that is, the average number of branches at each level, calculated as: Among them, b l Indicates the number of branches at each level in the user operation path; The interactive control logic I r The calculation formula is: Among them, R s The redundancy of user operation steps is calculated as: R s = Actual number of steps - theoretical minimum number of steps; R max is the preset maximum redundancy threshold, R max =5.

7. The method for evaluating the complexity of a multimodal information interface according to claim 1, wherein: The step 3 is specifically as follows: The step 3 is specifically as follows: Normalize the values ​​of each indicator in step 2 separately, and the calculation formula is: Among them, X j is the value of the jth indicator calculated according to the corresponding formula in step 2, X min is the minimum value of the jth index, X max is the maximum value of the jth index, x j is the normalized value of the j-th indicator.

8. The method for evaluating the complexity of a multimodal information interface according to claim 1, wherein: The step 4 is specifically as follows: Use the entropy weight method to assign weights to each indicator or use the CRITIC method to assign weights to each indicator; The entropy weight method is used to assign weights to each indicator as follows: Calculate the probability: where p ij is the probability distribution of the jth indicator of the i-th interface sample, x ij is the normalized value of the jth index of the i-th interface sample, and n is the total number of interface samples; Calculate information entropy: e j is the information entropy of the jth indicator, ln(n) is the normalization factor; Calculate weights: Among them, w j is the weight of the jth indicator; e g is the information entropy of the g-th indicator, G is the total number of indicators; The CRITIC method is used to assign weights to each indicator as follows: Calculate the standard deviation: where σ j is the standard deviation of the jth indicator, x ij is the normalized value of the jth indicator of the i-th interface sample; It represents the mean value of the jth indicator on all n interface samples; the calculation formula is: n is the total number of interface samples; Calculate the correlation coefficient: Where i represents the interface sample index, i = 1, 2, ..., n; x ip represents the normalized value of the pth indicator of the i-th interface sample, is the mean of the pth indicator on all n interface samples; r pj The value range of is [-1,1]. The larger its absolute value is, the stronger the linear correlation between the pth indicator and the jth indicator is, and the lower the conflict is. pj represents the Pearson correlation coefficient between the pth indicator and the jth indicator; Calculate the comprehensive information volume: Among them C j is the comprehensive information volume; G is the total number of indicators; Calculate weights: Among them, w j is the weight of the jth indicator; C g is the comprehensive information amount of the g-th indicator.

9. The method for evaluating the complexity of a multimodal information interface according to claim 1, wherein: The step 5 is specifically as follows: Calculate the comprehensive score R: where w j is the weight of the jth indicator; x j is the normalized value of the j-th indicator.

10. A multimodal information interface complexity assessment system, characterized by: include: Multimodal interface element acquisition module: used to acquire multimodal interface elements, including visual data, auditory data and interaction data; Interface complexity analysis module: used to divide interface complexity into information complexity, visual complexity, auditory complexity and cognitive complexity; It is used to calculate the indicators of information complexity, visual complexity, auditory complexity and cognitive complexity respectively; the indicators of information complexity include the number of sensory modalities, the type of symbolic modality and the degree of cross-modal fusion; the indicators of visual complexity include static information density, dynamic information complexity and visual consistency; the indicators of auditory complexity include semantic clarity, background noise interference index and cross-modal synchronization; the indicators of cognitive complexity include layout order rationality, information hierarchy depth and interactive control logic; Normalization processing module: used to normalize all indicators separately; Indicator weight allocation module: used to assign weights to each indicator; Comprehensive score analysis module: used to calculate the comprehensive score through the normalized indicator values ​​and the weight corresponding to each indicator.

Images