Multi-modal data fusion and interactive visualization method

Abstract

The invention relates to the technical field of data fusion, and particularly discloses a multi-modal data fusion and interactive visualization method, which comprises the following steps of: capturing a fixation point coordinate sequence of a user corresponding to a current fixation page at a sampling rate of 200Hz through eye movement tracking equipment, and synchronously acquiring skin conductivity change rate and heart rate variability data of the user; on the basis of operation handle holding of the embedded pressure sensor array, operation force space-time distribution data of the user are obtained; evaluating the information entropy absorption rate of the user so as to output a dynamic adjustment coefficient of the user; and finally, the display mode of the currently watched page is autonomously adjusted, so that comprehensive monitoring and analysis of the cognitive state of the user can be realized. The navigation capability and the information processing efficiency of the user in a complex information environment can be improved, and the satisfaction degree and the use experience of the user can be enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of data fusion, and relates to multi-modal data fusion and interactive visualization methods. Background Art

[0002] Multi-modal data fusion and interactive visualization monitoring are of great significance in the fields of modern user experience design and human-computer interaction. With the continuous development of technology, the behavioral and physiological data generated by users when using systems become increasingly complex, and a single data source cannot comprehensively reflect the true experience and cognitive state of users. By fusing data from different sensors, such as eye tracking, skin conductance, heart rate variability, and operation behavior, etc., a more comprehensive user portrait can be provided, revealing the attention distribution, emotional response, and cognitive load of users in specific situations. The integration of such multi-modal data enables designers to understand user needs more deeply, thereby optimizing the interface design and information presentation methods.

[0003] Existing user interface designs often rely on static user feedback and a single data source, such as relying on users' self-reports or simple click data to evaluate the user experience. This method has obvious limitations because users' self-reports may be affected by subjective factors and cannot reflect the true cognitive state of users in real time. Through real-time eye tracking and physiological signal monitoring, objective and accurate data can be provided, revealing the true reactions of users during the interaction process. The combination of such multi-modal monitoring can more comprehensively capture the attention distribution, emotional fluctuations, and cognitive load of users, thereby providing a more scientific basis for interface design.

[0004] Traditional methods often only focus on the click-through rate or operation success rate of users, while ignoring the subtle changes during the operation process. The change in the holding force not only reflects the tension and concentration of users, but also reveals the perception and reaction of users to interface elements. By combining the analysis of the holding force and eye movement data, the system can more accurately identify the cognitive state of users and provide more detailed guidance for interface adjustment. Summary of the Invention

[0005] In view of the above problems existing in the prior art, the present invention provides a multi-modal data fusion and interactive visualization method to solve the above technical problems.

[0006] In order to achieve the above object and other objects, the technical solution adopted by the present invention is as follows: On the one hand, the present invention provides a multi-modal data fusion and interactive visualization method, and the method includes the following steps: Step 1. Multimodal data collection: Use an eye-tracking device to capture the sequence of fixation point coordinates of the user corresponding to the current fixation page at a sampling rate of 200 Hz. Synchronously collect the skin conductance change rate and heart rate variability data of the user. Then, based on the operation handle grip of the embedded pressure sensor array, obtain the spatio-temporal distribution data of the user's operation force. Step 2. Cognitive state quantitative evaluation: Evaluate the information entropy absorption rate of the user, and thus output the dynamic adjustment coefficient of the user. Step 3. Visualization strategy generation: Based on the dynamic adjustment coefficient of the user, autonomously adjust the display mode of the current fixation page.

[0007] Use an eye-tracking device to capture the sequence of fixation point coordinates of the user corresponding to the current fixation page at a sampling rate of 200 Hz. The data expression form is: , where n is the number of each fixation time point, and the value range of n is from 1 to N. are respectively the timestamp, horizontal coordinate, and vertical coordinate of the user corresponding to the current fixation page at the nth fixation time point. The user's heart rate variability data includes heart rate variability low-frequency power and heart rate variability high-frequency power. The spatio-temporal distribution data of the user's operation force includes the grip force at each fixation time point.

[0008] Evaluating the information entropy absorption rate of the user includes: Obtain the total number M of interface elements of the user corresponding to the current fixation page, and obtain the set of position area coordinates of each page element in the current fixation page. Compare the horizontal coordinate and vertical coordinate of the user corresponding to the current fixation page at the nth fixation time point with the set of position area coordinates of each page element in the current fixation page to obtain the fixation page element number of the user at the nth fixation time point. Finally, integrate the fixation duration of the user corresponding to the ith page element. , where i is the number of the page element, i = 1, 2,... M; Thus, through the analysis formula , evaluate the information entropy absorption rate H of the user. In the above calculation formula, N is the total fixation duration. is the information complexity of the ith page element in the current fixation page of the user. is the display area of the ith page element in the current fixation page of the user. is the attention weight factor of the ith page element in the current fixation page of the user; e is the natural constant, k is the set adjustment coefficient, P’ represents the set pressure threshold. is the cognitive pressure index of the ith page element in the current fixation page of the user.

[0009] The information complexity corresponding to the i-th page element on the currently gazed page by the user is calculated as follows: ; In the above formula is the text complexity corresponding to the i-th page element on the currently gazed page by the user, and its specific calculation formula is , is the total number of characters within the i-th page element on the currently gazed page by the user, ; is the digital density of the i-th page element on the currently gazed page by the user, , Zd is the total number of fields on the currently gazed page by the user, is the number of digital fields within the i-th page element on the currently gazed page by the user, is the number of digits within the i-th page element on the currently gazed page by the user; is the number of technical terms within the i-th page element on the currently gazed page by the user; In the above formula is the visual complexity of the i-th page element on the currently gazed page by the user, and its specific calculation formula is , are respectively the total color difference degree, the graphic type weight, and the number of dynamic objects of the i-th page element on the currently gazed page by the user; In the above formula is the interaction complexity of the currently gazed page by the user, and its specific calculation formula is , Bz is the number of steps required to complete the operation on the currently gazed page by the user, and Jd is the operation accuracy requirement of the currently gazed page by the user; In the above formula is the structural complexity of the i-th page element on the currently gazed page by the user, and its specific calculation formula is , is the nesting layer index of the i-th page element on the currently gazed page by the user, is the proportion of the blank area of the i-th page element on the currently gazed page by the user.

[0010] The cognitive stress index corresponding to the i-th page element on the currently gazed page by the user is calculated as follows: ; In the above formula, β1, β2, and β3 respectively represent the predefined stress sensitivity coefficient, neuromodulation coefficient, and behavior weight coefficient, △GSR is the change in skin conductance of the currently gazed page by the user, and △t is the set change time interval, is the ratio of the low-frequency power of the heart rate variability to the high-frequency power of the heart rate variability corresponding to the page currently being gazed at by the user, is the maximum value of the gripping force corresponding to the i-th page element in the page currently being gazed at by the user, is the calibration reference value of the embedded pressure sensor array.

[0011] Output the dynamic adjustment coefficient of the user. The specific output formula is: .

[0012] Autonomously adjust the display mode of the currently gazed-at page, including: The display mode is divided into an expert mode, a guided mode, and a quick view mode; When the information entropy absorption rate H of the user ≥ 0.65 and the cognitive stress index of any page element corresponding to the page currently being gazed at by the user ≥ 0.4, activate the expert mode; dynamically associate the original data, derived metrics, and user operations through a three-dimensional topological map; When 0.4 < H < 0.65 of the user's information entropy absorption rate and the cognitive stress index of any page element corresponding to the page currently being gazed at by the user is greater than 0.4 and less than 0.7, activate the guided mode; generate a step-by-step decision wizard animation, use a particle system to simulate the operation influence path, and embed an interactive sandbox at key decision points to allow trial-and-error operations within the range of ±15% of the parameters; When the information entropy absorption rate H of the user ≤ 0.4 and the cognitive stress index of any page element corresponding to the page currently being gazed at by the user ≥ 0.7, activate the quick view mode; mark the key metric deviation with three colors: red, yellow, and green; dynamically fold secondary information modules and focus on displaying the core data dashboard.

[0013] On the other hand, the present invention provides a multi-modal data fusion and interactive visualization device, including a processor, a memory, and a communication bus; The memory stores a computer-readable program executable by the processor; The communication bus realizes the connection and communication between the processor and the memory; When the processor executes the computer-readable program, it executes to implement the multi-modal data fusion and interactive visualization method as described in the present invention.

[0014] As described above, the multi-modal data fusion and interactive visualization method provided by the present invention has at least the following beneficial effects: The multimodal data fusion and interactive visualization method provided by the present invention captures the sequence of the user's fixation point coordinates at a high sampling rate of 200 Hz through an eye-tracking device, synchronously collects the user's skin conductance change rate and heart rate variability data, and combines the spatio-temporal distribution data of the holding force of the operation handle obtained by an embedded pressure sensor array. The combination of these technical means provides strong support for the dynamic optimization of the user experience. First of all, the eye-tracking technology captures the user's fixation behavior at a high frequency, and can accurately analyze the specific areas and elements that the user focuses on the interface, revealing the attention distribution of the user in the information processing process. This refined data collection enables designers to deeply understand the user's cognitive process, thereby optimizing the interface layout and information presentation, and improving the readability and usability of the information; Secondly, the synchronous collection of the skin conductance change rate and heart rate variability data provides an important reference for the user's emotional and physiological states. Skin conductance reflects the user's physiological stress response, while heart rate variability is an indicator of the activity of the autonomic nervous system. By real-time monitoring these physiological signals, the system can timely identify the user's cognitive stress and emotional fluctuations, thereby providing a basis for adjusting the information display mode. This dynamic feedback mechanism can not only help users obtain a more concise information presentation under high-stress states, reducing their cognitive burden, but also improve the user's overall satisfaction and usage efficiency through timely adjustments; Furthermore, the spatio-temporal distribution data of the holding force of the operation handle provides an additional dimensional analysis of the user's behavior performance. The change in the holding force is usually closely related to the user's concentration and emotional state. By analyzing the relationship between the holding force and the fixation behavior, the system can more comprehensively understand the user's performance in a specific task, and then optimize the interaction design to make it more in line with the actual needs of the user. This multi-dimensional data fusion provides a solid foundation for the personalization and intelligence of the user experience; Finally, based on the above data, evaluating the user's information entropy absorption rate and outputting the user's dynamic adjustment coefficient can realize the autonomous adjustment of the information display mode. This automated process not only improves the intelligence level of the system, but also ensures that the information presentation method can adapt to the user's cognitive state and demand changes at any time. By dynamically adjusting the information density, layout and display mode, the system can effectively reduce the user's cognitive load and improve the efficiency and accuracy of information absorption.

[0015] In summary, by combining various physiological and behavioral data such as eye-tracking, skin conductance, heart rate variability and operation force, it is possible to comprehensively monitor and analyze the user's cognitive state. It can not only improve the user's navigation ability and information processing efficiency in a complex information environment, but also enhance the user's satisfaction and usage experience. Description of the Drawings

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0017] Figure 1 Schematic diagram of the connection of each step of the method of the present invention. Specific embodiments

[0018] The following will combine the above content of the embodiments of the present invention. The above content is only an example and explanation of the concept of the present invention. Those skilled in the art of this technology can make various modifications or supplements to the described specific embodiments or use similar methods to replace them, as long as they do not deviate from the concept of the invention or exceed the scope defined by the claims of the present invention, they should all belong to the protection scope of the present invention. Embodiment

[0019] Please refer to Figure 1 As shown, a multi-modal data fusion and interactive visualization method, the method includes the following steps: Step 1. Multi-modal data acquisition: Capture the fixation point coordinate sequence of the user corresponding to the current fixation page through an eye tracking device at a sampling rate of 200Hz, synchronously collect the skin conductance change rate and heart rate variability data of the user, and then obtain the spatio-temporal distribution data of the user's operation force based on the operation handle grip of the embedded pressure sensor array; Capture the fixation point coordinate sequence of the user corresponding to the current fixation page through an eye tracking device at a sampling rate of 200Hz, and the data expression form is: , where n is the number of each fixation time point, and the value range of n is from 1 to N, are respectively the time stamp, horizontal coordinate, and vertical coordinate of the user corresponding to the current fixation page at the nth fixation time point; The heart rate variability data of the user includes heart rate variability low-frequency power and heart rate variability high-frequency power; The spatio-temporal distribution data of the user's operation force includes the grip force at each fixation time point.

[0020] Step 2. Cognitive state quantitative evaluation: Evaluate the information entropy absorption rate of the user, so as to output the dynamic adjustment coefficient of the user; Evaluating the information entropy absorption rate of the user includes: Obtain the total number M of interface elements on the current fixation page of the user, and obtain the set of position area coordinates of each page element on the current fixation page. Compare the horizontal and vertical coordinates of the user's current fixation page at the nth fixation time point with the set of position area coordinates of each page element on the current fixation page to obtain the fixation page element number of the user at the nth fixation time point. Finally, integrate the fixation duration of the user corresponding to the i-th page element. , where i is the number of the page element, i = 1, 2,... M; the page elements include but are not limited to text boards, trend charts, and warning lights. Thus, through the analysis formula , evaluate the information entropy absorption rate H of the user. In the above calculation formula, N is the total fixation duration. is the information complexity of the i-th page element on the current fixation page of the user. is the display area of the i-th page element on the current fixation page of the user. is the attention weight factor of the i-th page element on the current fixation page of the user; e is the natural constant, k is the set adjustment coefficient, P’ represents the set pressure threshold. is the cognitive stress index of the i-th page element on the current fixation page of the user.

[0021] The information complexity of the i-th page element on the current fixation page of the user is specifically calculated as follows: ; In the above formula is the text complexity of the i-th page element on the current fixation page of the user, and its specific calculation formula is , is the total number of characters in the i-th page element on the current fixation page of the user. ; is the digital density of the i-th page element on the current fixation page of the user. , Zd is the total number of fields on the current fixation page of the user. is the number of digital fields in the i-th page element on the current fixation page of the user. is the number of digits in the i-th page element on the current fixation page of the user. The introduction of is to consider the impact of the number of digits in the text on the overall complexity. The presence of digits often increases the cognitive burden of the text. Therefore, including its quantity in the calculation helps to more accurately evaluate the complexity of the text. By taking the square root of the number of digits, its impact on complexity can be reduced, making the calculation of complexity more balanced and reasonable.

[0022] is the number of technical terms within the i-th page element on the currently gazed page for the user; In the above formula is the visual complexity of the i-th page element on the currently gazed page for the user, and its specific calculation formula is , are respectively the total color difference degree, the graphic type weight, and the number of dynamic objects of the i-th page element on the currently gazed page for the user; where the graphic types include but are not limited to scatter plots, three-dimensional surface plots, heat maps, line charts, and bar charts; among them, the image type weight of the scatter plot is 1.5, the image type weight of the three-dimensional surface plot is 1.3, the image type weight of the heat map is 1.2, the image type weight of the line chart is 0.8, and the image type weight of the bar chart is 0.5; In the above formula is the interaction complexity of the currently gazed page for the user, and its specific calculation formula is , Bz is the number of steps required for the user to complete the operation on the currently gazed page, and Jd is the operation accuracy requirement for the currently gazed page by the user; In the above formula is the structural complexity of the i-th page element on the currently gazed page for the user, and its specific calculation formula is , is the nesting level exponent of the i-th page element on the currently gazed page for the user, is the proportion of the blank area of the i-th page element on the currently gazed page for the user.

[0023] The nesting level exponent (is an index used to measure the complexity of the nested structure of elements in an interface or document. Simply put, it reflects the hierarchical relationship and organizational structure of the elements in the page; In a web page or application, elements (such as buttons, text boxes, images, etc.) can be contained within other elements. For example, a button may be within a frame (container), and this frame may in turn be within a larger area. Such a structure is called "nesting"; The level refers to the depth of the position of the element in the nested structure. The deeper an element is in the structure, it means it is contained within more other elements; Example: If a button is directly placed on the outermost layer of the page, then its nesting level is very shallow, with a low value; if this button is placed within a frame, and this frame is in turn placed within another frame, nested layer by layer, then its nesting level is very deep, with a high value.

[0024] Information complexity is an indicator that comprehensively evaluates the complexity of elements, consisting of text complexity, visual complexity, interaction complexity, and structural complexity, with the weights of each part being 0.25, 0.45, 0.20, and 0.10 respectively. The practical significance of this comprehensive calculation lies in comprehensively considering the complexity of elements in terms of text, vision, interaction, and structure, providing a more comprehensive and accurate measurement method; The text complexity part considers the complexity of text elements, including the number of words / characters, digital density, the number of technical terms, and the reading difficulty coefficient. The visual complexity part focuses on the complexity of visual elements, including color difference degree, graphic type weight, and the number of dynamic objects. The interaction complexity part considers the interaction complexity between users and elements, including the number of steps required to complete an operation and the operation accuracy requirement. The structural complexity part involves the layout complexity of elements on the page, including the nesting level and the proportion of the blank area.

[0025] The cognitive stress index of the user corresponding to the i-th page element on the currently gazed page is calculated as follows: ; In the above formula, β1, β2, and β3 respectively represent the predefined stress sensitivity coefficient, neuromodulation coefficient, and behavior weight coefficient, △GSR is the change in skin conductance of the user corresponding to the currently gazed page, △t is the set change time interval, is the ratio of the low-frequency power of heart rate variability to the high-frequency power of heart rate variability of the user corresponding to the currently gazed page, is the maximum value of the grip strength of the user corresponding to the i-th page element on the currently gazed page, is the calibration reference value of the embedded pressure sensor array.

[0026] The calculation formula of the cognitive stress index provides an effective method for quantifying the user's cognitive stress by integrating multiple physiological and behavioral indicators. The structure of this formula consists of three main components, namely the physiological stress term, the autonomic nerve term, and the behavior representation term. Each term represents different physiological signals or behavioral manifestations, and can reflect the user's cognitive load and stress level in a specific situation; First of all, the physiological stress term evaluates the user's physiological response by measuring the skin conductance change rate. Skin conductance is a physiological indicator closely related to sympathetic nerve activity, and usually increases significantly under stress or tension. By calculating the change in skin conductance per unit time, the physiological stress response of the user when facing a specific task can be effectively captured, and thus the change in their cognitive load can be reflected; Secondly, the autonomic nerve item uses the ratio of low frequency to high frequency in heart rate variability (HRV) to evaluate the balance state of the autonomic nervous system. HRV is an important indicator reflecting cardiac health and autonomic nerve function. The change in the ratio of low frequency to high frequency can indicate an individual's physiological adaptability when facing stress. When the autonomic nervous system is under stress, the low-frequency component of HRV relatively increases while the high-frequency component decreases, thus affecting the ratio. Through the natural logarithm transformation of this ratio, the formula can better handle the non-linear relationship and provide a more accurate stress assessment.

[0027] The behavioral characterization item focuses on the operating force of the user in a specific task and uses the ratio of the maximum grip force to the calibration value to reflect the user's behavior performance. The change in grip force is usually related to the user's tension and concentration levels. A higher grip force may indicate that the user is trying to maintain control in a high-pressure situation. The introduction of this item not only considers the user's physiological state but also provides a more comprehensive assessment of cognitive load through behavior performance; In summary, the calculation formula of the cognitive stress index provides a comprehensive and effective way to quantify the user's cognitive stress by integrating multi-dimensional indicators such as physiology and behavior. This comprehensive assessment can not only help researchers and developers better understand the user's reactions in specific situations but also provide data support for designing more user-friendly interaction systems. For example, when the cognitive stress index exceeds a certain threshold, the system can automatically adjust the way of information presentation, such as reducing the information density or switching to a simpler interface, so as to reduce the user's cognitive burden and improve the user experience.

[0028] Output the dynamic adjustment coefficient of the user. The specific output formula is: 。

[0029] The above formula dynamically adjusts the output adjustment coefficient η according to different values of the information entropy absorption rate H. In this formula, different adjustment coefficient values are determined according to different information absorption rate levels to achieve dynamic adjustment of information density: When the information entropy absorption rate H is less than or equal to 0.5, the adjustment coefficient η will increase accordingly. The initial value is 0.5, and then it linearly increases by 0.7 times according to the information entropy absorption rate to increase the information density. When the information entropy absorption rate is greater than 0.8, the adjustment coefficient η is fixed at 1.2 to maintain a high information density. For other cases, the adjustment coefficient η will linearly increase with the increase of the information entropy absorption rate to appropriately adjust the information density; Such a calculation formula can better meet the user's cognitive needs by dynamically adjusting the information density according to the user's information absorption ability. When the user's information absorption ability is low, reducing the information density can reduce the cognitive load and improve the efficiency of information understanding and absorption; when the user's information absorption ability is high, increasing the information density can provide more relevant information and promote deeper thinking and understanding; In addition, such a dynamic adjustment mechanism can enhance the user experience and system performance. By adjusting the information density according to the user's current information absorption ability, it enables the user to process information more easily, reduces the possibility of information overload and chaos, thereby improving the user's work efficiency and satisfaction. At the same time, this dynamic adjustment can be adjusted in real time according to the user's state, making the system more intelligent and personalized, enhancing the user experience and system adaptability; In summary, the practical significance of this dynamic adjustment coefficient calculation formula lies in dynamically adjusting the information density according to the user's information absorption ability to enhance the user experience, work efficiency, and system performance. This mechanism helps to balance the presentation of information and the user's cognitive load, providing a more personalized and intelligent information display method for the user, thereby achieving a better user experience and system performance.

[0030] Step 3: Visualization strategy generation: Based on the user's dynamic adjustment coefficient, autonomously adjust the display mode of the currently gazed page.

[0031] Autonomously adjust the display mode of the currently gazed page, including: The display modes are divided into expert mode, guided mode, and overview mode; When the user's information entropy absorption rate H ≥ 0.65 and the cognitive stress index of any page element corresponding to the user on the currently gazed page ≤ 0.4, activate the expert mode; dynamically associate the original data (blue nodes), derived indicators (orange nodes), and user operations (red connections) through a three-dimensional topological graph; When the user's information entropy absorption rate 0.4 < H < 0.65 and the cognitive stress index of any page element corresponding to the user on the currently gazed page is greater than 0.4 and less than 0.7, activate the guided mode; generate a step-by-step decision wizard animation, use a particle system to simulate the operation influence path, and embed an interactive sandbox at key decision points to allow trial-and-error operations within the range of +15% of the parameters; When the user's information entropy absorption rate H ≤ 0.4 and the cognitive stress index of any page element corresponding to the user on the currently gazed page ≥ 0.7, activate the overview mode; mark the deviation degree of key indicators with three colors: red (dangerous), yellow (warning), and green (normal); dynamically fold secondary information modules and focus on displaying the core data dashboard. Embodiment

[0032] A multimodal data fusion and interactive visualization device includes a processor, a memory, and a communication bus; A computer-readable program executable by the processor is stored on the memory; The communication bus realizes the connection and communication between the processor and the memory; When the processor executes the computer-readable program, it executes to implement the multimodal data fusion and interactive visualization method as described in the present invention.

[0033] It should be understood that in various embodiments of the present application, the magnitudes of the serial numbers of the above processes do not mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0034] It should be understood that determining B based on A does not mean determining B only based on A, but also B can be determined based on A and / or other information.

[0035] The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, and all should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0036] Finally: The above is only the preferred embodiment of the present invention and is not used to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multimodal data fusion and interactive visualization method, characterized in that: Including: Step 1, Multimodal data acquisition: Use an eye-tracking device to capture the sequence of fixation point coordinates of the user corresponding to the current gazed page at a sampling rate of 200Hz, synchronously collect the skin conductance change rate and heart rate variability data of the user, and then obtain the spatio-temporal distribution data of the user's operation force based on the grip of the operation handle with an embedded pressure sensor array; Step 2, Cognitive state quantitative assessment: Evaluate the information entropy absorption rate of the user, and thus output the dynamic adjustment coefficient of the user; Step 3, Visualization strategy generation: Based on the dynamic adjustment coefficient of the user, autonomously adjust the display mode of the current gazed page.

2. The multimodal data fusion and interactive visualization method according to claim 1, characterized in that: Use an eye-tracking device to capture the sequence of fixation point coordinates of the user corresponding to the current gazed page at a sampling rate of 200Hz, and the data expression form is: , n is the number of each fixation time point, and the value range of n is 1 to N. are respectively the timestamp, horizontal coordinate and vertical coordinate of the user at the nth gaze time point corresponding to the current gaze page; The heart rate variability data of the user includes heart rate variability low-frequency power and heart rate variability high-frequency power; The spatio-temporal distribution data of the user's operation force includes the grip force at each fixation time point.

3. The multimodal data fusion and interactive visualization method according to claim 2, characterized in that: Evaluating the information entropy absorption rate of the user includes: Get the total number M of interface elements corresponding to the current gaze page of the user, and get the location area coordinate collection of each page element in the current gaze page. Compare the horizontal coordinate and vertical coordinate of the current gaze page of the user at the nth gaze time point with the location area coordinate collection of each page element in the current gaze page, and get the number of the page element that the user is looking at at the nth gaze time point. Finally, integrate the gaze time of the user corresponding to the i-th page element. , i is the number of the page element, i=1,2,...M; Therefore, by analyzing the formula , evaluate the user's information entropy absorption rate H; In the above calculation formula, N is the total fixation time. is the information complexity of the i-th page element in the page that the user is currently looking at, is the display area of ​​the i-th page element in the page currently being looked at by the user, is the attention weight factor of the user corresponding to the i-th page element in the current page being looked at; e is a natural constant, k is a set adjustment coefficient, and P' represents the set pressure threshold. is the cognitive stress index of the user corresponding to the i-th page element in the current gaze page.

4. The multimodal data fusion and interactive visualization method according to claim 3, characterized in that: The information complexity of the i-th page element in the current gazed page by the user, and the specific calculation formula is as follows: ； In the above formula is the text complexity of the i-th page element in the current page that the user is looking at, and its specific calculation formula is: , is the total number of characters in the i-th page element of the user's current gaze page, ; is the digital density of the i-th page element in the page currently being looked at by the user, , Zd is the total number of fields in the page that the user is currently looking at, is the number of digital fields in the i-th page element of the user's current gaze page, is the number of digits in the i-th page element of the page that the user is currently looking at; is the number of professional terms in the i-th page element of the user's current gaze page; In the above formula is the visual complexity of the i-th page element in the current page that the user is looking at, and its specific calculation formula is: , are the sum of the color difference, graphic type weight, and number of dynamic objects of the i-th page element in the page currently being looked at by the user; In the above formula is the interaction complexity of the user corresponding to the current gaze page, and its specific calculation formula is: , Bz is the number of steps required for the user to complete the operation corresponding to the current gaze page, and Jd is the operation accuracy requirement of the user corresponding to the current gaze page; In the above formula is the structural complexity of the i-th page element in the page that the user is currently looking at, and its specific calculation formula is: , is the nesting index of the i-th page element in the page currently being looked at by the user, The percentage of the blank area of ​​the i-th page element in the page that the user is currently looking at.

5. The multimodal data fusion and interactive visualization method according to claim 3, characterized in that: The cognitive stress index of the i-th page element in the current gazed page by the user, and the specific calculation formula is as follows: ； In the above formula, β1, β2, and β3 represent the predefined stress sensitivity coefficient, neural regulation coefficient, and behavior weight coefficient, respectively; △GSR is the change in skin conductance of the user corresponding to the current gaze page; △t is the set change time interval. is the ratio of the user's heart rate variability low-frequency power / heart rate variability high-frequency power corresponding to the page currently being looked at, is the maximum value of the user's grip strength corresponding to the i-th page element in the current gaze page, is the calibration reference value of the embedded pressure sensor array.

6. The multimodal data fusion and interactive visualization method according to claim 1, characterized in that: Output the dynamic adjustment coefficient of the user, and the specific output formula is: 。 7. The multimodal data fusion and interactive visualization method according to claim 1, characterized in that: Autonomously adjust the display mode of the current gazed page, including: The display mode is divided into an expert mode, a guidance mode, and a quick view mode; When the user's information entropy absorption rate H 0.65 and the user has a cognitive stress index of any page element on the current page being looked at At 0.4, the expert mode is activated; the original data, derived indicators and user operations are dynamically associated through a three-dimensional topological map; When the information entropy absorption rate of the user is 0.4 < H < 0.65 and the cognitive stress index of any one page element in the current gazed page by the user is greater than 0.4 and less than 0.7, activate the guidance mode; Generate a step-by-step decision wizard animation, use a particle system to simulate the operation influence path, and embed an interactive sandbox at the key decision points to allow trial-and-error operations within the range of +15% of the parameters; When the user's information entropy absorption rate H 0.4 and the user has a cognitive stress index of any page element on the current page being looked at At 0.7, the quick view mode is activated; the deviation of key indicators is marked with red, yellow and green colors; the secondary information module is dynamically folded to focus on displaying the core data dashboard.

8. A multimodal data fusion and interactive visualization device, characterized in that: It is implemented based on the multimodal data fusion and interactive visualization method described in any one of claims 1-7, including a processor, a memory, and a communication bus; The memory stores a computer-readable program executable by the processor; The communication bus realizes the connection communication between the processor and the memory; When the processor executes the computer-readable program, it executes to implement the multimodal data fusion and interactive visualization method described in any one of claims 1-7.

Images