Multimodal processing methods, devices, and storage media based on electroencephalogram (EEG) signals

Abstract

This application discloses a multimodal processing method, device, and storage medium based on electroencephalogram (EEG) signals. This application relates to the field of signal data processing technology. The multimodal processing method based on EEG signals includes: acquiring EEG signals and eye-tracking signals for test materials; aligning the EEG signals and eye-tracking signals with the temporal sequence of the test materials to obtain target EEG signals and target eye-tracking signals after updating the timestamps; analyzing the emotional indicators, cognitive indicators, and motivational arousal indicators corresponding to the target EEG signals; analyzing the gaze heatmap and gaze trajectory map corresponding to the target eye-tracking signals; and fusing the emotional indicators, cognitive indicators, motivational arousal indicators, gaze heatmap, and gaze trajectory map to obtain suggested material layout information for the test materials. This application can achieve the technical effect of enhancing the appeal of materials to the target audience.

Description

Technical Field

[0001] This application relates to the field of signal processing technology, and in particular to a multimodal processing method, device and storage medium based on electroencephalogram (EEG) signals. Background Technology

[0002] Currently, electroencephalogram (EEG) signals are the core carriers reflecting brain activity and are widely used in scenarios such as emotion recognition and cognitive assessment. However, EEG signals are inherently weak and easily affected by muscle movements, eye movements, and environmental electromagnetic interference. Related technologies use filters, resulting in low signal-to-noise ratios for signals collected by single-point or limited electrodes on the forehead, making it difficult to extract stable and reliable emotion and cognitive indicators. Summary of the Invention

[0003] The main objective of this application is to provide a multimodal processing method, device, and storage medium based on electroencephalogram (EEG) signals, aiming to solve the technical problem of extracting stable and reliable emotion and cognitive indicators.

[0004] To achieve the above objectives, this application provides a multimodal processing method based on electroencephalogram (EEG) signals, which includes:

[0005] Acquire EEG and eye movement signals for the test subjects;

[0006] The EEG signal and eye movement signal are aligned with the time sequence of the test material to obtain the target EEG signal and target eye movement signal after the timestamp is updated.

[0007] Analyze the emotional, cognitive, and motivational arousal indicators corresponding to the target EEG signals;

[0008] Analyze the gaze heatmap and gaze trajectory map corresponding to the target eye movement signal;

[0009] By integrating emotional indicators, cognitive indicators, motivational arousal indicators, gaze heatmaps, and gaze trajectory maps, suggested information for the arrangement of test materials is obtained.

[0010] In one embodiment, the step of aligning the electroencephalogram (EEG) signal and eye movement (EMG) signal with the time sequence of the test material to obtain the target EEG signal and target EEG signal after updating the timestamp includes:

[0011] Obtain the baseline timing information of the test materials. The baseline timing information includes the playback or display timeline of the test materials, the start and end times of each key screen and element;

[0012] Extract the original timestamp sequences corresponding to the EEG signals and the original timestamp sequences corresponding to the eye movement signals. The original timestamp sequences are the signal generation times recorded in real time by the signal acquisition device.

[0013] Using the start time of the test material playback or display in the baseline timing information as the zero point, the time deviation values ​​between the original timestamp of the EEG signal, the original timestamp of the eye movement signal and the zero point are calculated respectively.

[0014] Based on the time deviation value, the original timestamp sequences of EEG signals and eye movement signals are calibrated and corrected to obtain a calibrated timestamp sequence that accurately corresponds to the baseline time sequence of the test material;

[0015] The calibrated timestamp sequence is associated and bound with the corresponding EEG signal data and eye movement signal data to generate target EEG signal and target eye movement signal that simultaneously contain signal data, calibration timestamps and corresponding test material times.

[0016] In one embodiment, the steps of parsing the emotional indicators, cognitive indicators, and motivational arousal indicators corresponding to the target EEG signal include:

[0017] Extract feature parameters from the target EEG signal, including event-related potential components and EEG rhythm frequency band energy;

[0018] Based on the analysis of feature parameters, emotion indicators, cognitive indicators, and motivational arousal indicators are obtained.

[0019] In one embodiment, the step of parsing emotion indicators, cognitive indicators, and motivational arousal indicators based on feature parameters includes:

[0020] Based on a pre-defined emotion analysis model, the amplitude and time of LPP waves and the proportion of β wave energy in the event-related potential components are input to obtain emotion indicators. The emotion indicators include at least excitement, pleasure, disgust, and boredom.

[0021] Based on a pre-defined cognitive analysis model, the latency and amplitude of the P300 wave and the energy ratio of the theta wave to the alpha wave in the event-related potential components are input to obtain cognitive indicators. The cognitive indicators include at least the degree of attention concentration and memory encoding efficiency.

[0022] Based on a pre-defined motivational arousal analysis model, the amplitude of the N400 wave and the energy ratio of the β wave to the α wave in the event-related potential components are input to obtain a motivational arousal index that quantifies the desire to purchase.

[0023] In one embodiment, the step of parsing the gaze heatmap and gaze trajectory map corresponding to the target eye movement signal includes:

[0024] Key eye movement parameters are extracted from the target eye movement signal. These key eye movement parameters include fixation point coordinates, duration of a single fixation point, order of fixation point appearance, and saccade amplitude.

[0025] Obtain the screen coordinate system of the test material and establish the mapping relationship between the gaze point coordinates in the key eye-tracking parameters and the screen coordinates of the test material;

[0026] Based on the mapping relationship and the duration of a single gaze point, a gaze heat map is generated. The gaze heat map uses thermal intensity gradient to represent the degree of gaze concentration in different regions, and thermal intensity is positively correlated with gaze duration.

[0027] Based on the order of gaze points and the timestamps corresponding to the timing of the test materials, the coordinates of each gaze point are connected according to the time sequence to generate a gaze trajectory diagram. The gaze trajectory diagram is marked with the timing number of each gaze point and the corresponding gaze duration.

[0028] In one embodiment, the step of fusing emotion indicators, cognitive indicators, motivational arousal indicators, gaze heatmaps, and gaze trajectory maps to obtain suggested information on the arrangement of test materials includes:

[0029] The visual logic corresponding to the target eye movement signal and the first-eye attraction parameters of each element in the test material are determined based on the gaze heat map and gaze trajectory map.

[0030] By integrating emotional indicators, cognitive indicators, motivational arousal indicators, visual logic, and the first-glance attractiveness parameters of each element in the test materials, suggested information on the arrangement of each element in the test materials is obtained.

[0031] In one embodiment, the step of integrating emotional indicators, cognitive indicators, motivational arousal indicators, visual logic, and the first-glance attractiveness parameters of each element in the test material to obtain suggested material layout information for each element in the test material includes:

[0032] The emotional indicators, cognitive indicators, motivational arousal indicators, and first-glance attraction parameters are quantified and standardized to obtain standardized emotional values, standardized cognitive values, standardized motivational arousal values, and standardized attraction values ​​of a unified dimension.

[0033] Based on the weight allocation rules obtained from training a preset sample set, determine the weight coefficients corresponding to each standardized indicator.

[0034] Based on the weighting coefficients, the comprehensive score of each element in the test material is calculated by weighted summation;

[0035] By combining visual logic analysis of the rationality of the gaze path of each element, the optimization type of each element is determined based on the comprehensive score and the rationality of the gaze path. The optimization types include core optimization elements to be retained, elements to be adjusted, and intermediate priority elements.

[0036] For elements of different optimization types, separate layout optimization directions are generated to obtain suggested material layout information.

[0037] In one embodiment, the step of generating layout optimization directions for elements of different optimization types to obtain material layout suggestion information includes:

[0038] For the core optimized and retained elements, layout optimization directions are generated with the goal of enhancing attention capture and positive neural response. Layout optimization directions include maintaining the current display position, increasing the display area of ​​the preset proportion, and improving the visual contrast between the elements and the surrounding environment.

[0039] For elements to be adjusted and optimized, layout optimization directions are generated with the goal of improving attention acquisition ability and neural response adaptability. Layout optimization directions include adjusting to the core area of ​​the gaze path, optimizing the color and shape of elements to enhance first-hand attractiveness, and avoiding the visual coverage of core optimized elements.

[0040] For intermediate priority elements, layout optimization directions are generated with the goal of adapting to the visual logic flow. Layout optimization directions include fine-tuning the display position to fit the direction of the gaze path extension, adjusting the display sequence to connect the gaze nodes of the core optimized and retained elements, and weakening unnecessary visual decorations to reduce attentional interference to the core elements.

[0041] The layout optimization directions for elements of different optimization types are integrated to form material layout suggestions information that include element identifiers, optimization actions, and optimization parameters. The optimization parameters include area adjustment ratio, position coordinate offset, and visual contrast adjustment threshold.

[0042] In addition, to achieve the above objectives, this application also provides a multimodal processing device based on electroencephalogram (EEG) signals. The multimodal processing device based on EEG signals includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. The computer program is configured to implement the steps of the multimodal processing method based on EEG signals as described above.

[0043] In addition, to achieve the above objectives, this application also provides a storage medium, which is a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the steps of the multimodal processing method based on EEG signals as described above.

[0044] This application provides a multimodal processing method based on electroencephalogram (EEG) signals. The method acquires EEG and eye-tracking signals for test materials; aligns the EEG and eye-tracking signals with the temporal sequence of the test materials to obtain target EEG and eye-tracking signals updated with timestamps; analyzes the emotional, cognitive, and motivational arousal indicators corresponding to the target EEG signals; analyzes the gaze heatmap and gaze trajectory map corresponding to the target eye-tracking signals; and fuses the emotional, cognitive, motivational arousal, gaze heatmap, and gaze trajectory map to obtain suggested material layout information for the test materials. By extracting EEG and eye-tracking signals, unconscious neural and visual responses are measured. By analyzing core signal indicators and visualizing the data, responses to test materials are evaluated, and subconscious emotions, cognition, and motivations are deeply explored. This method can enhance the persuasiveness of materials to the target audience. Attached Figure Description

[0045] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0046] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0047] Figure 1 This is a flowchart illustrating an embodiment of the multimodal processing method based on electroencephalogram (EEG) signals provided in this application.

[0048] Figure 2 This is a flowchart illustrating Embodiment 4 of the multimodal processing method based on electroencephalogram (EEG) signals provided in this application;

[0049] Figure 3 This is a flowchart illustrating Embodiment Six of the multimodal processing method based on electroencephalogram (EEG) signals provided in this application;

[0050] Figure 4 This is a schematic diagram of the structure of the multimodal processing device based on electroencephalogram (EEG) signals in the embodiments of this application.

[0051] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0052] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of this application and are not intended to limit this application.

[0053] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0054] Currently, electroencephalogram (EEG) signals are the core carriers reflecting brain activity and are widely used in scenarios such as emotion recognition and cognitive assessment. However, EEG signals are inherently weak and easily affected by muscle movements, eye movements, and environmental electromagnetic interference. Related technologies use filters, resulting in low signal-to-noise ratios for signals collected by single-point or limited electrodes on the forehead, making it difficult to extract stable and reliable emotion and cognitive indicators. Furthermore, the assessment dimensions are not comprehensive enough, primarily focusing on information flow and creative assessment, leading to relatively narrow application scenarios.

[0055] The main solution of this application is as follows: Acquire EEG and eye-tracking signals for the test material; align the EEG and eye-tracking signals with the time sequence of the test material to obtain the target EEG and target eye-tracking signals after updating the timestamps; analyze the emotional, cognitive, and motivational arousal indicators corresponding to the target EEG signals; analyze the fixation heatmap and fixation trajectory map corresponding to the target eye-tracking signals; and fuse the emotional, cognitive, motivational arousal, fixation heatmap, and fixation trajectory map to obtain suggested material layout information for the test material. By extracting EEG and eye-tracking signals, unconscious neural and visual responses are measured. By analyzing and visualizing the core signal indicators, responses to the test material are evaluated, and subconscious emotions, cognition, and motivations are deeply explored. This achieves the technical effect of extracting stable and reliable emotional and cognitive indicators.

[0056] It should be noted that the execution subject of this embodiment can be a multimodal processing device based on EEG signals, or a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or a multimodal processing device based on EEG signals capable of performing the above functions. This embodiment does not specifically limit it in this way. The following uses a multimodal processing device based on EEG signals as the execution subject as an example to describe this embodiment and the following embodiments.

[0057] Based on this, Embodiment 1 of this application proposes a multimodal processing method based on electroencephalogram (EEG) signals. Please refer to... Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the multimodal processing method based on electroencephalogram (EEG) signals according to this application. The multimodal processing method based on EEG signals includes steps S10 to S50:

[0058] Step S10: Obtain EEG signals and eye movement signals for the test material.

[0059] The test materials are stimulus carriers that stimulate test subjects to produce specific electroencephalographic (EEG) and eye-tracking responses using a 24-electroencephalogram (EEG) device and a high-precision eye tracker. Specific formats include videos, images, web pages, and simulated product shelves. The 24-electroencephalogram (EEG) device refers to an EEG signal acquisition device with a 24-lead electrode layout, meaning it is equipped with 24 acquisition electrodes positioned in different areas of the test subject's scalp, capable of simultaneously acquiring neuronal electrical activity signals from multiple functional areas of the brain. EEG signals refer to the potential changes generated by the electrical activity of brain neurons, acquired through electrodes, and are a direct electrophysiological reflection of brain neural activity. Eye-tracking signals refer to the set of parameters reflecting the state of eye movement, acquired through eye-tracking equipment.

[0060] In this embodiment, the EEG device No. 24 and the high-precision eye tracker initiate data acquisition. The EEG device No. 24 collects real-time data on the electrical activity potential changes of neurons in multiple brain regions of the test subject, generating EEG signals. The high-precision eye tracker collects real-time data on the eye movements of the test subject, generating eye movement signals, and records system timestamps during the acquisition process. The EEG device No. 24 improves the spatial resolution of the EEG signals, enabling the capture of differences in electrical activity across different functional areas of the brain, providing a high-quality data foundation for subsequent analysis.

[0061] In a first feasible implementation, step S10 may include: simultaneously activating the continuous acquisition mode of the EEG device 24 and the continuous acquisition mode of the eye tracker, recording the system timestamps during the acquisition process, and automatically adding material time markers to the acquired data from the EEG device and the eye tracker. Based on the presentation duration of the material, EEG signal segments and eye movement signal segments are extracted and integrated into a signal dataset for the current test material. Signal integrity verification is added to remove intervals in the extracted segments containing abnormal data such as electrode detachment or eye tracking failure, ensuring the validity of the integrated dataset.

[0062] In a second feasible implementation, step S10 may include: synchronizing the EEG device and eye tracker based on NTP (Network Time Protocol), synchronously initiating data acquisition by the EEG device and eye tracker via software commands, storing the raw signal data in compressed format in real time, and during the acquisition process, using a built-in algorithm to filter out power frequency interference in real time, and wirelessly transmitting the processed signal data to a local server for storage. This eliminates the need for additional hardware triggering links, reducing hardware deployment complexity and device compatibility barriers.

[0063] In a third feasible implementation, step S10 may include: establishing a synchronous trigger link between the EEG device (No. 24) and the high-precision eye tracker via a TTL (Transistor-Transistor Logic) trigger signal from the data acquisition card; setting the rising edge of the trigger signal as the signal acquisition start point; synchronously initiating the acquisition process of the EEG device and the eye tracker with the trigger signal; and acquiring and storing the EEG signal data stream and eye movement signal data stream in real time. Acquisition terminates at the falling edge of the trigger signal, and the acquired data is automatically labeled with material identifiers, test subject information, and acquisition timestamps. This achieves high-precision, low-noise acquisition of EEG and eye movement signals, meeting the high-precision requirements for subsequent indicator analysis.

[0064] The above are only three feasible implementation methods of step S10 provided in this embodiment. This embodiment does not specifically limit the specific implementation method of step S10.

[0065] Step S20: Align the EEG signal and eye movement signal with the time sequence of the test material to obtain the target EEG signal and target eye movement signal after updating the timestamp.

[0066] The target EEG signal refers to the EEG signal obtained by temporally aligning the EEG signal with the time sequence of the test material as the reference time axis. The target eye movement signal refers to the eye movement signal obtained by temporally aligning the eye movement signal with the time sequence of the test material as the reference time axis.

[0067] In this embodiment, timestamp sequences corresponding to EEG and eye-tracking signals are extracted, along with the start and end markers of the test material. Using the start time of the test material as the zero point, a unified reference timeline is established based on the total duration of the material. The time deviation between the original timestamps of the EEG and eye-tracking signals and the zero point is calculated, and the timestamps of both types of signals are corrected to eliminate time offset. The target EEG and target eye-tracking signals are then obtained with updated timestamps. By eliminating the time offset between the EEG and eye-tracking signals and the test material, precise synchronization of the data's timeline is achieved, providing a unified temporal reference for subsequent processing.

[0068] In a first feasible implementation, step S20 may include: extracting system clock markers synchronized based on the NTP protocol from the data streams of EEG and eye-tracking signals, and simultaneously extracting the system clock times of the start and end of the material. Using the system clock at the start of the test material as the time zero point, a reference timeline is established based on the total duration of the material. The time deviation between the start markers of the EEG and eye-tracking signals and the start time of the material is calculated respectively, and the timestamps of the two types of signals are shifted and calibrated based on the deviation values. Signal segments are extracted according to the time interval of the reference timeline, and secondary linear correction is performed on the signals using the clock markers of key content nodes in the material. No additional hardware support is required, reducing deployment complexity while ensuring alignment accuracy.

[0069] For example, to compensate for the slight clock drift in NTP synchronization, a quadratic linear correction is performed based on key nodes. For the calibrated EEG signal, 2-3 effective key nodes are selected, and the deviation between the calibrated time and the reference time at each node is calculated. A linear fitting model is constructed with the reference time t as the abscissa and the time deviation ΔT as the ordinate, and the fitted line equation is solved, ΔT=a t+b, where a and b are fitting coefficients. The fitted linear equation is used to perform a secondary correction on all calibrated timestamps of the EEG signal. The correction formula is: final timestamp = calibrated timestamp - ΔT, compensating for the slight clock drift of NTP synchronization. Using the same logic, key node deviation calculations, linear fitting, and secondary correction are performed on the calibrated eye-tracking signal to obtain the final timestamp of the eye-tracking signal. During long-term acquisition, NTP synchronization accumulates drift due to equipment crystal oscillator errors and network fluctuations, which cannot be solved by a single translation. Through multi-key node linear fitting, the drift pattern can be accurately captured, achieving drift compensation throughout the entire time period.

[0070] In a second feasible implementation, step S20 may include: extracting start and end marker points generated by TTL trigger signals from the data streams of EEG and eye-tracking signals. These marker points are completely synchronized with the start and end times of the test material. Using the start time of the test material as the global time zero point, a reference timeline with equally spaced scales is established according to the total duration of the material. Based on the synchronization characteristics of hardware triggering, the acquisition timestamps of the EEG and eye-tracking signals are directly mapped to the reference timeline of the test material without additional time deviation calculation. According to the total duration interval of the reference timeline, corresponding EEG and eye-tracking signal segments are extracted to generate target EEG and target eye-tracking signals. Relying on the strong synchronization of hardware triggering, zero-deviation time alignment is achieved, meeting the high-precision requirements of subsequent multimodal index fusion.

[0071] Step S30: Analyze the emotional indicators, cognitive indicators, and motivational arousal indicators corresponding to the target EEG signal.

[0072] Emotional indicators refer to a set of parameters derived from the characteristic bands of the target EEG signal, reflecting the test subject's emotional response to the test material. Cognitive indicators refer to a set of parameters derived from the time-domain and frequency-domain characteristics of the target EEG signal, reflecting the test subject's cognitive processing of the test material. Motivational arousal indicators refer to a set of parameters derived from the event-related potentials and band energy change trends of the target EEG signal, reflecting the test subject's sustained attention and exploration motivation towards the test material.

[0073] In this embodiment, event-related potential components, such as the amplitude and latency of P300 (Positive 300 milliseconds Wave) and N400 (Negative 400 milliseconds Wave), are extracted from the target EEG signal. The energy of EEG rhythm bands, such as alpha waves, beta waves, and theta waves, is also extracted, along with their energy values, proportions, and ratios in different brain regions. Through the analysis and calculation of these characteristic parameters, emotional, cognitive, and motivational arousal indicators are obtained. By extracting event-related potential components and EEG rhythm band energy, corresponding emotional, cognitive, and motivational arousal indicators are obtained, transforming abstract brain activity into quantifiable and interpretable indicators, thus giving EEG data clear application value.

[0074] In a first feasible implementation, step S30 may include: segmenting and filtering the target EEG signal, using bandpass filters to extract clean frequency band signals of alpha, beta, and theta waves respectively, and eliminating aliasing interference between frequency bands. Independent noise reduction is performed on the 24-lead signals from different functional brain regions, using an independent component analysis algorithm to separate residual noise from electrooculography and electromyography, preserving effective signals of neuronal electrical activity. The time-domain energy value, total frequency band energy ratio, and characteristic frequency band energy ratio, such as the alpha-beta ratio, of signals in different frequency bands of each functional region are calculated, and the dynamic change curves of each parameter at different stages of material presentation are recorded. The target EEG signal is superimposed and averaged to extract the peak amplitude, latency, and peak duration of components such as P300 and N400, while simultaneously marking the brain region where the components appear. A preset emotion analysis model is invoked, inputting characteristic parameters such as the regional alpha-beta ratio, gamma wave amplitude, and frontal lobe theta wave energy distribution. Through the model's built-in weighted calculation and threshold determination, an emotion index is output. The system invokes a pre-defined cognitive analysis model, inputting characteristic parameters such as the proportion of parietal β-wave energy, the theta-β ratio, and temporal lobe γ-wave rhythm stability. Through regression analysis and error calibration, the model outputs cognitive indicators. Similarly, it invokes a pre-defined motivational arousal analysis model, inputting characteristic parameters such as P300 amplitude, prefrontal β-wave decay rate, and β-wave stability duration. Through classification and regression calculations, the model outputs quantitative parameters of motivational arousal. By adapting specific regional characteristic parameters to the emotion, cognition, and motivation analysis models respectively, and utilizing built-in algorithms for weighted calculation, regression analysis, and classification, the system effectively integrates multi-dimensional characteristic parameters, eliminating the random errors of single features and significantly improving the discriminative power of the indicators.

[0075] In a second feasible implementation, step S30 may include: performing global bandpass filtering on the target EEG signal, extracting the effective frequency bands of the mixed α, β, and θ waves in one go, and using a simple smoothing filtering algorithm built into the device to remove high-frequency spike noise from the signal. The average energy value, energy ratio, and core ratio of the α, β, and θ waves are calculated, and the peak amplitude and latency of the P300 and N400 components in the target EEG signal are extracted. An emotion score is mapped based on the α-β ratio, such as outputting a continuous score from 0 to 10; attention concentration is mapped based on the parietal β wave energy ratio, cognitive load is mapped based on the θ-β ratio, and memory encoding efficiency is mapped based on the temporal lobe γ wave rhythm stability; a cognitive score is obtained through weighted calculation; interest arousal is mapped based on the P300 component amplitude, motivation intensity is mapped based on the prefrontal θ wave energy decay rate, and attention duration is mapped based on the β wave stability duration; a motivation arousal score is obtained through weighted calculation, resulting in corresponding emotion, cognitive, and motivation arousal indicators. Based on the direct mapping rule between feature parameters and indicators, multiple feature information is integrated through weighted calculation to eliminate the random error of a single feature, while taking into account both the quantification and readability of the indicators.

[0076] Step S40: Analyze the gaze heatmap and gaze trajectory map corresponding to the target eye movement signal.

[0077] A gaze heatmap is a visual visualization of the visual attention intensity of test material, generated based on parameters such as gaze coordinates and gaze duration in the target eye movement signal. A gaze trajectory map is a dynamic visualization of the visual attention path of the test object, generated based on parameters such as gaze coordinates, gaze switching sequence, and saccade path in the target eye movement signal.

[0078] In this embodiment, core parameters are extracted from the target eye movement signal, including the two-dimensional coordinates of the fixation point at each moment, the dwell time of each fixation point, and the start and end coordinates of the saccade movement. The display area of ​​the test material is divided into a pixel grid of equal size, such as a 50×50 grid. The number of fixation points in each grid and the cumulative dwell time of all fixation points are counted. According to the rule that the longer the cumulative dwell time or the more fixation points, the higher the heat value, the higher the heat value is assigned to the grid, such as red = high attention, yellow = medium attention, and blue = low attention. The heat color grid is superimposed on the original test material image to generate a fixation heat map. In the order of the presentation time of the test material, a timestamp is marked for each valid fixation point. In the material coordinate system, the positions of all fixation points are marked in chronological order. Different sized dots represent the dwell time, i.e., the larger the dot = the longer the dwell time. Adjacent fixation points are connected by line segments with arrows, and the direction of the arrows represents the direction of visual attention transfer, forming a saccade path. The marked fixation points and saccade paths are superimposed on the original test material image to generate a fixation trajectory map. By generating gaze heatmaps and gaze trajectory maps, abstract parameters are transformed into visual images, presenting the differences in attention to different areas of the material and restoring the user's browsing order, which can quickly help understand the user's visual attention patterns towards the test material.

[0079] In a first feasible implementation, step S40 may include: extracting all parameters from the target eye movement signal, including the two-dimensional coordinates of the fixation point, fixation duration, saccade start and end coordinates, saccade velocity, and pupil diameter change. Based on the importance of different content areas in the material, a non-uniform grid is used, with a finer grid in the core content area and a sparser grid in the secondary area. The grid attention level is calculated by combining three dimensions: the number of fixation points, the cumulative fixation duration, and the average pupil diameter. For example, the weight allocation is 0.5 for fixation duration, 0.3 for the number of fixation points, and 0.2 for pupil diameter. A 10-level color gradient is set to represent the attention level differences, superimposed on the original material image, and the specific attention level value of each high-attention grid is labeled to obtain the corresponding fixation heatmap. Each valid fixation point is timestamped to the millisecond. Different colored dots represent fixations at different time stages, dot size represents fixation duration (larger dots indicate longer fixation), line thickness represents saccade speed (thicker lines indicate slower saccade speed), and arrow direction represents visual shift direction. Key events such as long-duration fixations, rapid saccade shifts, and repeated fixation areas are marked. These are then linked to corresponding content nodes, and the frequency density of the trajectory path is superimposed on the trajectory map to show the areas the user repeatedly browses, resulting in a corresponding fixation trajectory map. Through refined processing and multi-dimensional parameter fusion, the generated fixation heatmap and fixation trajectory map accurately reproduce the details and dynamic changes of the user's visual attention.

[0080] For example, to filter out invalid interference data, after extracting the core feature parameters to be screened from the target eye movement signal, eye movement data during a 500ms silent period before the test material is presented is selected as the baseline. The mean and standard deviation of each feature parameter within this period are calculated to establish a normal eye movement feature baseline model. Dynamic threshold calculation rules are set for different feature parameters. Adaptive or fixed coefficients are set for the baseline coordinate standard deviation, the mean baseline fixation duration, the mean baseline saccade velocity, and the mean baseline pupillary area change rate, respectively, to obtain the corresponding coordinate fluctuation threshold, shortest fixation duration threshold, saccade velocity threshold, and pupillary area change rate threshold. Each data point of the target eye movement signal is traversed, and multi-feature joint anomaly identification is performed by verifying whether the fixation point coordinate fluctuation value exceeds the threshold, whether the fixation duration is lower than the threshold, whether the saccade velocity exceeds the threshold, and whether the pupillary area change rate exceeds the threshold. Data points that meet any of the conditions are marked as anomaly data points, and those that do not are valid data points. Outlier data points are removed in batches, and linear interpolation of valid data points before and after the initial screening is used to complete continuous abnormal data segments caused by prolonged blinking, ensuring the temporal continuity of eye movement data. Based on the initially screened valid data, the mean and standard deviation of feature parameters are recalculated, and adaptive coefficients are iteratively adjusted to optimize the threshold. Simultaneously, the distribution of eye movement data before and after screening is compared to verify whether the valid data points conform to normal visual fixation patterns, ensuring the accuracy of the screening results. The adaptive threshold, through baseline calibration and dynamic coefficient adjustment, can accurately identify and filter invalid data caused by non-visual attention such as blinking and head movements, ensuring that the retained valid data fully corresponds to natural eye fixation behavior, providing a high-quality data source for the subsequent generation of fixation heatmaps and fixation trajectory maps.

[0081] Furthermore, to reflect users' visual focus at different times, the test materials were divided into several consecutive time periods based on their content structure and presentation duration. Hotspot sub-maps were generated for each time period, and all sub-maps were arranged chronologically to form a time-segmented hotspot map. These time-segmented hotspot sub-maps reveal the shifting patterns of user attention over time, as well as the path of attention movement from one area to another. For example, they can identify whether users first focus on the title, then the core content, and finally ignore the ending information.

[0082] In a second feasible implementation, step S40 may include: extracting core parameters from the target eye-tracking signal, retaining only the two-dimensional coordinates of the gaze point and the gaze duration. The material display area is divided into a uniform grid of fixed size, such as 30×30, simplifying the calculation process. Grid attention is calculated based on the cumulative gaze duration; the longer the duration, the higher the attention. A three-level color gradient is set and superimposed on the original material to generate a global static gaze heatmap. The gaze points are divided into three stages—early, middle, and late—according to the material presentation order, distinguished by three colors. Gaze points are marked with dots of uniform size, and adjacent points are connected by ordinary line segments to represent the transfer path, generating a concise trajectory path map. By extracting only the two-dimensional coordinates of the gaze point and the gaze duration as core parameters, the data acquisition and processing load is reduced. The use of a fixed uniform grid simplifies the attention statistics logic. Calculating grid attention based on the cumulative gaze duration in a single dimension avoids the complexity of multi-parameter weighted calculations. By simplifying the parameter extraction and processing process, the efficiency of map generation is significantly improved.

[0083] Step S50: Integrate emotional indicators, cognitive indicators, motivational arousal indicators, gaze heatmaps, and gaze trajectory maps to obtain suggested information on the arrangement of test materials.

[0084] In this embodiment, emotional indicators, cognitive indicators, motivational arousal indicators, gaze heatmaps, and gaze trajectory maps are precisely matched by timestamps to establish a correspondence between user brain responses, visual attention behaviors, and content nodes. It accurately identifies the specific moments that trigger peak positive and negative responses, such as a product close-up at the 5-second mark, the advertising music at the 10-second mark, or key elements in a graphic, such as the position of the logotype on packaging or the color of a promotional label. An overall score is calculated, and a structured diagnostic report is provided, including narrative rhythm analysis, brand recall point effectiveness, cross-media comparison effects, and differences in responses among different groups. Through multimodal fusion, the limitations of single indicators are overcome, comprehensively uncovering the true feedback characteristics of users to the content.

[0085] In a first feasible implementation, step S50 may include: aligning the dynamic change curves of emotion indicators, cognitive indicators, and motivational arousal indicators frame-by-frame with time-segmented gaze hotspot sub-maps and time-stamped gaze trajectory maps to establish a database linking material content time, EEG physiological indicators, and eye-tracking visual behavior. Peak EEG data is extracted to accurately pinpoint the specific moments and corresponding material elements that trigger positive and negative peak user responses. Scoring weights are set, and scores for each material segment and the overall report are calculated. A comprehensive, structured diagnostic report is generated, covering narrative rhythm analysis, brand memory point effects, cross-media comparison effects, and differences in responses among different groups. Based on the diagnostic results, layout suggestions are output, the optimized material scheme is simulated, and the magnitude of indicator improvement is predicted. By aligning the frame-by-frame temporal sequence, the dynamic change curves of EEG indicators are bound to time-stamped eye-tracking indicators, breaking down the information barrier between internal EEG responses and external eye-tracking behavior. By extracting peak EEG data, the specific moments and corresponding material elements that trigger positive and negative user responses can be accurately pinpointed, uncovering the true feedback logic of users to the material and avoiding the one-sidedness of superficial data analysis.

[0086] For example, to accurately identify key moments that trigger positive or negative user responses, peak thresholds for emotional, cognitive, and motivational arousal indicators are set based on sample data accumulated in the laboratory. The dynamic curves of EEG indicators are traversed to identify and extract all data that meet the peak thresholds. The precise timestamp, specific score, and indicator type (e.g., positive or negative) for each peak are recorded. Based on the timestamp corresponding to the peak, the relevant visual material is located, and the specific elements triggering the peak response are precisely labeled. Dynamic material labels include, for example, a close-up of the product at second X, or background music in an advertisement at second Y; static material labels include, for example, the upper left corner of the logo on packaging, or the area of ​​a red promotional label. By setting scientific peak thresholds, key moments that trigger positive / negative user responses are accurately identified. Then, by matching the timestamps with the corresponding visual elements, it is clear which elements in the visual material affect the user experience, significantly improving the targeting of optimization.

[0087] In a second feasible implementation, step S50 may include: matching the score ranges of emotional indicators, cognitive indicators, and motivational arousal indicators with fixation heatmaps and fixation trajectory maps at the material fragment level to establish a content module-user response level association table. Screening for abrupt changes in EEG indicator levels to identify content modules and corresponding material elements that trigger changes in user responses. Setting scoring weights, calculating the overall score, and classifying it into levels such as excellent, satisfactory, and needing optimization. Generating a structured diagnostic report that includes core narrative rhythm, brand memory point effectiveness, and differences in responses between core and non-core user groups, and outputting standardized layout suggestions based on the level determination. By matching material fragments at the material fragment level, the score ranges of EEG indicators are associated with eye-tracking maps to quickly locate core response modules by screening for abrupt changes in indicator levels, significantly reducing computational complexity and time consumption.

[0088] This embodiment provides a multimodal processing method based on electroencephalogram (EEG) signals. It acquires EEG and eye-tracking signals for test materials; aligns the EEG and eye-tracking signals with the temporal sequence of the test materials to obtain target EEG and target eye-tracking signals after updating the timestamp; analyzes the emotional, cognitive, and motivational arousal indicators corresponding to the target EEG signals; analyzes the fixation heatmap and fixation trajectory map corresponding to the target eye-tracking signals; and fuses the emotional, cognitive, motivational arousal, fixation heatmap, and fixation trajectory map to obtain suggested material layout information for the test materials. By extracting EEG and eye-tracking signals, unconscious neural and visual responses are measured. By analyzing core signal indicators and visualizing them, responses to test materials are evaluated, and subconscious emotions, cognition, and motivations are deeply explored. This method achieves the technical effect of extracting stable and reliable emotional and cognitive indicators.

[0089] Based on Embodiment 1, in Embodiment 2 of this application, the content that is the same as or similar to that in Embodiment 1 can be referred to the above description, and will not be repeated hereafter. On this basis, the step of aligning the EEG signal and eye movement signal with the time sequence of the test material to obtain the target EEG signal and target eye movement signal after updating the timestamp includes:

[0090] Step S21: Obtain the baseline timing information of the test material. The baseline timing information includes the playback or display timeline of the test material, the start and end times of each key screen and element.

[0091] In this embodiment, the form type of the test material is determined. If it is dynamic material, such as video, a continuous global timeline is generated based on the playback frame rate of the material to determine the total duration of the material. If it is static material, such as image, a simulated browsing timeline is generated based on a preset standard browsing duration. If it is interactive material, such as webpage, a dynamic timeline containing the operation sequence is constructed based on the user's operation behavior as the trigger node. All content nodes of the material are traversed to accurately locate and mark the start and end times of each key screen and core element. For dynamic material, key screens are marked, such as product close-up shots, brand logo flashing shots, and core audio, such as the time interval of advertising slogans. For static material, core visual elements are marked, such as the simulated browsing period corresponding to the logo position, promotional labels, and core copy. For interactive material, the operation time interval and the timing of feedback content after the operation are marked for each interactive element, such as buttons and pop-ups. The start and end times of the timeline, key screens, and elements are associated and integrated to form a benchmark timing information table, providing a unified benchmark for the timing alignment of subsequent EEG signals and eye movement signals.

[0092] Step S22: Extract the original timestamp sequence corresponding to the EEG signal and the original timestamp sequence corresponding to the eye movement signal. The original timestamp sequence is the signal generation time recorded in real time by the signal acquisition device.

[0093] In this embodiment, raw timestamp sequences are extracted from EEG signal data. These sequences contain the real-time recording time of each set of EEG signal data. Similarly, raw eye-tracking timestamp sequences are extracted from eye-tracking signal data. These sequences also contain the real-time recording time of each set of eye-tracking signal data. The raw EEG and eye-tracking timestamp sequences are then standardized to remove invalid timestamps or timestamps from non-test periods. This achieves a precise correspondence between the raw signals and the acquisition time, providing data support for subsequent timing calibration and preventing timing misalignment between signals and materials due to missing timestamps.

[0094] Step S23: Using the start time of the test material playback or display in the reference timing information as the zero point of time, calculate the time deviation values ​​between the original timestamp of the EEG signal, the original timestamp of the eye movement signal and the zero point of time.

[0095] In this embodiment, the start playback and display time of the test material is extracted from the baseline timing information and set as the zero point of the entire test process. The original timestamp sequence of the EEG signals is traversed, and the difference between each timestamp and the zero point is calculated to obtain the deviation value corresponding to each EEG signal data. Similarly, the original timestamp sequence of the eye movement signals is traversed, and the difference between each timestamp and the zero point is calculated to obtain the deviation value corresponding to each eye movement signal data. The time deviation value provides a clear correction basis for subsequent timestamp calibration, avoiding the subjective error of manual calibration.

[0096] Step S24: Based on the time deviation value, the original timestamp sequence of the EEG signal and the original timestamp sequence of the eye movement signal are calibrated and corrected to obtain a calibrated timestamp sequence that accurately corresponds to the baseline time sequence of the test material.

[0097] In this embodiment, deviation correction rules are determined. For example, if the time deviation value is ≤ a set accuracy threshold (e.g., a deviation ≤ 5ms under millisecond-level calibration), the original timestamp is directly retained, and it is determined to be without significant deviation. If the time deviation value is > a set accuracy threshold, linear interpolation or time offset compensation is used for correction. For differences in the sampling frequency of the signal acquisition devices, the timestamps of low-sampling-frequency signals are interpolated and supplemented. Based on the calibration rules, the original timestamp sequence of the EEG signal is corrected point-by-point to generate an EEG calibration timestamp sequence precisely aligned with the baseline time sequence of the test material. Similarly, the original timestamp sequence of eye movements is corrected point-by-point to generate an eye movement calibration timestamp sequence precisely aligned with the baseline time sequence of the test material. A portion of the calibration timestamps are randomly selected to verify whether they match the corresponding moments in the material content of the baseline time sequence, ensuring no deviation. This eliminates timestamp deviations caused by differences in acquisition device hardware and clock asynchrony. The generated calibration timestamp sequence is completely aligned with the baseline time sequence of the test material, laying a core foundation for accurate matching of signal data and material content.

[0098] Step S25: Associate and bind the calibrated timestamp sequence with the corresponding EEG signal data and eye movement signal data respectively to generate target EEG signal and target eye movement signal that simultaneously contain signal data, calibration timestamp and corresponding test material time.

[0099] In this embodiment, the calibrated EEG signal timestamp sequence is associated and bound one-to-one with the corresponding EEG signal data, and the test material time corresponding to that timestamp is matched. Similarly, the calibrated eye movement signal timestamp sequence is associated and bound with the corresponding eye movement signal data, and the test material time corresponding to that timestamp is matched, thus obtaining the target EEG signal and the target eye movement signal. Through the precise binding of signal data, calibration timestamps, and test material time, each set of signal data can be accurately anchored to a specific node in the test material.

[0100] In this embodiment, the independent acquisition timestamps of EEG signals and eye movement signals are uniformly calibrated to a time system that matches the material, which completely solves the problem of timing misalignment caused by asynchronous clocks of acquisition devices and time differences between material playback and signal acquisition, and achieves a one-to-one accurate correspondence between signal data and material content time.

[0101] Based on any of the above embodiments of this application, Embodiment 3 of this application proposes a multimodal processing method based on electroencephalogram (EEG) signals, which can be referred to the above description and will not be repeated hereafter. Based on this, the steps of parsing the emotional indicators, cognitive indicators, and motivational arousal indicators corresponding to the target EEG signal include:

[0102] Step S31: Extract feature parameters from the target EEG signal. Feature parameters include event-related potential components and EEG rhythm frequency band energy.

[0103] Event-related potential (ERP) components refer to the potential changes recorded in the brain after receiving or processing a specific external stimulus, precisely locked in time with the stimulus event. EEG rhythm frequency band energy refers to the signal energy value or proportion of EEG signals within different frequency ranges; energy changes in different frequency bands correspond to different physiological and psychological states of the brain.

[0104] In this embodiment, the target EEG signals within a specified time window are superimposed and averaged to cancel out random noise from spontaneous EEG. Event-related potential components, such as P300 and N400, are identified from the superimposed waveform. Core parameters of each component, such as latency, amplitude, and distribution area, are extracted to obtain event-related potential component data. The target EEG signals are divided according to EEG rhythm frequency bands, for example, into five sub-bands: delta wave (0.5–4 Hz), theta wave (4–8 Hz), alpha wave (8–13 Hz), beta wave (13–30 Hz), and gamma wave (30–100 Hz). Power spectral density analysis is used to calculate the energy value and energy percentage of each sub-band signal within the specified time window. The frequency band energy values ​​and energy percentages for each time period are integrated according to the time axis of the test material to obtain EEG rhythm frequency band energy data. The superimposed averaging method effectively separates event-related potential components strongly correlated with the stimulus events in the material. Through frequency band decomposition and energy calculation, the rhythmic activity characteristics of the brain under different states are quantitatively characterized.

[0105] Step S32: Based on the feature parameters, obtain the emotion index, cognitive index and motivation arousal index.

[0106] In this embodiment, feature parameter index mapping rules are established. For example, for emotion index mapping rules, an increased proportion of alpha wave energy corresponds to a relaxed state, an increased proportion of theta wave energy corresponds to anxiety, an increased P300 amplitude corresponds to positive emotional arousal, and an abnormally increased N400 amplitude corresponds to negative emotional reactions. For cognitive index mapping rules, a prolonged P300 latency and decreased amplitude correspond to increased cognitive processing difficulty, an increased proportion of gamma wave energy corresponds to active memory encoding, and an increased theta / β ratio corresponds to high cognitive load. For motivational arousal index mapping rules, an increased proportion of β wave energy corresponds to focused attention and enhanced motivation, a continuously decreasing proportion of alpha wave energy and a stable proportion of β wave energy correspond to a high motivational maintenance state, and the frequency and amplitude of the P300 component correspond to the attractiveness of the material elements to the user. Feature parameters related to emotion, cognition, and motivational arousal are selected, and emotion indices, cognitive indices, and motivational arousal indices are obtained through preset analytical models. The emotion indices, cognitive indices, and motivational arousal indices are then bound to the test materials according to calibration timestamps. By establishing mapping rules between feature parameters and indicators, emotional, cognitive, and motivational states that cannot be directly observed are transformed into quantifiable and comparable objective indicators, thus achieving accurate assessment of psychological states.

[0107] In this embodiment, the extracted EEG feature parameters and the analyzed indicators are bound to the specific time and key elements of the test material. This allows for precise identification of which material content triggered which psychological reactions in users, providing direct data support for subsequent assessment of the quality of the material and the direction of optimization.

[0108] Based on any of the above embodiments of this application, Embodiment 4 of this application proposes a multimodal processing method based on electroencephalogram (EEG) signals, which can be referred to the above description and will not be repeated hereafter. Based on this, please refer to... Figure 2 , Figure 2 This is a flowchart illustrating Embodiment 4 of the multimodal processing method based on electroencephalogram (EEG) signals provided in this application. The steps for obtaining emotion indicators, cognitive indicators, and motivational arousal indicators based on feature parameter parsing include:

[0109] Step S321: Based on the preset emotion analysis model, input the amplitude and time of LPP wave and the proportion of β wave energy in the event-related potential components, and analyze to obtain emotion indicators. The emotion indicators include at least excitement, pleasure, disgust and boredom.

[0110] LPP (Late Positive Potential) waves are event-related potentials with a relatively long latency, reflecting the brain's sustained emotional processing and motivational salience assessment of stimuli. LPP wave amplitude refers to the peak potential difference of the LPP wave within a time window, a key indicator for measuring the intensity of emotional processing. LPP wave duration refers to the duration from the appearance of the LPP wave to its return to baseline levels, reflecting the duration of emotional processing. Beta waves are an important frequency band in brain electrical rhythms, primarily related to the brain's arousal state, active attention, and emotional arousal. Beta wave energy ratio refers to the ratio of the signal energy in the beta wave band to the total energy of the entire brain electrical signal band, used to measure the excitability of the cerebral cortex.

[0111] In this embodiment, the LPP wave amplitude, LPP wave duration, and β wave energy percentage are obtained and calculated. Following the model input format, these parameters are input into the model. The model outputs quantitative results based on its built-in algorithm. For example, the LPP wave amplitude is used as the core weight feature to determine the positive or negative aspect of the emotion; a higher amplitude indicates higher levels of pleasure and excitement, while a lower amplitude indicates higher levels of aversion and boredom. The LPP wave duration is used to assist in determining the intensity of the emotion's duration; a longer duration results in higher scores for pleasure and excitement, while a shorter duration results in higher scores for boredom. The β wave energy percentage is used to determine emotional arousal; a higher percentage results in higher scores for excitement, while a lower percentage results in higher scores for boredom. The model outputs quantitative scores for excitement, pleasure, aversion, and boredom, with higher scores indicating stronger corresponding emotions. By combining the time-domain characteristics of LPP waves with the energy proportion of β waves as complementary inputs, the limitations of single features are overcome, and four types of quantitative indicators—excitement, pleasure, aversion, and boredom—are output simultaneously, comprehensively covering the dimensions of users' emotional response to the material.

[0112] Step S322: Based on the preset cognitive analysis model, input the latency and amplitude of the P300 wave and the energy ratio of the theta wave to the alpha wave in the event-related potential components, and analyze to obtain cognitive indicators. The cognitive indicators include at least the degree of attention concentration and memory encoding efficiency.

[0113] The P300 wave is one of the most representative positive waveforms in event-related potentials (ERPs), reflecting the brain's attentional allocation, memory updating, and decision-making processes in response to stimulus information. It is highly sensitive to key visual and auditory elements related to the test material. The P300 wave latency refers to the time interval from the occurrence of the stimulus event in the test material to the P300 wave reaching its peak potential, used to measure cognitive processing efficiency. The P300 wave amplitude refers to the potential difference between the peak P300 wave and the baseline potential, used to measure the depth of cognitive processing and the intensity of attentional engagement. Theta waves belong to the low-frequency range of EEG rhythms, ranging from 4–8 Hz, and are mainly associated with light sleep, inattention, and excessive cognitive load. When users are confused or fatigued by the content, theta wave energy increases significantly. Alpha waves belong to the mid-frequency range of EEG rhythms, ranging from 8–13 Hz, and are mainly associated with the brain's awakening, relaxation, and stable attentional state. When users are focused on processing information and have a moderate cognitive load, alpha wave energy remains at a reasonable level.

[0114] In this embodiment, the P300 wave latency and amplitude are extracted, and the theta wave energy and alpha wave energy values ​​within the corresponding time window are calculated. The ratio between these two values ​​is then calculated. The P300 wave latency, amplitude, and the theta-alpha wave ratio are input into a preset cognitive analysis model. The model performs calculations according to preset logic. For example, a higher P300 wave amplitude and shorter latency result in a higher attention concentration score, indicating more efficient brain processing of stimuli. A lower theta-alpha wave ratio corresponds to higher attention concentration, while a higher ratio indicates a greater cognitive load and easier distraction. A higher P300 wave amplitude also results in a higher memory encoding efficiency score, as a high amplitude P300 corresponds to deeper brain processing of stimulus information, which is more conducive to storing information in long-term memory. A moderate theta-alpha wave ratio, such as 0.5–1.5, provides optimal memory encoding efficiency; ratios that are too high or too low will reduce encoding efficiency. The model outputs quantitative scores for attention concentration and memory encoding efficiency. By combining the time-domain features of P300 waves with the energy ratios of theta and alpha waves as dual inputs, cognitive states are cross-validated from two dimensions: stimulus processing efficiency and EEG rhythm state, thereby reducing the error rate of single feature analysis.

[0115] Step S323: Based on the preset motivation arousal analysis model, input the N400 wave amplitude value and the energy ratio of β wave to α wave in the event-related potential components, and analyze to obtain the motivation arousal index that quantifies the purchase desire.

[0116] The N400 wave is a typical negative waveform in event-related potentials, reflecting the brain's semantic integration of stimulus information, anticipatory violation, and cognitive conflict processing. The N400 wave amplitude refers to the absolute value of the potential difference between the N400 wave trough and the baseline potential, used to measure the degree of cognitive conflict the user experiences with the information presented.

[0117] In this embodiment, N400 wave amplitude data is extracted, and the energy values ​​of β waves and α waves within the corresponding time window are calculated. The ratio between the two is calculated, and the N400 wave amplitude and the energy ratio of β waves to α waves are input into the motivation arousal analysis model. The model performs calculations according to preset logic. For example, the N400 wave amplitude is negatively correlated with purchase desire; the lower the N400 wave amplitude, the higher the purchase desire score. A lower amplitude indicates that the user has no cognitive conflict with the product or selling points in the materials and has a high acceptance level. A higher amplitude indicates that the user has doubts or misunderstandings about the selling points, resulting in a lower purchase desire. The energy ratio of β waves to α waves is positively correlated with purchase desire; the higher the ratio, the higher the purchase desire score. β waves correspond to brain arousal and active attention, while α waves correspond to relaxation and distraction. A high ratio indicates that the user maintains high attention and exploration interest in the product, and the degree of motivation arousal is high. The model outputs a quantitative score for motivation arousal. By combining the cognitive conflict representation of the N400 wave with the arousal state representation of the β and α waves, we can analyze purchasing desire from two dimensions: cognitive acceptance and attention arousal, making the quantitative results more consistent with the user's real decision-making psychology.

[0118] In this embodiment, as Figure 2 Based on feature parameters, the system performs analysis. The emotion analysis model uses LPP wave amplitude, duration, and beta wave energy ratio as input to calculate emotional indicators such as excitement and pleasure. The cognitive analysis model uses P300 wave latency, amplitude, and the energy ratio of theta and alpha waves as input, and outputs cognitive indicators such as attention concentration and memory encoding efficiency. The motivational arousal analysis model relies on N400 wave amplitude and the energy ratio of beta and alpha waves to quantify motivational arousal levels. Corresponding emotional, cognitive, and motivational arousal indicators are output. By selectively choosing EEG feature parameters strongly correlated with emotion, cognition, and motivation, and combining them with pre-defined analysis models, subjective psychological states such as excitement, pleasure, attention concentration, and purchasing desire, which cannot be directly observed, are transformed into standardized and comparable quantitative scores, enabling precise monitoring of users' psychological responses.

[0119] Based on any of the above embodiments of this application, Embodiment 5 of this application proposes a multimodal processing method based on electroencephalogram (EEG) signals, which can be referred to the above description and will not be repeated hereafter. Based on this, the steps of parsing the gaze heatmap and gaze trajectory map corresponding to the target eye movement signal include:

[0120] Step S41: Extract key eye movement parameters from the target eye movement signal. Key eye movement parameters include fixation point coordinates, duration of a single fixation point, order of fixation point appearance, and saccade amplitude.

[0121] Fixation point coordinates refer to the two-dimensional position coordinates of the user's gaze in the coordinate system of the eye-tracking device or the coordinate system of the source image when the user's gaze is stably fixed. The duration of a single fixation point refers to the length of time the user's gaze remains stably at the same fixation point coordinate position. For example, in eye-tracking signal analysis, a gaze displacement less than a preset threshold (e.g., <0.5° visual angle) and a duration greater than 100ms is considered a fixation. Gait dwells shorter than this duration are considered instantaneous saccades and are not counted as valid fixation duration. The order of fixation points refers to the chronological sequence of all valid fixations during the user's viewing of the source image. The saccade amplitude refers to the distance or angle of gaze movement as the user's gaze shifts from one fixation point to the next.

[0122] In this embodiment, fixation point coordinates are extracted by reading the eye gaze position coordinates corresponding to each effective fixation moment from the target eye movement signal; the duration of a single fixation point is extracted by calculating the duration from start to finish of each fixation point and recording it as the duration data of that fixation point; the order of fixation point occurrence is extracted by sorting all fixation points according to the order of calibration timestamps to generate a fixation point sequence containing sequential numbers; and the saccade amplitude is extracted by calculating the coordinate distance between two adjacent fixation points and combining it with the pixel actual distance conversion parameters of the eye movement device to obtain the physical amplitude or pixel amplitude of each saccade, representing the span of the user's gaze shift. By acquiring key eye movement parameters including fixation point coordinates, single fixation point duration, fixation point occurrence order, and saccade amplitude, a data foundation is provided for subsequent visualization.

[0123] Step S42: Obtain the screen coordinate system of the test material and establish the mapping relationship between the gaze point coordinates in the key eye-tracking parameters and the screen coordinates of the test material.

[0124] In this embodiment, a standardized coordinate system for the test material is established based on its shape. For example, the lower left corner of the material image is used as the origin, with the horizontal direction to the right as the positive x-axis and the vertical direction upward as the positive y-axis. The maximum coordinate value is set according to the resolution of the material, clearly defining the coordinate value corresponding to each pixel. Several feature anchor points in the test material image are selected, such as prominent elements at the four corners and center of the image. The standard coordinates of these anchor points in the material coordinate system are recorded, and the original coordinate data of the eye-tracking device when the user gazes at these anchor points are retrieved. A linear transformation algorithm is used to construct a mapping relationship from the original coordinates of the eye-tracking device to the coordinates of the material image. The offset and scaling factor of the coordinate transformation are calculated. Non-anchor elements in the material image are randomly selected to verify whether the transformed gaze point coordinates match the actual visual element positions. Using the mapping relationship, all gaze point coordinates are batch-transformed to obtain the target gaze point coordinates corresponding to the coordinates of the test material image. By establishing the mapping relationship, the original coordinates of the eye-tracking device are converted into the standard coordinates of the material image, providing a core foundation for subsequent positioning of high-attention material elements.

[0125] Step S43: Based on the mapping relationship and the duration of a single gaze point, a gaze hotspot map is generated. The gaze hotspot map uses thermal intensity gradient to represent the degree of gaze concentration in different regions, and the thermal intensity is positively correlated with the gaze duration.

[0126] In this embodiment, the test material is divided into several continuous analysis regions according to its content structure. If no directional analysis is required, the image is divided into uniform grid regions with a fixed pixel size, such as 50×50 pixels. Based on the mapped target gaze coordinates, the image region to which each gaze point belongs is determined. The duration of all gaze points within each region is accumulated to obtain the total gaze duration for each region, and the number of gaze points in each region is calculated. A positive correlation mapping rule is established between heat map intensity and total gaze duration; the longer the total gaze duration, the higher the heat map intensity and the darker the color, for example, a gradient change from blue, yellow, to red, visualizing the heat map intensity of each region. A heat map layer can be overlaid at the corresponding position on the test material image, marking the numerical range of heat map intensity and the meaning of the colors. The heat map can directly determine which regions in the material attract user attention and which regions are ignored, guiding the optimization of the position arrangement of material elements.

[0127] Step S44: Based on the order of appearance of fixation points and the timestamps corresponding to the timing of the test material, connect the coordinates of each fixation point according to the time sequence to generate a fixation trajectory diagram. The fixation trajectory diagram is marked with the timing number of each fixation point and the corresponding fixation duration.

[0128] In this embodiment, each gaze point is supplemented with a time sequence number, a timestamp corresponding to the material, and gaze duration information. Using the playback and display timeline of the test material as the horizontal axis and the material screen coordinates as the vertical axis, arrowed line segments connect the coordinate positions of each gaze point sequentially according to their time sequence numbers. The direction of the line segments represents the user's gaze shift path. The time sequence number and gaze duration are marked at the coordinate positions of each gaze point, presenting the order and duration of the user's gaze lingering. By analyzing the trajectory and time sequence, it is possible to determine whether the user browses according to the preset logic of the material and whether there are any information omissions due to gaze jumps, providing a crucial basis for optimizing the content presentation order of the material and reducing the user's browsing cost.

[0129] In this embodiment, the user attention preferences and browsing logic reflected in the generated gaze heatmap and gaze trajectory map provide a clear decision-making basis for optimizing the layout of materials, deleting content, and adjusting the order, helping to improve the information transmission efficiency of materials and user experience.

[0130] Based on any of the above embodiments of this application, Embodiment Six of this application proposes a multimodal processing method based on electroencephalogram (EEG) signals, which can be referred to the above description and will not be repeated hereafter. Based on this, please refer to... Figure 3 , Figure 3This is a flowchart illustrating Embodiment Six of the multimodal processing method based on electroencephalogram signals provided in this application. The step of fusing emotion indicators, cognitive indicators, motivational arousal indicators, gaze heatmaps, and gaze trajectory maps to obtain suggested material arrangement information for test materials includes:

[0131] Step S51: Determine the visual logic corresponding to the target eye movement signal and the first-eye attraction parameters of each element in the test material based on the gaze heat map and gaze trajectory map.

[0132] First-glance attraction parameters are quantitative scores or parameters calculated for each independent visual element in the test material based on the heat distribution of the gaze heatmap and the first-order gaze characteristics of the gaze trajectory map. They reflect the attention-attracting effect when users first encounter the material. For example, first-glance attraction parameters include first-glance probability, which is the proportion of times a certain material element is the first thing a user looks at in the test sample out of the total number of tests. For example, if users look at the product image first 65 times in 100 tests, the first-glance probability of the product image is 0.65; first-glance duration refers to the average stable dwell time when a user first looks at a certain element. The longer the duration, the higher the depth of user attention the element attracts at the first moment; first-glance heatmap percentage refers to the proportion of the cumulative gaze duration of a certain element in the first-glance phase to the total gaze duration of that element.

[0133] In this embodiment, a set of high-heat regions (such as the top 30% of heat intensity areas) and low-heat blind spots (such as the bottom 10% of heat intensity areas) are extracted from the gaze heatmap. Test material elements corresponding to each region are recorded, such as logos, product images, and text. The gaze point sequence, gaze shift path characteristics, and long-stay areas are extracted from the gaze trajectory map. Based on the gaze point appearance order and gaze shift path, and combined with the material's pre-set information delivery logic (such as title, product, selling points, and purchase button), a matching process is performed. If the user's gaze path matches the material's pre-set logic, it is determined to be positive visual logic. If there are numerous cross-regional jumps or the first gaze point falls in a non-core area, it is determined to be reversed or chaotic visual logic. The visual logic judgment result is output, including the logic type and deviation nodes, such as the user skipping the selling points and directly focusing on the button, or an optimization direction prompt. The first gaze probability, average first gaze duration, and average heat intensity ratio of each material element in the test sample are statistically analyzed, the data is normalized, and a weighted algorithm is used to calculate the first-glance attraction score. The first-glance attraction parameter transforms behavioral characteristics such as the probability of first-glance and dwell time of an element into a comparable quantitative score, objectively reflecting the element's initial attraction to users and providing an intuitive basis for subsequent layout priority determination.

[0134] Step S52: Integrate emotional indicators, cognitive indicators, motivational arousal indicators, visual logic, and the first-glance attractiveness parameters of each element in the test material to obtain suggested material layout information for each element in the test material.

[0135] In this embodiment, emotional indicators, cognitive indicators, and motivational arousal indicators are acquired and broken down by material element dimension to obtain visual logic judgment results and first-glance attraction parameters for each element. All input parameters are normalized, such as unifying the units to the 0–1 range, to eliminate the influence of numerical differences between different indicators. A fusion weight is set for each dimension parameter; for example, emotional indicator weight 0.2, cognitive indicator weight 0.15, motivational arousal indicator weight 0.3, visual logic weight 0.1, and first-glance attraction parameter weight 0.25. For individual material elements, a weighted calculation is performed. Elements are sorted from high to low based on their overall score. Elements with higher scores are recommended to be placed in the visual core area of ​​the material, such as the upper half of the screen or the left-hand priority reading area. Less important elements with lower scores are recommended to be removed or placed in secondary visual areas, such as corners or the bottom. Based on the visual logic judgment results, the element arrangement order is optimized. If the visual logic is chaotic, it is recommended to adjust the element order to match the user's normal browsing path, such as sorting by highly attractive elements, core selling points, and purchase guidance. By integrating all optimization directions, standardized material layout suggestions are generated, including element priority, recommended positions, display format suggestions, and reasons for adjustments. The integrated analysis results are directly transformed into priority rankings, recommended positions, and display format suggestions for each element, providing a quantitative plan to directly guide material modification and effectively improve the information delivery efficiency and user conversion potential of the materials.

[0136] For example, taking advertising creatives as an example, the advertisement is broken down into three independent tracks: visual track, audio track, and copy track. Each track is further subdivided into several elements, with different elements containing their corresponding time intervals. For example, the main visual image and auxiliary animation elements in the visual track; background music and sound effects in the audio track; and the title copy and details copy elements in the copy track. For each element in each track, its emotional index, cognitive index, motivational arousal index, and first-glance attraction parameters are extracted. For each creative element, a weighted calculation is performed to calculate the element's comprehensive score, and elements with higher scores are assigned higher priority. For example, high-priority elements are assigned to the golden time interval of the advertisement, such as the time when user attention is most concentrated, such as the opening of the advertisement. The duration ratio of each element is optimized simultaneously, that is, the time interval corresponding to each element is modified. At the same time, combined with the principle of cross-track element collaborative matching, the timing of collaborative presentation of visual track, audio track, and copy track elements is determined, such as matching the title copy display time with the corresponding main visual image and voice-over. We provide a complete optimization plan for ad creative placement, including the placement order, duration, and combination methods of sub-creations within each track, as well as suggestions for the timing of cross-track creative collaboration. Leveraging multi-dimensional weighted scoring and allocation within prime time slots, we enhance the exposure efficiency of high-value elements and improve user attention capture. Through cross-track element collaborative matching, we achieve synchronized interaction between visuals, audio, and copy, enhancing the coherence and completeness of ad message delivery.

[0137] In this embodiment, as Figure 3 After inputting multimodal data, the system processes two core information streams: an eye-tracking data analysis module, which receives gaze heatmaps and gaze trajectory maps, and calculates and outputs visual logic and initial attraction parameters for each element; and an EEG data analysis module, which integrates emotion indicators, cognitive indicators, and motivational arousal indicators, ultimately outputting comprehensive psychophysiological indicators. Based on the fusion of these two types of multimodal information, and using fusion rules, the system generates and outputs element placement suggestions, thus transforming physiological and behavioral data into concrete, actionable solutions to guide interface or advertising layout optimization. The results of the fusion analysis are transformed into material placement suggestions, which directly correspond to material modification actions and can be quickly implemented in the material design and iteration process, significantly improving optimization efficiency. The generated placement suggestions are optimization strategies based on real user physiological and behavioral feedback, helping to improve the information delivery efficiency of materials, reduce user cognitive costs, and strengthen purchase motivation arousal.

[0138] Based on any of the above embodiments of this application, Embodiment Seven of this application proposes a multimodal processing method based on electroencephalogram (EEG) signals, which can be referred to the above description and will not be repeated hereafter. Based on this, the step of fusing emotion indicators, cognitive indicators, motivational arousal indicators, visual logic, and the first-glance attractiveness parameters of each element in the test material to obtain suggested material arrangement information for each element in the test material includes:

[0139] Step S521 involves quantifying and standardizing the emotion index, cognitive index, motivation arousal index, and first-glance attraction parameter to obtain standardized emotion value, standardized cognitive value, standardized motivation arousal value, and standardized attraction value with a unified dimension.

[0140] In this embodiment, emotional indicators, cognitive indicators, motivational arousal indicators, and initial attraction parameters are acquired. Extreme outliers in each indicator data are removed, such as values ​​exceeding the mean ± 3 standard deviations. Missing data is imputed using the nearest neighbor mean method. The data is then normalized, for example, by using the min-max normalization algorithm to map all indicator data to the 0–1 range. Standardized emotional values, standardized cognitive values, standardized motivational arousal values, and standardized attraction values ​​are obtained. Outlier filtering and missing value imputation effectively eliminate interference from noisy data, ensuring that the indicator data of each material element truly reflects user feedback and avoiding scoring bias caused by individual outliers.

[0141] Step S522: Based on the weight allocation rules obtained from training the preset sample set, determine the weight coefficients corresponding to each standardized indicator.

[0142] In this embodiment, a historical test sample set containing different types of materials and different user groups is selected. The sample set must include the correspondence between indicator data and material optimization effects. The sample set data is standardized. The weights are determined using the analytic hierarchy process (AHP) combined with a linear regression model. First, the importance of each indicator is qualitatively ranked based on expert experience using the AHP. Then, the linear regression model is used, with the material optimization effect in the sample set as the dependent variable and each standardized indicator as the independent variable, to fit the weight coefficients of each indicator and output the corresponding weight coefficients for each standardized indicator. The weight allocation rules obtained based on training on a large-scale sample set can be adapted to different types of test materials, improving the versatility of multimodal fusion analysis.

[0143] Step S523: Based on the weighting coefficients, calculate the comprehensive score of each element in the test material by weighted summation.

[0144] In this embodiment, based on determined weight coefficients, a weighted summation formula for the comprehensive score is set. The comprehensive score for each element is the sum of the products of standardized emotion value, standardized cognitive value, standardized motivation arousal value, and standardized attraction value with their respective weights. All elements of the test material are traversed, and the comprehensive score for each element is calculated. The contribution of core indicators is quantified and integrated through the weighted summation formula. The output comprehensive score can intuitively reflect the user feedback performance of each material element, providing a clear quantitative basis for subsequent optimization type determination.

[0145] Step S524: Combine visual logic analysis to determine the rationality of the gaze path of each element, and determine the optimization type of each element based on the comprehensive score and the rationality of the gaze path. The optimization types include core optimization elements to be retained, elements to be adjusted, and intermediate priority elements.

[0146] In this embodiment, the rationality of the gaze path of each material element is analyzed. If an element is located at a core node of the user's normal gaze path, it is determined to be a rational element; if an element is located in a blind spot where the user's gaze jumps, it is determined to be an unreasonable element. Optimization type rules are set. For example, an overall score ≥ 0.8 indicates a rational path with no significant weaknesses (i.e., a single indicator ≥ 0.6), and it is judged as a core optimization element to be retained; an overall score ≤ 0.4 < 0.8 indicates an unreasonable path with 1–2 weaknesses, and it is judged as an element to be adjusted and optimized; an overall score ≤ 0.4 < 0.8 indicates a rational path with no significant weaknesses, and it is judged as a medium priority element; elements with an overall score < 0.4 are marked as low priority elements to be deleted. Based on the judgment rules, a corresponding optimization type is matched for each material element. These three optimization types provide a clear classification guide for subsequent layout suggestions, avoiding chaotic suggestions and facilitating implementation.

[0147] Step S525: Generate layout optimization directions for elements of different optimization types to obtain material layout suggestion information.

[0148] In this embodiment, for core optimization elements, it is recommended to place them in the visual core area of ​​the material to enhance their display, such as by enlarging their size, increasing color contrast, and adding dynamic effects, maintaining their core position in the gaze path. For elements to be adjusted and optimized, their position in the material should be adjusted first, moving them to reasonable nodes in the visual path, and optimizing their presentation for weaker indicators. For intermediate priority elements, their existing positions should be maintained, with minor adjustments to display details. For low priority elements to be removed, it is recommended to remove them or move them to secondary areas at the edge of the material to avoid occupying core visual resources. Arrangement suggestions for each element are categorized by optimization type, clearly defining recommended positions, display optimization schemes, and reasons for adjustment. Based on the overall visual logic, suggestions for the arrangement order of material elements are supplemented. Material arrangement suggestion information is generated, including element name, optimization type, recommended position, and specific optimization measures. Differentiated arrangement strategies are developed for different optimization types. Each suggestion corresponds to specific position adjustments and display optimization measures, with clear reasons for adjustment, which can be directly used as a reference for modification.

[0149] In this embodiment, by integrating visual logic into the gaze path rationality analysis, a comprehensive scoring and path adaptability judgment standard is constructed. The core optimization retention, elements to be adjusted and optimized, and intermediate priority elements are accurately divided into three categories. Different layout optimization directions are output for different types of elements, forming material layout suggestions that can be directly implemented. This avoids subjective decision-making bias and greatly improves the scientificity, accuracy and execution efficiency of material optimization.

[0150] Based on any of the above embodiments of this application, Embodiment Eight of this application proposes a multimodal processing method based on electroencephalogram (EEG) signals, which can be referred to the above description and will not be repeated hereafter. Based on this, the step of generating arrangement optimization directions for elements of different optimization types to obtain material arrangement suggestion information includes:

[0151] Step S526: For the core optimized elements, generate layout optimization directions aimed at enhancing attention capture and positive neural responses. Layout optimization directions include maintaining the current display position, increasing the display area of ​​the preset ratio, and improving the visual contrast between the elements and the surrounding environment.

[0152] In this embodiment, the core optimized elements are selected based on their current display position, area ratio, visual contrast parameters, and corresponding highly standardized indicator characteristics. Position optimization is performed on these core optimized elements to maintain their current display position within the visual core area, preventing reduced user attention due to position changes. Area optimization involves increasing the display area by a preset ratio, such as 10%–20%, with the specific ratio referencing the element's original area's compatibility with the visual core area. Contrast optimization enhances the visual contrast between the element and its surrounding environment, such as adjusting color saturation and brightness differences to strengthen visual prominence. By maintaining the core position, increasing the display area, and improving visual contrast, the core elements' ability to capture user attention is strengthened, further amplifying their advantages of high emotional positive feedback and high motivation arousal, ensuring that the core value of the material can be quickly perceived by users.

[0153] Step S527: For the elements to be adjusted and optimized, generate an arrangement optimization direction aimed at improving attention acquisition ability and neural response adaptability. The arrangement optimization direction includes adjusting to the core area of ​​the gaze path, optimizing the color and shape of the elements to enhance first-glance attraction, and avoiding the visual coverage of the core optimized elements.

[0154] In this embodiment, the core weaknesses of the element to be adjusted and optimized, as well as its visual positional conflicts with core elements, are analyzed. The element is repositioned by moving it to the core area of ​​the gaze path; visual optimization is performed by improving the element's color scheme and shape design; and avoidance optimization involves adjusting the element's position and size to avoid conflict with core elements while preserving its visual coverage and preventing attention distraction caused by competition for visual resources. By employing strategies such as relocating to the core area of ​​the gaze path, optimizing visual design, and avoiding conflict with core elements, the element's initial visual appeal and adaptability to user neural responses are effectively improved.

[0155] Step S528: For intermediate priority elements, generate layout optimization directions aimed at adapting to the visual logic flow. The layout optimization directions include fine-tuning the display position to fit the direction of the gaze path extension, adjusting the display sequence to connect the gaze nodes of the core optimized and retained elements, and weakening unnecessary visual decorations to reduce attentional interference to the core elements.

[0156] In this embodiment, for intermediate priority elements, the visual logic analysis is used to determine their visual connection with core elements and elements to be adjusted, thus confirming their functional positioning within the overall content. The intermediate priority elements are then fine-tuned in position, slightly adjusting their display location to align with the user's gaze path; their timing is adjusted, for dynamic content, to connect with the gaze nodes of the core optimized elements; and visual simplification is implemented, weakening unnecessary visual embellishments to reduce attentional interference with core elements. By fine-tuning their position to align with the gaze path and adjusting their timing to connect with core elements, the overall browsing logic for the user is optimized, reducing the cognitive cost of information transmission.

[0157] Step S529: Integrate the layout optimization directions for elements of different optimization types to form material layout suggestion information that includes element identifiers, optimization actions, and optimization parameters. The optimization parameters include area adjustment ratio, position coordinate offset, and visual contrast adjustment threshold.

[0158] In this embodiment, the layout optimization directions for elements of different optimization types are integrated to construct a material layout suggestion template, which includes fields such as element identifier, optimization action, and optimization parameters, and the corresponding content is filled in. For example, the element identifier is a close-up product image, the optimization type is a core optimization and retention element, the optimization action is to increase the display area and improve visual contrast, and the optimization parameters are to increase the area by 15%, keep the position unchanged, and increase the contrast to 1.3 times. The integrated suggestion information covers all element types, taking into account both local element optimization and global visual logic coordination, ensuring that the optimized materials form a display effect that highlights the core, has clear layers, and has smooth logic.

[0159] In this embodiment, by customizing the layout optimization directions for core optimized elements, elements to be adjusted and optimized, and intermediate priority elements, precise layered optimization of each element of the material is achieved. By integrating the optimization actions and quantitative parameters of all elements, material layout suggestions that can be directly implemented are formed. This not only ensures that the advantages of core elements are amplified, but also improves the display effect of weak elements. At the same time, it strengthens the overall visual logic coherence of the material, greatly improving the information transmission efficiency and user feedback conversion ability of the material.

[0160] For example, to help understand the technical concept or principle of the multimodal processing method based on EEG signals after combining this embodiment with the above embodiments, the specific details are as follows:

[0161] This embodiment's multimodal processing method based on electroencephalogram (EEG) signals can be used for creative research and optimization. Specifically, it can be used in the production of commercials or print posters to conduct neuroscience A / B testing, thereby optimizing narrative rhythm, music, and visual elements; media strategy formulation, assessing the impact of different media environments on the neural responses of the same advertisement, and scientifically allocating budgets; packaging and design testing, testing the shelf impact and design element attractiveness of product packaging before production; pricing and promotion strategies, studying the impact of price display methods on the brain's value perception and loss aversion areas, verifying the anchoring effect, and optimizing promotional information; spokesperson / KOL selection, realistically measuring the trust and positive emotions evoked when a spokesperson appears on screen, and assessing their neural compatibility with the brand; brand health and crisis management, long-term tracking of brand-related neural association changes, and monitoring potential negative neural emotions after negative events; and cross-screen and complex environment research, understanding how advertising effectively competes for attention when consumers are multitasking. This enables scientific decision-making across the entire chain from creativity, media, and design to brand strategy, providing objective, in-depth, and actionable optimization basis, significantly improving the accuracy and conversion efficiency of advertising and brand operations.

[0162] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the multimodal processing method based on EEG signals in this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0163] This application provides a multimodal processing device based on electroencephalogram (EEG) signals. The multimodal processing device based on EEG signals includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the multimodal processing method based on EEG signals in the above embodiment 1.

[0164] The following is for reference. Figure 4 The diagram illustrates a structural schematic of a multimodal processing device based on electroencephalogram (EEG) signals suitable for implementing embodiments of this application. The multimodal processing device based on EEG signals in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablets, and in-vehicle terminals, as well as fixed terminals such as digital TVs and desktop computers. Figure 4 The illustrated multimodal processing device based on EEG signals is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0165] like Figure 4As shown, a multimodal processing device based on EEG signals may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1002 or a program loaded from storage device 1003 into random access memory (RAM) 1004. The random access memory 1004 also stores various programs and data required for the operation of the multimodal processing device based on EEG signals. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the EEG-based multimodal processing device to wirelessly or wiredly communicate with other devices to exchange data. Although the figure shows an EEG-based multimodal processing device with various systems, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0166] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0167] The multimodal processing device based on EEG signals provided in this application, employing the multimodal processing method based on EEG signals in the above embodiments, can solve the technical problem of difficulty in extracting stable and reliable emotion and cognitive indicators. Compared with related technologies, the beneficial effects of the multimodal processing device based on EEG signals provided in this application are the same as those of the multimodal processing device based on EEG signals provided in the above embodiments, and other technical features in this multimodal processing device based on EEG signals are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0168] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0169] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0170] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the multimodal processing method based on electroencephalogram (EEG) signals in the above embodiments.

[0171] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, radio frequency (RF), or any suitable combination thereof.

[0172] The aforementioned computer-readable storage medium may be included in a multimodal processing device based on electroencephalogram (EEG) signals; or it may exist independently and not assembled into an EEG-based multimodal processing device.

[0173] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by a multimodal processing device based on electroencephalogram (EEG) signals, the multimodal processing device based on EEG signals performs the following actions: acquires EEG signals and eye-tracking signals for the test material; aligns the EEG signals and eye-tracking signals with the temporal sequence of the test material to obtain target EEG signals and target eye-tracking signals after updating the timestamp; analyzes the emotion indicators, cognitive indicators, and motivational arousal indicators corresponding to the target EEG signals; analyzes the gaze heatmap and gaze trajectory map corresponding to the target eye-tracking signals; and fuses the emotion indicators, cognitive indicators, motivational arousal indicators, gaze heatmap, and gaze trajectory map to obtain suggested material layout information for the test material.

[0174] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0175] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0176] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0177] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described multimodal processing method based on electroencephalogram (EEG) signals. This solves the technical problem of difficulty in extracting stable and reliable emotion and cognitive indicators. Compared with related technologies, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the multimodal processing method based on EEG signals provided in the above embodiments, and will not be repeated here.

[0178] This application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the multimodal processing method based on electroencephalogram (EEG) signals as described above.

[0179] The computer program product provided in this application can solve the technical problem of difficulty in extracting stable and reliable emotion and cognitive indicators. Compared with related technologies, the beneficial effects of the computer program product provided in this application are the same as those of the multimodal processing method based on EEG signals provided in the above embodiments, and will not be repeated here.

[0180] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent scope of this application.

Claims

1. A multimodal processing method based on electroencephalogram (EEG) signals, characterized in that, The multimodal processing method based on electroencephalogram (EEG) signals includes: Acquire EEG and eye movement signals for the test subjects; The EEG signal and the eye movement signal are aligned with the time sequence of the test material to obtain the target EEG signal and target eye movement signal after the timestamp is updated. Analyze the emotional indicators, cognitive indicators, and motivational arousal indicators corresponding to the target EEG signals; Analyze the gaze heatmap and gaze trajectory map corresponding to the target eye movement signal; The visual logic corresponding to the target eye movement signal and the first eye attraction parameter of each element in the test material are determined based on the gaze heat map and the gaze trajectory map. The emotional indicators, cognitive indicators, motivational arousal indicators, and first-glance attraction parameters are quantified and standardized to obtain standardized emotional values, standardized cognitive values, standardized motivational arousal values, and standardized attraction values ​​of a unified dimension. Based on the weight allocation rules obtained from training a preset sample set, determine the weight coefficients corresponding to each standardized indicator. Based on the aforementioned weighting coefficients, the comprehensive score of each element in the test material is calculated by weighted summation; The rationality of the gaze path of each element is analyzed in conjunction with the visual logic. Based on the comprehensive score and the rationality of the gaze path, the optimization type of each element is determined. The optimization type includes core optimization retention elements, elements to be adjusted optimization elements, and intermediate priority elements. For elements of different optimization types, separate layout optimization directions are generated to obtain suggested material layout information.

2. The multimodal processing method based on electroencephalogram (EEG) signals as described in claim 1, characterized in that, The step of aligning the EEG signal and the eye movement signal with the time sequence of the test material to obtain the target EEG signal and target eye movement signal after updating the timestamp includes: Obtain the baseline timing information of the test material, which includes the playback or display timeline of the test material, the start and end times of each key frame and element; Extract the original timestamp sequence corresponding to the EEG signal and the original timestamp sequence corresponding to the eye movement signal. The original timestamp sequence is the signal generation time recorded in real time by the signal acquisition device. Using the start time of the test material playback or display in the reference timing information as the zero point, calculate the time deviation between the original timestamp of the EEG signal, the original timestamp of the eye movement signal and the zero point; Based on the time deviation value, the original timestamp sequence of the EEG signal and the original timestamp sequence of the eye movement signal are calibrated and corrected to obtain a calibrated timestamp sequence that precisely corresponds to the baseline time sequence of the test material; The calibrated timestamp sequence is associated and bound with the corresponding EEG signal data and eye movement signal data to generate target EEG signal and target eye movement signal that simultaneously contain signal data, calibration timestamps and corresponding test material times.

3. The multimodal processing method based on electroencephalogram (EEG) signals as described in claim 1, characterized in that, The steps of analyzing the emotional indicators, cognitive indicators, and motivational arousal indicators corresponding to the target EEG signal include: Extract feature parameters from the target EEG signal, including event-related potential components and EEG rhythm frequency band energy; The emotion index, cognitive index, and motivation arousal index are obtained by parsing the feature parameters.

4. The multimodal processing method based on electroencephalogram (EEG) signals as described in claim 3, characterized in that, The steps of parsing the emotion index, cognitive index, and motivational arousal index based on the feature parameters include: Based on a preset emotion analysis model, the amplitude and time course of the LPP wave and the proportion of β wave energy in the event-related potential components are input to obtain emotion indicators. The emotion indicators include at least excitement, pleasure, disgust and boredom. Based on a preset cognitive analysis model, the latency and amplitude of the P300 wave and the energy ratio of the theta wave to the alpha wave in the event-related potential components are input, and cognitive indicators are obtained through analysis. The cognitive indicators include at least the degree of attention concentration and memory encoding efficiency. Based on a preset motivational arousal analysis model, the amplitude of the N400 wave and the energy ratio of the β wave to the α wave in the event-related potential components are input to analyze and obtain a motivational arousal index that quantifies the desire to purchase.

5. The multimodal processing method based on electroencephalogram (EEG) signals as described in claim 1, characterized in that, The steps of analyzing the gaze heatmap and gaze trajectory map corresponding to the target eye movement signal include: Key eye movement parameters are extracted from the target eye movement signal. These key eye movement parameters include fixation point coordinates, duration of a single fixation point, order of fixation point appearance, and saccade amplitude. Obtain the image coordinate system of the test material and establish the mapping relationship between the gaze point coordinates and the image coordinates of the test material in the key eye movement parameters; Based on the mapping relationship and the duration of a single gaze point, a gaze heat map is generated. The gaze heat map uses thermal intensity gradient to represent the degree of gaze concentration in different regions, and the thermal intensity is positively correlated with the gaze duration. Based on the order of appearance of the fixation points and the timestamps corresponding to the time sequence of the test materials, the coordinates of each fixation point are connected according to the time sequence to generate a fixation trajectory diagram. The fixation trajectory diagram is marked with the time sequence number of each fixation point and the corresponding fixation duration.

6. The multimodal processing method based on electroencephalogram (EEG) signals as described in claim 1, characterized in that, The step of generating layout optimization directions for elements of different optimization types to obtain material layout suggestion information includes: For core optimized elements, layout optimization directions are generated to enhance attention capture and positive neural responses. These layout optimization directions include maintaining the current display position, increasing the display area by a preset ratio, and improving the visual contrast between the elements and the surrounding environment. For the elements to be adjusted and optimized, an optimized layout direction is generated with the goal of improving attention acquisition ability and neural response adaptability. The optimized layout direction includes adjusting to the core area of ​​the gaze path, optimizing the color and shape of the elements to enhance their first-glance attraction, and avoiding the visual coverage of the core optimized elements. For intermediate priority elements, an optimized layout direction is generated to adapt to the visual logic flow. The optimized layout direction includes fine-tuning the display position to fit the extension direction of the gaze path and adjusting the display sequence to connect the gaze nodes of the core optimized and retained elements. The layout optimization directions for elements of different optimization types are integrated to form material layout suggestion information that includes element identifiers, optimization actions, and optimization parameters. The optimization parameters include area adjustment ratio, position coordinate offset, and visual contrast adjustment threshold.

7. A multimodal processing device based on electroencephalogram (EEG) signals, characterized in that, The multimodal processing device based on electroencephalogram (EEG) signals includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the multimodal processing method based on EEG signals as described in any one of claims 1 to 6.

8. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the multimodal processing method based on EEG signals as described in any one of claims 1 to 6.

Images