Advertisement effect evaluation method, system and device based on multi-modal perception and medium

Abstract

This invention proposes a method, system, device, and medium for evaluating advertising effectiveness based on multimodal perception, belonging to the field of advertising effectiveness evaluation. The method includes: collecting first-modal and second-modal data of the test subject based on the advertising stimulus material to be tested; preprocessing the first-modal and second-modal data to extract first-modal features and second-modal features, and constructing a joint feature vector; inputting the joint feature vector into a preset multi-path advertising effectiveness evaluation model, and outputting advertising effectiveness evaluation results and strategy matching results. This invention realizes the transformation of advertising evaluation from result observation to process measurement, can quantify the contribution of internal advertising elements, reveal the formation mechanism from visual attention to advertising effectiveness, and provide a unified and interpretable evaluation framework for multi-scenario advertising.

Description

Technical Field

[0001] This invention belongs to the field of advertising effectiveness evaluation technology, and in particular relates to an advertising effectiveness evaluation method, system, device and medium based on multimodal perception. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] As digital marketing rapidly evolves towards intelligence, dynamism, and contextualization, advertising creatives and delivery methods are undergoing profound changes. On the one hand, AI-generated content (AIGC) technology enables advertising creatives to generate multiple versions of images, text, short videos, and interactive materials at lower cost and higher frequency. On the other hand, the rise of new marketing formats such as live-streaming e-commerce, short-video sales, and real-time promotions has transformed advertising from a static display into a multimodal communication scenario that integrates visual elements, linguistic information, temporal cues, interactive prompts, and promotional mechanisms.

[0004] Against this backdrop, advertisers, brands, and marketing platforms not only need to determine whether an advertisement is "effective," but also need to further identify which specific elements of the advertisement are effective, in what scenarios they are effective, for which target audiences they are most effective, and how to optimize their advertising strategies accordingly. However, existing advertising effectiveness evaluation technologies still have significant shortcomings when faced with these new multimodal, dynamic, and highly contextualized advertising evaluation tasks.

[0005] Currently, the field mainly relies on the following types of technical means: First, questionnaires based on subjective self-report, which evaluate advertising attitude, brand attitude, and purchase intention after the advertisement is exposed; Second, statistical analysis methods based on online behavioral data, which estimate the results through indicators such as click-through rate, conversion rate, dwell time, and interaction rate; Third, evaluation methods based on single physiological signals, such as collecting only EEG data or only eye movement data, to determine the level of consumer attention or preference.

[0006] While the methods described above can reflect advertising performance to some extent, each has significant limitations. Subjective evaluation methods rely on hindsight, making them susceptible to memory bias and social desirability bias, and they struggle to capture consumers' immediate attention allocation and early psychological reactions during ad viewing. Online behavioral data is an outcome-based indicator; it can only indicate whether an ad triggered a certain behavior, but it cannot explain why consumers engaged in that behavior, nor can it identify the contribution of different elements within the ad to the behavioral outcome, resulting in a "black box" problem of "visible results, unclear mechanisms." Single physiological signal assessments have limited dimensions; using eye-tracking data alone cannot determine intrinsic perceived value or emotional responses, while using EEG signals alone lacks the ability to spatially locate specific advertising elements, making it difficult to form complete, stable, and interpretable advertising evaluation conclusions.

[0007] In addition, existing technologies are mostly designed for static web page ads or traditional print ads, lacking a unified analysis framework that is compatible with various types of advertising stimuli such as static creatives, dynamic videos, live promotions, and AIGC-generated materials. They also lack high-precision spatiotemporal alignment and interest zone-level analysis capabilities, making it difficult to effectively transform evaluation results into actionable campaign optimization strategies. Summary of the Invention

[0008] To overcome the shortcomings of the prior art, the present invention provides an advertising effectiveness evaluation method, system, device and medium based on multimodal perception, aiming to solve the problems of insufficient objectivity, unclear mechanism, poor cross-scenario applicability and difficulty in supporting strategy optimization in existing advertising evaluation technologies.

[0009] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: The first aspect of this invention provides a method for evaluating advertising effectiveness based on multimodal perception; Multimodal perception-based advertising effectiveness evaluation methods include: Based on the advertising stimulus material to be tested, first-modal data and second-modal data of the test subjects were collected; After preprocessing the first modality data and the second modality data, the first modality features and the second modality features are extracted, and a joint feature vector is constructed. The joint feature vector is input into a preset multi-path advertising effectiveness evaluation model, and the advertising effectiveness evaluation results and strategy matching results are output. The multi-path advertising effectiveness evaluation model decomposes the joint feature vector into rational processing eye movement features, emotional processing eye movement features, overall attention features, psychological cognitive features, and individual difference features. Cognitive path variables were calculated based on rational processing eye movement characteristics and cognitive-related psychological characteristics, and emotional experience path variables were calculated based on emotional processing eye movement characteristics and emotional-related psychological characteristics. By integrating cognitive path variables, emotional experience path variables, and overall attention characteristics, advertising attitude or brand attitude variables can be calculated. Based on cognitive path variables, emotional experience path variables, and attitude variables, and incorporating individual differences, the comprehensive advertising effect is calculated as the advertising effect evaluation result. After obtaining comprehensive advertising effectiveness, cognitive path variables, emotional experience path variables, advertising attitude or brand attitude variables, and individual difference moderating effects, the multi-path advertising effectiveness evaluation model further aggregates the results of multiple test subjects under the same advertising version based on the above calculation results, calculates the comprehensive score of the advertising version, AOI contribution, and path strength; and determines the preferred advertising version for different sub-segments based on the comprehensive score of the advertising version, determines the interest regions that contribute more to the advertising effectiveness based on the AOI contribution, and determines whether the advertising effectiveness is mainly driven by the cognitive path or the emotional experience path based on the path strength, thereby outputting advertising effectiveness evaluation results and strategy matching results for advertising version, interest region, and sub-segments.

[0010] As a further technical solution, the advertising stimulus material to be tested has at least one area of ​​interest (AOI) predefined.

[0011] As a further technical solution, the first modal data is eye movement physiological characteristic data, including at least fixation point coordinate data, pupil data, fixation event data, saccade event data, blink and eye state data, AOI mapping data, validity and quality control data, and event synchronization marker data.

[0012] The second modality data consists of psychological and cognitive characteristic data and individual difference characteristic data collected based on the scale.

[0013] As a further technical solution, preprocessing is performed on the first modal data and the second modal data, including: The system performs invalid sample removal, missing segment interpolation, pupil data smoothing, and identification of fixation and saccade events on the first modality data. The second modality data is subjected to item completeness verification, reverse scoring, dimensional aggregation, and standardization. The preprocessed first modal data and second modal data are mapped to a unified time axis for time synchronization. The coordinates of the sampling points of the first modal data are matched with the boundaries of the regions of interest to determine the regions of interest to which each sampling point belongs, thus achieving spatial alignment.

[0014] As a further technical solution, first modality features and second modality features are extracted based on the preprocessed data, and a joint feature vector is constructed, including: Based on the data after time synchronization processing and spatial mapping, the first modal features are extracted according to the region of interest. The first modal features include at least one of the following: total fixation duration, number of fixations, initial fixation latency, fixation duration percentage, number of retrospectives, mean pupil diameter, and pupil diameter change rate. Based on the preprocessed second modality data, second modality features are extracted. The second modality features include the scores of each latent variable corresponding to the psychological cognitive feature data and the scores of each latent variable corresponding to the individual difference feature data. The first modal features are concatenated with the second modal features to construct a joint feature vector for each test subject and each stimulus material.

[0015] As a further technical solution, the cognitive path variable is:

[0016] in, Represents cognitive path variables; This represents a subvector related to cognitive appraisal within psychological cognitive characteristics; To rationally process eye-tracking feature vectors; This is the cognitive path intercept term; Transpose of the weight coefficient matrix corresponding to rational processing of eye-tracking features; This is the transpose of the weight coefficient matrix corresponding to the features of the cognitive assessment scale. For residual terms; The emotional experience path variables are:

[0017] in, Represents variables related to the path of emotional experience; Represents the sensory processing eye-movement feature vector; This represents a subvector in psychological cognitive characteristics that is related to emotional experience. For the intercept term of the emotional experience path; This is the transpose of the weighting coefficient matrix corresponding to the sensory processing eye movement features; This is the transpose of the weight coefficient matrix corresponding to the features of the emotional experience scale; This is the residual term.

[0018] As a further technical solution, the advertising attitude or brand attitude variable is:

[0019] in, Variables representing advertising attitude or brand attitude; For the attitude integration path intercept term; This represents the coefficient of influence of cognitive path variables on advertising attitudes or brand attitudes. Represents cognitive path variables; The coefficient representing the influence of emotional experience path variables on advertising attitude or brand attitude; Represents variables related to the path of emotional experience; This represents the transpose of the weight coefficient matrix corresponding to the overall attention features, used to measure the impact of overall visual attention allocation on attitude variables; Indicates the overall characteristics of attention; This represents the transpose of the weight coefficient matrix corresponding to the features of the attitude scale, which is used to correct the attitude variable using subjective scale scores; This represents a subvector in psychological cognitive characteristics that is related to advertising attitude or brand attitude. This is the residual term.

[0020] A second aspect of the present invention provides an advertising effectiveness evaluation system based on multimodal perception.

[0021] A multimodal perception-based advertising effectiveness evaluation system includes: The data acquisition module is configured to: collect the first modality data and the second modality data of the test subjects based on the advertising stimulus material to be tested; The feature extraction module is configured to: preprocess the first modality data and the second modality data, extract the first modality features and the second modality features, and construct a joint feature vector; The evaluation result output module is configured to: input the joint feature vector into a preset multi-path advertising effect evaluation model, and output advertising effect evaluation results and strategy matching results; The multi-path advertising effectiveness evaluation model decomposes the joint feature vector into rational processing eye movement features, emotional processing eye movement features, overall attention features, psychological cognitive features, and individual difference features. Cognitive path variables were calculated based on rational processing eye movement characteristics and cognitive-related psychological characteristics, and emotional experience path variables were calculated based on emotional processing eye movement characteristics and emotional-related psychological characteristics. By integrating cognitive path variables, emotional experience path variables, and overall attention characteristics, advertising attitude or brand attitude variables can be calculated. Based on cognitive path variables, emotional experience path variables, and attitude variables, and incorporating individual differences, the comprehensive advertising effect is calculated as the advertising effect evaluation result. The method calculates the overall score of the ad version, AOI contribution, and path strength. Based on the calculation results, it outputs ad performance evaluation results and strategy matching results for ad version, interest region, and segmented audience. A third aspect of the invention provides a computer-readable storage medium storing a program that, when executed by a processor, implements the steps of the multimodal perception-based ad performance evaluation method described in the first aspect of the invention.

[0022] A fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in the advertising effectiveness evaluation method based on multimodal perception as described in the first aspect of the present invention.

[0023] The above one or more technical solutions have the following beneficial effects: (1) This invention achieves continuous measurement of the entire process of consumer advertising exposure by collecting first modal data such as eye movement in real time during the advertising presentation process and second modal data such as psychological cognition and individual differences after presentation. At the same time, by using spatiotemporal alignment technology, consumers' visual attention is mapped to specific interest areas such as product area, price area, anchor area, and promotion area, so that the contribution of each element within the advertisement can be quantitatively analyzed, thereby improving the interpretability of advertising evaluation results.

[0024] (2) This invention constructs a universal evaluation infrastructure decoupled from specific advertising formats through unified stimulus presentation management, multi-source data acquisition interfaces, standardized spatiotemporal alignment mechanisms, and modular feature extraction processes. Whether facing static text and images, dynamic videos, or live streams, adaptation can be achieved through dynamic interest region tracking and unified timeline mapping. Simultaneously, the individual difference moderating variable introduced into the model enables the system to identify differences in advertising response paths among different audience groups, outputting differentiated strategies for segmented audiences, and achieving a leap in refined evaluation capabilities.

[0025] (3) This invention overcomes the limitations of traditional methods that only output a single effect score by constructing a multi-path advertising effect evaluation model that includes cognitive path, emotional experience path, attitude fusion path, and moderating effect. This model can simultaneously quantify the differentiated impact of rational information processing and emotional experience on advertising attitudes, and reveal the moderating effect of individual differences (such as product involvement, price sensitivity, etc.) on path strength. By calculating the path coefficient, indirect effect, total effect, and contribution of interest area, the model can clearly determine the dominant driving factor of advertising effect (product information driven or creative atmosphere driven), and output differentiated optimization strategies for different sub-segments accordingly, realizing a technological leap from "outcome evaluation" to "mechanism diagnosis" and then to "strategy generation".

[0026] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0027] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0028] Figure 1 This is a flowchart of the method in the first embodiment.

[0029] Figure 2 This is a schematic diagram of the architecture of the multi-path advertising effectiveness evaluation model in the first embodiment.

[0030] Figure 3 This is a schematic diagram of the AOI division and eye movement heat zone of the advertising stimulus material in the first embodiment.

[0031] Figure 4 This is a system structure diagram of the second embodiment. Detailed Implementation

[0032] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0033] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0034] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0035] Example 1 This embodiment discloses an advertising effectiveness evaluation method based on multimodal perception. By using advertising stimuli in predefined interest regions, eye-tracking data, psychological cognition, and individual difference data are collected in real time. Spatiotemporal alignment and feature extraction are performed on the multimodal data. Joint features are constructed and input into a multi-path evaluation model to output advertising effectiveness evaluation results. This invention realizes the transformation of advertising evaluation from result observation to process measurement, can quantify the contribution of internal elements of advertising, reveal the formation mechanism from visual attention to advertising effectiveness, and provide a unified and interpretable evaluation framework for multi-scenario advertising.

[0036] like Figure 1 As shown, the advertising effectiveness evaluation method based on multimodal perception includes: Step S1: Based on the advertising stimulus material to be tested, collect the first modality data and the second modality data of the test subjects.

[0037] Step S1.1: First, present the advertising stimulus material to be tested to the test subjects. The stimulus material can be a static advertising image, AIGC-generated advertising text and images, short video ads, live promotional clips, or marketing interfaces with price or countdown cues. The stimulus material is grouped and arranged according to the experimental design, and one or more Regions of Interest (AOIs) are predefined for each stimulus material. These AOIs include, but are not limited to: product main area, brand logo area, price area, promotion area, countdown area, presenter area, interactive prompt area, non-product creative area, and background atmosphere area. Before the stimulus is presented, a gaze calibration page, a blank screen page, or a central gaze cross page is set to eliminate the residual effects of the previous stimulus and establish a unified time baseline.

[0038] Step S1.2: During the presentation of the stimulus material, the first modality data of the test subject is acquired in real time. The first modality data is preferably eye-tracking physiological characteristic data, which can be acquired by a screen-based eye tracker or a wearable eye-tracking device. In extended embodiments, other physiological signals such as electroencephalogram (EEG), electrodermal conductance (EDA), and heart rate variability can be further integrated, but the basic implementation of this invention includes at least eye-tracking data. In one specific embodiment, the first modality data can be obtained using eye-tracking and multimodal synchronous acquisition equipment already configured in the laboratory. The eye-tracking device includes one or more of the following: a screen-based eye tracker, a telemetry eye tracker, wearable eye-tracking glasses, and a virtual reality integrated eye-tracking device. For example, in testing static image and text advertisements, web page advertisements, or short video advertisements, screen-based eye trackers can be used to collect data on the gaze coordinates, gaze events, saccade events, pupil diameter, and eye state of the test subject on the screen stimulus material; in testing live shopping, offline shelves, real-world scene advertisements, or mobile interactive advertisements, wearable eye-tracking devices can be used to collect data on the gaze points, scene videos, and AOI mapping data of the test subject in natural scenes; and in testing virtual advertisements or immersive advertisements, VR integrated eye-tracking devices can be used to collect gaze behavior data in virtual environments.

[0039] The first modal data includes at least one or more of the following original or derived fields: (1) Fixation point coordinate data: including the x-coordinate of the fixation point after fusion of the left eye, right eye, or both eyes. y-axis With corresponding timestamp ; (2) Pupil data: including the diameter of the left pupil Right pupil diameter Average pupil diameter of both eyes and pupillary change rate; (3) Fixation event data: including fixation start time, fixation end time, fixation duration, and fixation center coordinates; (4) Scanning event data: including scan start point, scan end point, scan range, scan speed, and scan direction; (5) Blinking and eye state data: including blink start time, blink end time, blink frequency, and eye state markers; (6) AOI mapping data: including the region of interest number, AOI switching order, and number of back looks at each time point or each gaze event; (7) Validity and quality control data: including valid sampling markers, head movement compensation status, tracking confidence, and missing sample markers; (8) Event synchronization marker data: including event timestamps such as stimulus start, stimulus end, page switching, button trigger, questionnaire jump, etc.

[0040] To facilitate subsequent unified processing, the first... Eye-tracking sampling points Represented as:

[0041] in, Indicates the gaze coordinates, Indicates the diameter of the pupil. Indicates eye status or blink markers. Indicates a marker of sample quality or validity. This represents a unified timestamp.

[0042] Furthermore, after gaze recognition, the k-th gaze event can be identified. Represented as:

[0043] in, For the coordinates of the gaze center, The start and end times of this gaze event. For the duration of fixation, This is the corresponding area of ​​interest number.

[0044] In practice, the first modality data not only reflects "where it was seen", but also "how long it was viewed, where it was viewed first, whether it was viewed again, and whether it showed an increase in cognitive effort". Therefore, it can be used to measure the allocation of visual attention and as an objective representation of the intensity of cognitive processing of advertising.

[0045] Step S1.3: After the stimulus material is presented, the second modality data of the test subjects is obtained through the questionnaire collection terminal. The second modality data does not necessarily need to be extracted directly from an existing external database, but is preferably derived from the structured scale data filled out by the test subjects after viewing the stimulus material of the advertisement to be tested in this advertising evaluation task. The structured scale data includes psychological cognitive characteristic data collected based on the scale and individual difference characteristic data collected based on the scale.

[0046] In a preferred embodiment, an advertising evaluation scale item library is pre-established, which includes a psychological cognition scale item library, an individual difference scale item library, and a scenario-based item library. The psychological cognition scale item library is used to collect variables such as perceived usefulness, perceived entertainment value, advertising attitude, brand attitude, purchase intention, perceived interactivity, and social presence; the individual difference scale item library is used to collect variables such as product involvement, social cue sensitivity, price sensitivity, brand familiarity, advertising involvement, or media usage habits; the scenario-based item library is used to call corresponding items according to different advertising scenarios such as static graphic ads, AIGC ads, short video ads, and live promotional ads.

[0047] Furthermore, the advertising evaluation scale item bank can be updated based on historical advertising evaluation experiments, publicly available mature scales, expert-revised items, and anonymized historical evaluation samples accumulated by this system, forming a historical evaluation database. This historical evaluation database is used for scale item calibration, model parameter training, sample stratification benchmark comparison, and anomaly result identification. In specific evaluations, the second modality data primarily uses the real-time questionnaire completion results of the current test subjects, while the historical evaluation database is used as supplementary calibration data.

[0048] To improve the representativeness of the second modality data, test subjects can be stratified by age, gender, consumption experience, product usage experience, platform usage experience, or target consumer group attributes; and the reliability and validity of the second modality data can be controlled by item completeness checks, reverse item verification, reliability tests, combined reliability tests, and average variance extraction tests.

[0049] To ensure the structured and comparable nature of the second modality data, a built-in contextualized measurement question bank for different marketing scenarios is used to support multi-dimensional psychological and behavioral intention assessment. A 5-point or 7-point Likert scale is preferred for data collection; each latent variable is preferably configured with 3 to 6 items, and the corresponding subset of the scale is invoked based on the specific advertising scenario.

[0050] Specifically, the psychological cognitive characteristic data collected in this embodiment includes at least one or more of the following: Perceived usefulness: This measures whether advertising helps consumers understand products, compare options, and improve decision-making efficiency. Perceived entertainment: Used to measure whether an advertisement is interesting, enjoyable, and engaging. Advertising attitude: Used to measure the test subjects' overall liking, enjoyment, and positive evaluation of the advertisement; Brand attitude: Used to measure the overall evaluation of the brand involved in the advertisement by the test subjects; Purchase intention: Used to measure the test subject's tendency to try, search for, and purchase the product or service; Perceived interactivity: Applicable to live streaming or interactive advertising scenarios, used to measure the audience's perception of controllability, responsiveness, and a sense of two-way communication; Social presence: Applicable to live streaming or social advertising scenarios, used to measure whether the audience feels "someone is present" and whether they feel a real interactive atmosphere.

[0051] Furthermore, the collected individual difference characteristic data shall include at least one or more of the following: Product involvement: measures the relevance, importance, and level of attention that the advertised product has to a consumer's personal life; Interpersonal influence susceptibility or social cue sensitivity: measures whether consumers are more susceptible to the opinions of others, social proof, and group cues; Price sensitivity or promotional sensitivity: measures how sensitive consumers are to price changes, discounts, and limited-time offers. Brand awareness: Measures consumers' prior knowledge of the brand featured in an advertisement; Advertising involvement or media usage habits: This measures familiarity with relevant advertising formats.

[0052] Finally, the scores for each item on the scale are calculated as follows: Suppose a certain latent variable contains The first item, the first The score for each question is For reverse-selection questions, the scoring will be reversed first:

[0053] in, The highest score on the scale; for example, in a 7-point scale. .

[0054] The observed score of this latent variable can then be expressed as: in, For the first The scale scores of the latent variables.

[0055] If necessary, combined reliability (CR) and average variance extraction (AVE) can also be used to test the internal consistency and convergent validity of the scale.

[0056] Step S2: After preprocessing the first modality data and the second modality data, extract the first modality features and the second modality features, and construct a joint feature vector.

[0057] Step S2.1 involves cleaning, interpolating, filtering, and event recognition of the first modality data.

[0058] Eliminate Or sampling points that are outside the screen range, severely lost, or have insufficient tracking quality; perform linear interpolation or spline interpolation on short-term missing segments with a duration less than a preset threshold; use median filtering, moving average, or low-pass filtering to eliminate transient outliers in the pupil sequence; use I-VT or I-DT methods to segment gaze events, taking the velocity threshold method as an example, defining the gaze movement velocity between two adjacent sampling points as:

[0059] when And the duration is not less than When a gaze event occurs, the corresponding segment is identified as a fixation event; otherwise, it is identified as a saccade event. For speed threshold, This is the minimum gaze duration threshold.

[0060] Step S2.2 involves performing item completeness checks, outlier detection, reverse scoring, dimensional aggregation, and standardization on the scale data in the second modality. For missing items, mean imputation, regression imputation, or direct removal of samples with excessive missing items can be used. Z-score standardization can be performed on data of different scales.

[0061] in, The sample mean. This represents the sample standard deviation.

[0062] Step S2.3: Perform time and space alignment on the preprocessed data.

[0063] First, during the time alignment process, let the unified reference timestamp of the stimulus presentation system be... The local timestamp of the eye-tracking device is A linear time correction model can then be established using trigger signals, a unified clock, or event markers.

[0064] in, This is the clock scaling factor. This is the clock offset.

[0065] By minimizing the time error between multiple sets of synchronous trigger points, we can obtain The estimated value is used to map the first modality data to a unified time axis.

[0066] Furthermore, for each stimulus material Define its presentation window as:

[0067] Then, the set of eye-tracking samples of the test subject u within the s-th advertising stimulus presentation window can be extracted from a unified timeline. and the questionnaire results of the same test subject after the stimulus. Align the stimulus number with the test subject number.

[0068] During spatial alignment, eye-tracking sampling points or gaze events are mapped to corresponding regions of interest (AOIs). Let the m-th AOI in the s-th advertising stimulus be denoted as . For dynamic ads, video ads, or live streams, the region of interest can change over time, denoted as... Then the AOI attribution function of the i-th sampling point can be expressed as:

[0069] in, For the first AOI attribution function for each sampling point.

[0070] For static advertisements For dynamic ads, video ads, or live streams, dynamic interest regions can be defined using frame-level updates. This means that AOI can change over time.

[0071] Step S2.4: After time and spatial alignment are completed, extract the first modal features for advertising effectiveness analysis. These include: total fixation duration, number of fixations, first fixation latency, fixation duration percentage, number of regressions, mean pupil diameter, and pupil diameter change rate.

[0072] The total fixation time is:

[0073] in, This represents the total fixation duration on the m-th region of interest (AOI). represents the number of effective eye-tracking sampling points or effective fixation events within the presentation window of the s-th advertising stimulus material; i represents the ith effective eye-tracking sampling point or the ith fixation event on the unified time axis; This indicates the AOI number to which the i-th sampling point or gaze event is mapped; For the characteristic function, when The value is 1 if the condition is met, and 0 otherwise. It represents the sampling time interval corresponding to the i-th sampling point, or the duration of the i-th gaze event when calculating based on gaze events.

[0074] Number of fixations:

[0075] in, K represents the number of gaze events on the m-th area of ​​interest (AOI); K represents the total number of gaze events identified within the current advertising stimulus presentation window; k represents the gaze event number. This indicates the AOI number to which the k-th gaze event belongs; This is an indicator function, which takes the value 1 when the k-th gaze event falls into the m-th AOI, and takes the value 0 otherwise.

[0076] The initial gaze latency period is:

[0077] in, The time interval from the initial presentation of the advertising stimulus to the first fixation of the m-th AOI for the test subjects; This indicates the start time of the k-th gaze event on the unified timeline; This indicates that the k-th gaze event falls into the m-th AOI; This represents the earliest starting gaze time among all gaze events falling into the m-th AOI; This indicates the start time of the presentation of the s-th advertising stimulus. If the test subject does not fixate on the m-th AOI during stimulus presentation, then... Record it as a missing value, or set it as the upper limit of the presentation duration of the stimulus material.

[0078] The fixation percentage is:

[0079] in, This represents the percentage of fixation time for the m-th AOI; The m-th AOI represents the total fixation duration; M represents the total number of AOIs identified by the current advertising stimulus; r represents the AOI summation index. This represents the total fixation duration of the r-th AOI; This represents the total gaze duration of the test subject across all areas of interest (AOIs). It should be noted that... In this context, DT is an abbreviation for total fixation duration, and it is not a combination of independent D and D. Two parameters.

[0080] Number of replays:

[0081] in, is the number of re-views for the m-th AOI; K represents the total number of gaze events identified within the current advertising stimulus presentation window; k represents the current gaze event number; j represents the historical gaze event number that occurred before the current gaze event; This indicates the AOI number to which the k-th gaze event belongs; Indicates the AOI number to which the j-th historical gaze event belongs; This indicates that the test subject has gazed at the m-th AOI at least once before the k-th gaze event. Therefore, when the k-th gaze event falls on the m-th AOI again, this gaze is counted as a retrospective.

[0082] Mean pupil size and pupil variability rate:

[0083]

[0084] in, The average pupil diameter within the m-th AOI; Let be the pupil diameter of the i-th valid pupil sampling point; This represents the number of valid pupil sampling points mapped to the m-th AOI; This represents the AOI number corresponding to the i-th pupil sampling point. The pupil change rate within the m-th AOI; This represents the average pupil diameter of the test subject when it first enters the m-th AOI, or the average pupil diameter within the initial statistical window of the m-th AOI; This represents the average pupil diameter of the test subject before leaving the m-th AOI, or the average pupil diameter within the statistical window after the m-th AOI ends. and These represent the corresponding moments of the starting and ending statistical windows on a unified timeline. If... Then you can Record it as a missing value or set it to 0 to avoid the denominator being zero.

[0085] Furthermore, feature subsets can be extracted based on AOI type, such as product-related AOI features, price / promotion AOI features, anchor / person AOI features, non-product creative AOI features, and copywriting AOI features, thereby forming a more interpretable feature matrix.

[0086] The second modal feature includes the scores of each latent variable corresponding to the psychological cognitive feature data and the scores of each latent variable corresponding to the individual difference feature data. Specifically, the second modal feature is formed according to the scale dimensions:

[0087] in, This is the second modality feature vector; , , , , , , The psychological and cognitive characteristics are collectively referred to as These represent scores on scales for perceived usefulness, perceived entertainment, advertising attitude, brand attitude, purchase intention, perceived interactivity, and social presence, respectively. The individual differences in characteristics are collectively referred to as This is used to represent individual differences in variables such as product involvement, price sensitivity, social cue sensitivity, and brand familiarity. Therefore, the second modality feature vector can be represented as... Part C represents psychological and cognitive characteristics, and Part Z represents individual differences.

[0088] Therefore, for any test object and stimulating materials This can form a joint feature vector:

[0089] in, This represents the eye-tracking feature vector extracted from the first modality. This represents the feature vector of the second modality scale.

[0090] Step S3: Input the joint feature vector into a preset multi-path advertising effectiveness evaluation model, and output the advertising effectiveness evaluation result and strategy matching result. In this embodiment, the multi-path advertising effectiveness evaluation model does not perform single-path regression on the joint feature vector, but decomposes the joint feature vector into feature subsets corresponding to different advertising cognitive processing paths, and obtains the advertising effectiveness evaluation result through parallel path calculation, attitude fusion calculation, moderating effect calculation, and strategy matching calculation.

[0091] like Figure 2As shown, the multi-path advertising effectiveness evaluation model includes a feature grouping layer, a path variable calculation layer, an attitude fusion layer, an effect output and moderating effect calculation layer, and a strategy matching layer connected in sequence. After the joint feature vector is input into the model, the feature grouping layer first decomposes it into rational processing eye-tracking features, emotional processing eye-tracking features, overall attention features, psychological cognitive features, and individual difference features; then, the path variable calculation layer generates cognitive path variables and emotional experience path variables respectively; next, the attitude fusion layer generates advertising attitude or brand attitude variables; then, the effect output and moderating effect calculation layer generates comprehensive advertising effectiveness variables; finally, the strategy matching layer outputs advertising evaluation results and strategy matching results based on comprehensive advertising effectiveness variables, path strength, and AOI contribution.

[0092] Step S3.1, the multi-path advertising effectiveness evaluation model evaluates the joint feature vector based on AOI type and scale dimensions. Grouping:

[0093] In the formula, u represents the test subject number, and s represents the advertising stimulus material number; Represents the rational processing of eye-tracking feature vectors; Represents the sensory processing eye-movement feature vector; Indicates the overall characteristics of attention; Indicates psychological and cognitive characteristics; This represents individual differences. It mainly consists of the gaze characteristics of AOI, such as the product body area, brand logo area, price area, promotion area, and selling point area; It mainly consists of the gaze characteristics of AOI, such as the character area, the anchor area, the background area, the creative area, and the copywriting area; This is the overall attention feature obtained by aggregating the attention metrics of each key AOI.

[0094] Step S3.2: Calculate cognitive path variables based on rational processing eye-tracking characteristics and cognitive-related psychological characteristics; calculate emotional experience path variables based on emotional processing eye-tracking characteristics and emotion-related psychological characteristics. Cognitive path variables characterize the test subjects' understanding and perceived usefulness of product information, price information, and functional selling points in the advertisement; emotional experience path variables characterize the test subjects' perception of the entertainment value, attractiveness, and emotional impact of the characters, atmosphere, creativity, and interactive cues in the advertisement.

[0095] First, cognitive path variables are calculated based on characteristics related to rational processing:

[0096] in, Cognitive path variables related to functional understanding, information diagnostics, and perceived usefulness; This represents a sub-vector in psychological cognitive characteristics that is related to cognitive evaluation, such as perceived usefulness and product understanding. This is the cognitive path intercept term; Transpose of the weight coefficient matrix corresponding to rational processing of eye-tracking features; Transpose of the weight coefficient matrix corresponding to the features of the cognitive assessment scale This represents the residual term. Through this path, the model can obtain the path score of the advertisement in terms of "information comprehension and perceived usefulness" based on the test subjects' attention to AOIs such as product, price, and selling points, as well as the results of their corresponding cognitive evaluation scales.

[0097] Secondly, the emotional experience path variables are calculated based on the characteristics related to emotional processing:

[0098] in, These represent emotional experience path variables related to entertainment value, atmosphere perception, creative appeal, and social presence. This represents sub-vectors related to emotional experience in psychological and cognitive characteristics, such as perceived entertainment, perceived interactivity, and social presence. For the intercept term of the emotional experience path; This is the transpose of the weighting coefficient matrix corresponding to the sensory processing eye movement features; This is the transpose of the weight coefficient matrix corresponding to the features of the emotional experience scale; This is the residual term. Through this path, the model can obtain the path score of the advertisement in terms of "emotional appeal and creative experience" based on the test subject's attention allocation to AOIs such as people, anchors, backgrounds, creative ideas, and copywriting, as well as their corresponding emotional and interactive experience evaluations.

[0099] Step S3.3: Calculate advertising attitude or brand attitude variables based on cognitive path variables, emotional experience path variables, and overall attention characteristics. These variables are used to characterize the test subjects' overall evaluation of the advertisement or brand.

[0100] Specifically, cognitive path variables and emotional experience path variables are input into the attitude fusion path to calculate advertising attitude or brand attitude variables:

[0101] in, Variables representing advertising attitude or brand attitude; This represents a subvector in psychological cognitive characteristics that is related to advertising attitude or brand attitude. For the attitude integration path intercept term; This represents the coefficient of influence of cognitive path variables on advertising attitudes or brand attitudes. The coefficient representing the influence of emotional experience path variables on advertising attitude or brand attitude; This represents the transpose of the weight coefficient matrix corresponding to the overall attention features, used to measure the impact of overall visual attention allocation on attitude variables; This represents the transpose of the weight coefficient matrix corresponding to the features of the attitude scale, which is used to correct the attitude variable using subjective scale scores; This is the residual term.

[0102] Through this path, the model can determine whether advertising attitudes are primarily driven by rational information processing or by emotions, atmosphere, and creative experiences.

[0103] Step S3.4: Based on the cognitive path variables, emotional experience path variables, and attitude variables, calculate the purchase intention, conversion tendency, or overall advertising effectiveness output variables, as shown below:

[0104] in, Indicates purchase intention, conversion tendency, or overall advertising effectiveness; Output the path intercept term for the effect; This represents the influence coefficient of cognitive path variables on overall advertising effectiveness; The coefficient representing the influence of emotional experience path variables on the overall advertising effect; The coefficient representing the influence of advertising attitude or brand attitude variables on the overall advertising effect; This represents the main effect coefficient of the individual difference variable; This represents the moderating effect coefficient of individual difference variables on cognitive pathways; This represents the moderating effect coefficient of individual difference variables on the emotional experience path; This represents the interaction term between cognitive path variables and individual difference variables; This represents the interaction term between emotional experience path variables and individual difference variables; This is the residual term.

[0105] Furthermore, to measure the differences in the influence of cognitive and emotional experience paths on the overall advertising effect under different test subjects or different subgroups, individual differences were used as moderating variables. An interaction term between path variables and individual differences was introduced to calculate the differences in the influence of cognitive and emotional experience paths on the overall advertising effect under different test subjects or different subgroups.

[0106] For individual difference variables Introducing a moderating effect. Individual difference variables. It is not merely used as a regular input feature, but rather to characterize the differences in responses of different test subjects to the same advertising stimulus. Since individual differences such as product involvement, price sensitivity, social cue sensitivity, and brand familiarity can affect the conversion relationship between advertising attention, attitude, and purchase intention, this embodiment introduces individual difference variables as moderating variables into the model.

[0107] Specifically, firstly, regarding individual difference variables... Standardization was performed; secondly, interaction terms between cognitive path variables and individual difference variables were constructed. And the interaction terms between emotional experience path variables and individual difference variables. Then, the above interactive items are input into the effect output path to calculate the influence of cognitive path and emotional experience path on the overall advertising effect under different individual differences.

[0108] With a moderating term, the marginal effect of cognitive path variables on the overall advertising effect can be expressed as:

[0109] The marginal impact of emotional experience path variables on the overall advertising effect can be expressed as:

[0110] Therefore, when a certain test subject or a certain segment of the population... When the values ​​change, the strength of the influence of cognitive and emotional experience paths on the final advertising effect also changes. For example, for audiences with high product involvement, rational processing-related features such as product areas, price areas, and selling point areas may have a stronger impact on purchase intention; for audiences with high sensitivity to social cues, emotional processing-related features such as anchor areas, character areas, and interactive prompt areas may have a stronger impact on purchase intention; and for audiences with high price sensitivity, attention metrics in price areas, promotion areas, and countdown areas may have a stronger impact on the final advertising effect. Through the above adjustment calculations, the model can output the comprehensive advertising effect for different audience segments as the advertising effect evaluation result.

[0111] In a preferred embodiment, indirect effects, total effects, and standardized path coefficients can be further calculated to explain how cognitive path variables and emotional experience path variables influence the overall advertising effect through advertising attitude or brand attitude variables. For example:

[0112]

[0113]

[0114]

[0115] in, This indicates the indirect effect of cognitive path variables on overall advertising effectiveness via advertising attitude or brand attitude variables; This indicates the indirect effect of emotional experience path variables on overall advertising effectiveness through advertising attitude or brand attitude variables; This represents the overall effect of cognitive path variables on the overall advertising effect. This represents the overall effect of emotional experience path variables on the overall advertising effectiveness. Through comparison... and The size of the ad can indicate whether the advertising effect is primarily driven by rational information processing or by emotional experience and creative appeal.

[0116] Step S3.5: Based on the comprehensive evaluation score of the ad version, AOI contribution, path coefficient, indirect effect, total effect and moderating effect, generate strategy matching results for ad version, interest region and segmented audience.

[0117] Specifically, in the strategy matching phase, the system uses the cognitive path variables obtained in step S3.2 above. and emotional experience path variables The advertising attitude or brand attitude variables obtained in step S3.3 The comprehensive advertising effectiveness variables obtained in step S3.4 Instead of directly generating strategy conclusions based on a single formula, the system takes moderating effects and path strength as inputs. First, it aggregates results from multiple test subjects within the same ad version and the same demographic segment to calculate the overall ad version score. Second, it calculates the AOI contribution by combining standardized eye-tracking features corresponding to the AOI with model path coefficients. Third, it determines the dominant driving path of ad effectiveness based on the total effect of cognitive paths and the total effect of emotional experience paths. Finally, it generates strategy matching results tailored to ad version, interest region, and demographic segment by combining ad version score, AOI contribution, path strength, and demographic characteristics.

[0118] First, for each ad version v and each audience segment g, the system calculates the overall performance score for that ad version within that audience segment:

[0119] set up This represents the set of advertising stimuli belonging to the v-th ad version. This represents the set of test subjects belonging to the g-th segment. For any ad version v and any segment g, the system first checks if the following conditions are met. and Calculate the average of the samples:

[0120]

[0121]

[0122]

[0123]

[0124] in, This represents the overall score of the v-th ad version within the g-th audience segment; This represents the set of advertising stimuli belonging to the v-th advertising version; This represents the set of test subjects belonging to the g-th sub-group. , , , They represent respectively to and of , , , Calculate the average of the overall advertising effectiveness, advertising attitude, cognitive path, and emotional experience path. This represents the average percentage of views for the key AOI in the v-th ad version within the g-th audience segment; This represents the average first-look latency of the key AOI for the v-th ad version in the g-th audience segment; This is a preset weighting parameter. Since a shorter first fixation latency usually indicates earlier attention is attracted, this item uses a negative weight in the overall score.

[0125] Then, based on the comprehensive evaluation score, the preferred ad version for a specific audience segment is determined:

[0126] Where V represents the set of candidate ad versions, This represents the optimal ad version recommended for the g-th segment.

[0127] Furthermore, the contribution of each AOI to the overall advertising effect is calculated. For the m-th AOI, the system further calculates the contribution of each AOI to the overall advertising effect. For the m-th AOI, the v-th ad version, and the g-th segment, the contribution is calculated by weighting the standardized eye-tracking features and model path coefficients corresponding to that AOI:

[0128] in, This represents the contribution of the m-th AOI in the v-th ad version and the g-th segment. represents the set of eye-tracking features belonging to the m-th AOI; r represents the feature index in the AOI feature set; This represents the standardized coefficient of the r-th eye-tracking feature in the model path; This represents the standardized value of the r-th eye movement feature when the test subject u views the advertising stimulus material s; This represents the standardized mean of eye-tracking features obtained by aggregating the test subjects from the g subgroups and the advertising stimulus material under the v-th advertising version.

[0129] Finally, the strategy matching results are generated by combining the overall ad version score, AOI contribution, and path strength. For example, when the v-th ad version is used in the g-th segment... The highest, and the total effect of the cognitive path Greater than the total effect of the emotional experience path Meanwhile, the corresponding product area, price area, or selling point area When the overall effect of the emotional experience path is high, the system outputs ad versions that are more suitable for that segment of the audience, highlighting product features, price advantages, or core selling points; when the overall effect of the emotional experience path is high... Greater than the total effect of cognitive pathways And the corresponding character area, anchor area, background area or creative area When the score is high, the system outputs ad versions that are more suitable for that specific audience segment, emphasizing endorsements, interactive atmosphere, creative storytelling, or emotional appeal. Therefore, in live streaming advertising scenarios, this can be achieved by comparing price zones, host zones, and countdown zones. Identify the key areas that drive advertising effectiveness; in AIGC advertising scenarios, this can be achieved by comparing the product subject area and non-product creative areas. Determine whether the final advertising effect is primarily driven by product information or by creative expression.

[0130] Step S3.6: Generate an advertising evaluation report based on the calculation results of the above model. The advertising evaluation report shall include at least the overall advertising effectiveness score, attention index of key AOI, scores of each psychological cognitive variable, path coefficients of cognitive path and emotional experience path, indirect effect, total effect, moderating effect, comparison results of different advertising versions or different segmented groups, as well as creative optimization suggestions and precise targeting suggestions.

[0131] Furthermore, to facilitate the demonstration of the analytical method for the contribution of this invention to the internal elements of an advertisement, an AOI (Area of ​​Interest) division and eye-tracking heatmap diagram of the advertising stimulus material can be generated. For example... Figure 3 As shown, the advertisement screen is divided into multiple interest areas, and the gaze distribution of the test subjects is overlaid as heat zones. This diagram visually demonstrates the differences in attention allocation among different test subjects or different demographic segments to elements within the advertisement.

[0132] Furthermore, through six sets of comparative experiments covering two core scenarios—AIGC advertising and live-streaming promotional advertising—the scientific validity and practicality of the method of this invention were fully verified from three core dimensions: the effectiveness of eye-tracking features, the rationality of the path mechanism, and the stability of scenario adaptation.

[0133] In the three experiments of AIGC advertising scenarios, the fixation duration (2.36±0.37), fixation count (7.25±0.96), and fixation percentage (30.26±4.56%) of relevant AIGC ads in the product area were significantly higher than those of divergent ads (1.21±0.31, 3.83±1.39, and 15.64±5.38%), respectively. This confirms that eye-tracking features of product-related AOIs can effectively characterize consumers' cognitive processing paths. Simultaneously, the attention concentration areas of the two types of ads showed significant differentiation. Users of divergent ads focused more on non-product creative areas, while relevant ads showed higher fixation density in the product area. This verifies the ability of non-product creative AOI eye-tracking features to characterize emotional experience paths. In mixed advertising scenarios, the p-values ​​of AOI fixation indicators for both relevant and divergent ads were greater than 0.05 in the mixed group and the single-type group, showing no statistically significant difference. This confirms that the model has stable evaluation capabilities in mixed scenarios with multiple ad versions. In two sets of experiments on live-stream promotional advertising scenarios, for limited-time promotions, users' attention to the countdown area accounted for 6.00% and the number of glances accounted for 6.32%, while for limited-quantity promotions, users' attention to the inventory bar accounted for 4.88% and the number of glances accounted for 5.22%. The core attention AOIs of the two types of promotions showed significant differences, validating the rationality of the model's AOI contribution breakdown. Simultaneously, the collective sense of synchronicity (4.38±0.95) in the limited-time group was significantly higher than that in the limited-quantity group (3.86±0.99), and the sense of interactive control (4.37±0.98) in the limited-quantity group was significantly higher than that in the limited-time group (3.86±1.02), both reaching highly significant levels. This confirms that the advertising effect formation of different promotion types has differentiated core paths, supporting the design logic of the model's multi-path variable breakdown. The social cue sensitivity moderating effect β=0.850 also reached a highly significant level, confirming that the formation of advertising effect is the result of multi-path chain mediation, and that individual differences have a significant moderating effect. This provides core empirical support for the model's path coefficient calculation and segmented audience strategy matching.

[0134] Example 2 This embodiment discloses an advertising effectiveness evaluation system based on multimodal perception; like Figure 4 As shown, the advertising effectiveness evaluation system based on multimodal perception includes: The data acquisition module is configured to: collect the first modality data and the second modality data of the test subjects based on the advertising stimulus material to be tested; The feature extraction module is configured to: preprocess the first modality data and the second modality data, extract the first modality features and the second modality features, and construct a joint feature vector; The evaluation result output module is configured to: input the joint feature vector into a preset multi-path advertising effect evaluation model, and output advertising effect evaluation results and strategy matching results; The multi-path advertising effectiveness evaluation model decomposes the joint feature vector into rational processing eye movement features, emotional processing eye movement features, overall attention features, psychological cognitive features, and individual difference features. Cognitive path variables were calculated based on rational processing eye movement characteristics and cognitive-related psychological characteristics, and emotional experience path variables were calculated based on emotional processing eye movement characteristics and emotional-related psychological characteristics. By integrating cognitive path variables, emotional experience path variables, and overall attention characteristics, advertising attitude or brand attitude variables can be calculated. Based on cognitive path variables, emotional experience path variables, and attitude variables, and incorporating individual differences, the comprehensive advertising effect is calculated as the advertising effect evaluation result. After obtaining comprehensive advertising effectiveness, cognitive path variables, emotional experience path variables, advertising attitude or brand attitude variables, and individual difference moderating effects, the evaluation result output module further aggregates multiple sample results under the same advertising version and the same segmented audience, calculates the comprehensive score of the advertising version, AOI contribution, and path strength, and determines the preferred advertising version under different segmented audiences based on the comprehensive score of the advertising version, determines the interest regions that contribute more to the advertising effectiveness based on the AOI contribution, and determines whether the advertising effectiveness is mainly driven by the cognitive path or the emotional experience path based on the path strength, thereby outputting the advertising effectiveness evaluation results and the strategy matching results for advertising version, interest region, and segmented audience.

[0135] Example 3 The purpose of this embodiment is to provide a computer-readable storage medium.

[0136] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the advertising effectiveness evaluation method based on multimodal perception as described in Example 1.

[0137] Example 4 The purpose of this embodiment is to provide an electronic device.

[0138] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the advertising effectiveness evaluation method based on multimodal perception as described in Embodiment 1.

[0139] The steps and methods involved in the apparatuses of Embodiments 2, 3, and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0140] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0141] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for evaluating advertising effectiveness based on multimodal perception, characterized in that, include: Based on the advertising stimulus material to be tested, first-modal data and second-modal data of the test subjects were collected; After preprocessing the first modality data and the second modality data, the first modality features and the second modality features are extracted, and a joint feature vector is constructed. The joint feature vector is input into a preset multi-path advertising effectiveness evaluation model, and the advertising effectiveness evaluation results and strategy matching results are output. The multi-path advertising effectiveness evaluation model decomposes the joint feature vector into rational processing eye movement features, emotional processing eye movement features, overall attention features, psychological cognitive features, and individual difference features. Cognitive path variables were calculated based on rational processing eye movement characteristics and cognitive-related psychological characteristics, and emotional experience path variables were calculated based on emotional processing eye movement characteristics and emotional-related psychological characteristics. By integrating cognitive path variables, emotional experience path variables, and overall attention characteristics, advertising attitude or brand attitude variables can be calculated. Based on cognitive path variables, emotional experience path variables, and attitude variables, and incorporating individual differences, the comprehensive advertising effect is calculated as the advertising effect evaluation result. Calculate the overall score of the ad version, AOI contribution, and path strength. Based on the calculation results, output the ad performance evaluation results and the strategy matching results for ad versions, interest regions, and segmented audiences.

2. The advertising effectiveness evaluation method based on multimodal perception as described in claim 1, characterized in that, The advertising stimulus material to be tested has at least one area of ​​interest (AOI) predefined.

3. The advertising effectiveness evaluation method based on multimodal perception as described in claim 1, characterized in that, The first modal data is eye movement physiological characteristic data, including at least fixation point coordinate data, pupil data, fixation event data, saccade event data, blink and eye state data, AOI mapping data, validity and quality control data, and event synchronization marker data; The second modality data consists of psychological and cognitive characteristic data and individual difference characteristic data collected based on the scale.

4. The advertising effectiveness evaluation method based on multimodal perception as described in claim 1, characterized in that, Preprocessing of the first and second modal data includes: The system performs invalid sample removal, missing segment interpolation, pupil data smoothing, and identification of fixation and saccade events on the first modality data. The second modality data is subjected to item completeness verification, reverse scoring, dimensional aggregation, and standardization. The preprocessed first modal data and second modal data are mapped to a unified time axis for time synchronization. The coordinates of the sampling points of the first modal data are matched with the boundaries of the regions of interest to determine the regions of interest to which each sampling point belongs, thus achieving spatial alignment.

5. The advertising effectiveness evaluation method based on multimodal perception as described in claim 1, characterized in that, Based on the preprocessed data, first-modality features and second-modality features are extracted, and a joint feature vector is constructed, including: Based on the data after time synchronization processing and spatial mapping, the first modal features are extracted according to the region of interest. The first modal features include at least one of the following: total fixation duration, number of fixations, initial fixation latency, fixation duration percentage, number of retrospectives, mean pupil diameter, and pupil diameter change rate. Based on the preprocessed second modality data, second modality features are extracted. The second modality features include the scores of each latent variable corresponding to the psychological cognitive feature data and the scores of each latent variable corresponding to the individual difference feature data. The first modal features are concatenated with the second modal features to construct a joint feature vector for each test subject and each stimulus material.

6. The advertising effectiveness evaluation method based on multimodal perception as described in claim 1, characterized in that, The cognitive path variables are: in, Represents cognitive path variables; This represents a subvector related to cognitive appraisal within psychological cognitive characteristics; To rationally process eye-tracking feature vectors; This is the cognitive path intercept term; Transpose of the weight coefficient matrix corresponding to rational processing of eye-tracking features; This is the transpose of the weight coefficient matrix corresponding to the features of the cognitive assessment scale. For residual terms; The emotional experience path variables are: in, Represents variables related to the path of emotional experience; Represents the sensory processing eye-movement feature vector; This represents a subvector in psychological cognitive characteristics that is related to emotional experience. For the intercept term of the emotional experience path; This is the transpose of the weighting coefficient matrix corresponding to the sensory processing eye movement features; This is the transpose of the weight coefficient matrix corresponding to the features of the emotional experience scale; This is the residual term.

7. The advertising effectiveness evaluation method based on multimodal perception as described in claim 1, characterized in that, The advertising attitude or brand attitude variables are: in, Variables representing advertising attitude or brand attitude; For the attitude integration path intercept term; This represents the coefficient of influence of cognitive path variables on advertising attitudes or brand attitudes. Represents cognitive path variables; The coefficient representing the influence of emotional experience path variables on advertising attitude or brand attitude; Represents variables related to the path of emotional experience; This represents the transpose of the weight coefficient matrix corresponding to the overall attention features, used to measure the impact of overall visual attention allocation on attitude variables; Indicates the overall characteristics of attention; This represents the transpose of the weight coefficient matrix corresponding to the features of the attitude scale, which is used to correct the attitude variable using subjective scale scores; This represents a subvector in psychological cognitive characteristics that is related to advertising attitude or brand attitude. This is the residual term.

8. An advertising effectiveness evaluation system based on multimodal perception, characterized in that, include: The data acquisition module is configured to: collect the first modality data and the second modality data of the test subjects based on the advertising stimulus material to be tested; The feature extraction module is configured to: preprocess the first modality data and the second modality data, extract the first modality features and the second modality features, and construct a joint feature vector; The evaluation result output module is configured to: input the joint feature vector into a preset multi-path advertising effect evaluation model, and output advertising effect evaluation results and strategy matching results; The multi-path advertising effectiveness evaluation model decomposes the joint feature vector into rational processing eye movement features, emotional processing eye movement features, overall attention features, psychological cognitive features, and individual difference features. Cognitive path variables were calculated based on rational processing eye movement characteristics and cognitive-related psychological characteristics, and emotional experience path variables were calculated based on emotional processing eye movement characteristics and emotional-related psychological characteristics. By integrating cognitive path variables, emotional experience path variables, and overall attention characteristics, advertising attitude or brand attitude variables can be calculated. Based on cognitive path variables, emotional experience path variables, and attitude variables, and incorporating individual differences, the comprehensive advertising effect is calculated as the advertising effect evaluation result. Calculate the overall score of the ad version, AOI contribution, and path strength. Based on the calculation results, output the ad performance evaluation results and the strategy matching results for ad versions, interest regions, and segmented audiences.

9. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the advertising effectiveness evaluation method based on multimodal perception as described in any one of claims 1-7.

10. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the advertising effectiveness evaluation method based on multimodal perception as described in any one of claims 1-7.

Images