Marketing video effect evaluation method combining eye tracking and emotion recognition

Abstract

The application provides a marketing video effect evaluation method combining eye movement tracking and emotion recognition, and relates to the technical field of marketing effect evaluation, and is characterized in that the method comprises the following steps: multi-modal data synchronous acquisition, eye movement data processing and visual attention area extraction, facial expression recognition and emotion state analysis, multi-modal data space-time alignment and feature fusion, multi-dimensional evaluation index calculation, video segment level effect evaluation, comprehensive effect evaluation, generation of an evaluation report and optimization suggestions.The application has the following advantages: multi-dimensional evaluation indexes are constructed, the marketing video effect can be more objectively, comprehensively and accurately evaluated, multi-dimensional, fine and objective evaluation of the marketing video effect is realized, targeted and personalized optimization suggestions are given and an effect improvement value is predicted, and a scientific basis is provided for optimization and iteration of the marketing video.

Description

Technical Field

[0001] This invention relates to the field of marketing effectiveness evaluation technology, and in particular to a method for evaluating the effectiveness of marketing videos that combines eye tracking and emotion recognition. Background Technology

[0002] With the rapid development of marketing formats such as short videos and live streaming, marketing videos have become a core medium for enterprises to convey brand information and promote product conversion, and their effectiveness directly affects the success or failure of marketing activities. Traditional methods for evaluating the effectiveness of marketing videos mainly rely on data such as subjective questionnaires, click-through rates, and completion rates. These methods not only have limited sample sizes and poor timeliness, but also suffer from memory bias and social expectation bias, making it difficult to capture the real, instantaneous psychological activities of viewers during the viewing process.

[0003] Eye-tracking technology can accurately capture data such as viewer gaze points, gaze duration, and saccade amplitude, reflecting the viewer's visual attention patterns to video content. Emotion recognition technology, by analyzing facial expressions, can quantify the viewer's emotional state and reveal their emotional response to the video content. Combining the two allows for multi-dimensional evaluation of marketing video effectiveness, including visual attention, emotional response, and cognitive processing, overcoming the shortcomings of traditional evaluation methods. However, current technologies suffer from poor synchronization between eye-tracking and emotion data, insufficient feature fusion, and limited evaluation metrics, failing to achieve precise evaluation at the video segment level. Furthermore, they lack feasible implementation plans, making it difficult to meet the needs of enterprises for refined evaluation of marketing video effectiveness.

[0004] Therefore, there is an urgent need for a marketing video effectiveness evaluation method that can achieve simultaneous collection and fusion of multimodal data, accurate multi-dimensional evaluation, and is operable, in order to solve the above-mentioned technical problems. Summary of the Invention

[0005] The purpose of this invention is to provide a marketing video effectiveness evaluation method that combines eye tracking and emotion recognition. This method solves the problems of strong subjectivity, single modality, lack of depth, and asynchronous data in the existing marketing video effectiveness evaluation. It enables multi-dimensional, refined, and objective evaluation of marketing video effectiveness, providing accurate basis for video content optimization.

[0006] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows:

[0007] A marketing video effectiveness evaluation method combining eye tracking and emotion recognition includes the following steps:

[0008] S1. Multimodal data synchronous acquisition: Play the marketing video to be evaluated, and simultaneously acquire the target audience's eye movement data and facial video data based on the frame timestamp; record the timestamps of the eye movement data and facial video data to ensure that the timelines of the two types of data are aligned;

[0009] S2. Eye-tracking data processing and visual attention region extraction: Based on video frames, eye-tracking and emotion data are aligned, and misaligned timestamp data is removed; eye-tracking data is preprocessed; visual attention regions are extracted for each frame of the video based on fixation point coordinates and fixation duration; visual attention regions are mapped to the spatial coordinate system of the video screen to generate a dynamic visual heatmap.

[0010] The method for extracting visual attention regions is as follows: set a gaze duration threshold of 0.1-0.3s, cluster gaze points whose gaze duration exceeds the threshold to form visual attention regions; the generation rule of dynamic visual heatmap is: the higher the gaze density and the longer the gaze duration, the darker the heatmap color, where gaze density is the number of gaze points per unit area.

[0011] S3. Facial Expression Recognition and Emotional State Analysis: Frame-level analysis is performed on facial video data to extract facial motion units; based on the combination of facial motion units, the basic emotion category and its confidence level of each frame are identified; the basic emotion categories include joy, sadness, surprise, fear, anger, disgust, and neutrality;

[0012] Facial action unit extraction employs a deep learning-based facial keypoint detection algorithm, extracting 68 facial keypoints. Facial action units are identified based on the displacement changes of these keypoints. Emotion recognition uses a convolutional neural network model, inputting combined features of facial action units and outputting the confidence scores of each basic emotion category. The confidence score threshold is set to 0.5. When the confidence score of a certain emotion category exceeds the threshold, the emotion corresponding to that frame is determined to be that category.

[0013] S4. Spatiotemporal Alignment and Feature Fusion of Multimodal Data: Eye-tracking data and facial expression data are precisely aligned according to timestamps to construct a spatiotemporally synchronized multimodal feature sequence; for each frame of the video, the following features are extracted:

[0014] Eye movement characteristics: fixation point coordinates, fixation duration, saccade amplitude, and pupil diameter changes;

[0015] Emotional features: emotion category probability vector, emotion intensity, facial action unit activation value;

[0016] Interactive features: emotional response values ​​corresponding to visual attention areas, and the temporal relationship between emotional changes and gaze shifts;

[0017] Multimodal data spatiotemporal alignment uses a timestamp interpolation matching method. When there is a discrepancy between the timestamps of eye-tracking data and facial video data, the eye-tracking data is linearly interpolated based on the video frame timestamp, so that each video frame corresponds to a set of eye-tracking data and a set of emotion data. Feature fusion uses an attention mechanism, assigning different weights to eye-tracking features, emotion features, and interaction features. The weights are determined through iterative optimization using training samples.

[0018] S5. Calculation of Multidimensional Evaluation Indicators: Based on the fused multimodal features, the following evaluation indicators are calculated:

[0019] Visual attention intensity: Based on fixation density and fixation duration, it reflects the degree of visual engagement of the viewer with the video content; visual attention intensity Where T is the total number of video frames, and gaze density is the ratio of the number of gaze points in the current frame to the screen area;

[0020] Emotional Resonance Index: Based on the match between the emotional category and the expected emotion of the video, as well as the intensity of the emotion, it reflects the degree of emotional identification among viewers; Emotional Resonance Index Where C is the set of emotion categories, and the matching degree is the cosine similarity between the current emotion and the expected emotion;

[0021] Cognitive processing depth: Based on changes in pupil diameter and fixation duration, this reflects the viewer's cognitive processing depth of the video content; ,in , These are the weighting coefficients;

[0022] Emotion and attention coherence: Based on the correlation between fixation shift and emotion change, it reflects the consistency between the viewer's emotional experience and visual attention; Emotion and attention coherence = Pearson correlation coefficient (fixation shift sequence, emotion change sequence).

[0023] False Expression Detection Coefficient: Based on the abnormal matching degree between facial expressions and eye movement patterns, it identifies potentially socially desired expressions; False Expression Detection Coefficient Among them, the abnormal matching degree is the confidence level when the emotion category is "neutral" but the pupil diameter changes significantly;

[0024] S6. Video Segment-Level Performance Evaluation: Divide the marketing video into multiple evaluation units based on scenes or shots; for each evaluation unit, calculate the above indicators and generate the performance evaluation vector for that unit.

[0025] The video evaluation unit is divided according to the following rules: video shot switching is used as the dividing node, the duration of a single evaluation unit is 3-10 seconds, and shots less than 3 seconds are merged with adjacent shots into one evaluation unit; the effect evaluation vector of each evaluation unit is [visual attention intensity, emotional resonance index, cognitive processing depth, emotion and attention coordination degree, false expression detection coefficient];

[0026] S7. Overall Effect Evaluation: Based on the indicator vectors of each evaluation unit, a weighted fusion method is used to calculate the overall video effect score; Overall Effect Score Where n is the number of indicators. Preset weights (e.g., visual attention intensity weight 0.3, emotional resonance index weight 0.4, cognitive processing depth weight 0.2, and emotion and attention synergy weight 0.1).

[0027] The weights of the weighted fusion method are determined by the analytic hierarchy process (AHP). By constructing a judgment matrix, the weights of each evaluation index are calculated, and the sum of the weights is 1. The overall video effect score is the average of the weighted sum of the effect evaluation vectors of each evaluation unit and their corresponding weights.

[0028] S8. Generate evaluation report and optimization suggestions: Generate video effect evaluation report, including time change curves of various indicators, visual heatmap overlay of key frames, emotion trajectory map, and provide content optimization suggestions for weak links;

[0029] The criteria for identifying weaknesses are: a certain evaluation indicator is lower than a preset threshold, which is determined through training with a large number of samples; optimization suggestions include video content adjustment, visual element optimization, and emotional guidance optimization, specifically: for units with low visual attention intensity, it is recommended to add visual focus elements; for units with low emotional resonance index, it is recommended to adjust the emotional expression of the content to match the target audience; for units with high false expression identification coefficient, it is recommended to optimize the authenticity of the content and reduce guiding hints.

[0030] Furthermore, the eye-tracking data preprocessing in step S2 includes the following sub-steps:

[0031] S21. Use median filtering algorithm to remove high-frequency noise from eye-tracking data;

[0032] S22. Identify blinking actions based on a blink detection algorithm and remove eye movement data during blinking;

[0033] S23. Detect head movement using a head pose estimation algorithm. When the head pose fluctuation exceeds a preset threshold, remove the eye movement data for the corresponding time period.

[0034] S24. Linear interpolation is used to supplement the missing data after removing invalid data to ensure the continuity of eye-tracking data.

[0035] The beneficial effects of this invention are as follows:

[0036] 1. Based on frame timestamps, it achieves precise synchronization between eye-tracking data and facial video data. By integrating eye-tracking, emotion, and interaction features through an attention mechanism, it overcomes the problems of data asynchrony and insufficient feature fusion in existing technologies, and can more comprehensively reflect the viewer's viewing experience.

[0037] 2. Several core evaluation indicators, such as visual attention intensity and emotional resonance index, have been constructed to quantify the effectiveness of marketing videos from five dimensions: visual attention, emotional response, cognitive processing, synergy between emotion and attention, and identification of false expressions, thus making up for the shortcomings of traditional evaluation methods with their single indicators.

[0038] 3. Enables video segment-level evaluation, accurately identifying the strengths and weaknesses of each segment, providing precise basis for video content optimization, and avoiding blind optimization.

[0039] 4. Based on objective eye-tracking and facial expression data, avoiding interference from subjective factors, the evaluation method and steps are clear, providing specific parameter settings and calculation methods, which can be directly implemented and are suitable for evaluating the effectiveness of various marketing videos. Attached Figure Description

[0040] Figure 1 This is an overall flowchart of the marketing video effectiveness evaluation method that combines eye tracking and emotion recognition according to the present invention.

[0041] Figure 2 This is a partial flowchart of the marketing video effectiveness evaluation method combining eye tracking and emotion recognition of the present invention. Detailed Implementation

[0042] To make the content of this invention easier to understand, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Identical components are represented by the same reference numerals. It should be noted that the terms "front," "rear," "left," "right," "up," and "down" used in the following description refer to directions in the accompanying drawings, while the terms "inner" and "outer" refer to directions toward or away from the geometric center of a specific component, respectively.

[0043] like Figure 1 As shown, it includes the following steps:

[0044] S1. Multimodal data synchronous acquisition: Play the marketing video to be evaluated, and simultaneously acquire the target audience's eye movement data and facial video data based on the frame timestamp; record the timestamps of the eye movement data and facial video data to ensure that the timelines of the two types of data are aligned;

[0045] S2. Eye-tracking data processing and visual attention region extraction: Based on video frames, eye-tracking and emotion data are aligned, and misaligned timestamp data is removed; eye-tracking data is preprocessed; visual attention regions are extracted for each frame of the video based on fixation point coordinates and fixation duration; visual attention regions are mapped to the spatial coordinate system of the video screen to generate a dynamic visual heatmap.

[0046] The method for extracting visual attention regions is as follows: set a gaze duration threshold of 0.1-0.3s, cluster gaze points whose gaze duration exceeds the threshold to form visual attention regions; the generation rule of dynamic visual heatmap is: the higher the gaze density and the longer the gaze duration, the darker the heatmap color, where gaze density is the number of gaze points per unit area.

[0047] S3. Facial Expression Recognition and Emotional State Analysis: Frame-level analysis is performed on facial video data to extract facial motion units; based on the combination of facial motion units, the basic emotion category and its confidence level of each frame are identified; the basic emotion categories include joy, sadness, surprise, fear, anger, disgust, and neutrality;

[0048] Facial action unit extraction employs a deep learning-based facial keypoint detection algorithm, extracting 68 facial keypoints. Facial action units are identified based on the displacement changes of these keypoints. Emotion recognition uses a convolutional neural network model, inputting combined features of facial action units and outputting the confidence scores of each basic emotion category. The confidence score threshold is set to 0.5. When the confidence score of a certain emotion category exceeds the threshold, the emotion corresponding to that frame is determined to be that category.

[0049] S4. Spatiotemporal Alignment and Feature Fusion of Multimodal Data: Eye-tracking data and facial expression data are precisely aligned according to timestamps to construct a spatiotemporally synchronized multimodal feature sequence; for each frame of the video, the following features are extracted:

[0050] Eye movement characteristics: fixation point coordinates, fixation duration, saccade amplitude, and pupil diameter changes;

[0051] Emotional features: emotion category probability vector, emotion intensity, facial action unit activation value;

[0052] Interactive features: emotional response values ​​corresponding to visual attention areas, and the temporal relationship between emotional changes and gaze shifts;

[0053] Multimodal data spatiotemporal alignment uses a timestamp interpolation matching method. When there is a discrepancy between the timestamps of eye-tracking data and facial video data, the eye-tracking data is linearly interpolated based on the video frame timestamp, so that each video frame corresponds to a set of eye-tracking data and a set of emotion data. Feature fusion uses an attention mechanism, assigning different weights to eye-tracking features, emotion features, and interaction features. The weights are determined through iterative optimization using training samples.

[0054] S5. Calculation of Multidimensional Evaluation Indicators: Based on the fused multimodal features, the following evaluation indicators are calculated:

[0055] Visual attention intensity: Based on fixation density and fixation duration, it reflects the degree of visual engagement of the viewer with the video content; visual attention intensity Where T is the total number of video frames, and gaze density is the ratio of the number of gaze points in the current frame to the screen area;

[0056] Emotional Resonance Index: Based on the match between the emotional category and the expected emotion of the video, as well as the intensity of the emotion, it reflects the degree of emotional identification among viewers; Emotional Resonance Index Where C is the set of emotion categories, and the matching degree is the cosine similarity between the current emotion and the expected emotion;

[0057] Cognitive processing depth: Based on changes in pupil diameter and fixation duration, this reflects the viewer's cognitive processing depth of the video content; ,in , These are the weighting coefficients;

[0058] Emotion and attention coherence: Based on the correlation between fixation shift and emotion change, it reflects the consistency between the viewer's emotional experience and visual attention; Emotion and attention coherence = Pearson correlation coefficient (fixation shift sequence, emotion change sequence).

[0059] False Expression Detection Coefficient: Based on the abnormal matching degree between facial expressions and eye movement patterns, it identifies potentially socially desired expressions; False Expression Detection Coefficient Among them, the abnormal matching degree is the confidence level when the emotion category is "neutral" but the pupil diameter changes significantly;

[0060] S6. Video Segment-Level Performance Evaluation: Divide the marketing video into multiple evaluation units based on scenes or shots; for each evaluation unit, calculate the above indicators and generate the performance evaluation vector for that unit.

[0061] The video evaluation unit is divided according to the following rules: video shot switching is used as the dividing node, the duration of a single evaluation unit is 3-10 seconds, and shots less than 3 seconds are merged with adjacent shots into one evaluation unit; the effect evaluation vector of each evaluation unit is [visual attention intensity, emotional resonance index, cognitive processing depth, emotion and attention coordination degree, false expression detection coefficient];

[0062] S7. Overall Effect Evaluation: Based on the indicator vectors of each evaluation unit, a weighted fusion method is used to calculate the overall video effect score; Overall Effect Score Where n is the number of indicators. Preset weights (e.g., visual attention intensity weight 0.3, emotional resonance index weight 0.4, cognitive processing depth weight 0.2, and emotion and attention synergy weight 0.1).

[0063] The weights of the weighted fusion method are determined by the analytic hierarchy process (AHP). By constructing a judgment matrix, the weights of each evaluation index are calculated, and the sum of the weights is 1. The overall video effect score is the average of the weighted sum of the effect evaluation vectors of each evaluation unit and their corresponding weights.

[0064] S8. Generate evaluation report and optimization suggestions: Generate video effect evaluation report, including time change curves of various indicators, visual heatmap overlay of key frames, emotion trajectory map, and provide content optimization suggestions for weak links;

[0065] The criteria for identifying weaknesses are: a certain evaluation indicator is lower than a preset threshold, which is determined through training with a large number of samples; optimization suggestions include video content adjustment, visual element optimization, and emotional guidance optimization, specifically: for units with low visual attention intensity, it is recommended to add visual focus elements; for units with low emotional resonance index, it is recommended to adjust the emotional expression of the content to match the target audience; for units with high false expression identification coefficient, it is recommended to optimize the authenticity of the content and reduce guiding hints.

[0066] like Figure 2 As shown, eye-tracking data preprocessing in step S2 includes the following sub-steps:

[0067] S21. Use median filtering algorithm to remove high-frequency noise from eye-tracking data;

[0068] S22. Identify blinking actions based on a blink detection algorithm and remove eye movement data during blinking;

[0069] S23. Detect head movement using a head pose estimation algorithm. When the head pose fluctuation exceeds a preset threshold, remove the eye movement data for the corresponding time period.

[0070] S24. Linear interpolation is used to supplement the missing data after removing invalid data to ensure the continuity of eye-tracking data.

[0071] Example 1: Marketing video to be evaluated: A foundation product sales promotion video, 60 seconds long, 30fps, 1080P resolution. The video content is divided into four core segments: product display, face application test, makeup lasting test, and discount explanation. It belongs to the category of sales promotion marketing videos.

[0072] Target audience: 50 women aged 20-35, including 38 who have experience using foundation and 12 who have no experience using it, divided into three groups: 20-25 years old, 26-30 years old, and 31-35 years old.

[0073] Data acquisition equipment: The eye tracker used is TobiiProX3-120 (sampling rate 120Hz, gaze point acquisition error ≤0.3° angle of view), the facial video acquisition uses a high-definition infrared camera (resolution 1080P, frame rate 30fps), and the environmental parameter acquisition uses a temperature, humidity and light meter and a noise meter.

[0074] Experimental environment: light intensity 500 lux, ambient noise ≤30dB, viewing distance 60cm, screen size 27 inches, ensuring consistency of the experimental environment.

[0075] Step S1: Synchronous acquisition of multimodal data and recording of basic information

[0076] Nine-point eye tracker calibration was performed on 50 target viewers to ensure that the fixation point acquisition error was ≤0.3° of visual angle;

[0077] Simultaneously play the marketing video to be evaluated, collect eye movement data such as the viewer's gaze point coordinates, gaze duration, saccade amplitude, pupil diameter change, and blink frequency through an eye tracker, collect facial video data of the viewer through a high-definition infrared camera, collect environmental parameters such as light, noise, and viewing distance through environmental instruments, and record basic information such as the viewer's age, gender, occupation, and foundation usage experience.

[0078] Based on the video frame timestamp, a unified timestamp is added to all types of data. After initial calibration, the time axis deviation between eye-tracking data and facial video data is ensured to be ≤0.01s.

[0079] Step S2: Refined processing of eye-tracking data and extraction of dynamic visual attention regions

[0080] Secondary timestamp verification: Invalid data with timestamp misalignment exceeding 0.01 seconds in eye-tracking and facial video data were removed, leaving 98.5% of the data as valid data.

[0081] Multi-dimensional preprocessing of eye-tracking data:

[0082] S21: A 3×3 window median filtering algorithm is used to remove high-frequency noise from eye-tracking data;

[0083] S22: Identify blinking actions and remove eye movement data during blinking; the average blinking frequency of the audience is calculated to be 15 times / minute.

[0084] S23: Detect head movement and remove eye movement data with head posture fluctuations exceeding ±15°, accounting for 0.8%;

[0085] S24: Outliers in the eye-tracking data were detected and removed using the Isolation Forest algorithm, accounting for 0.5%;

[0086] S25: Linear interpolation is used to supplement missing data to ensure the continuity of eye-tracking data;

[0087] S26: Dynamically set the gaze duration threshold based on the complexity of the video content. For product demonstrations / makeup test segments with complex visual elements, set the threshold to 0.15s. For on-face trial / promotion explanation segments with simple visual elements, set the threshold to 0.3s.

[0088] Visual attention region extraction: The DBSCAN clustering algorithm is used to cluster gaze points whose gaze duration exceeds the dynamic threshold and extract the visual attention region of each frame. For example, in the product demonstration, the audience's visual attention region is mainly concentrated on the foundation bottle and the color number display area.

[0089] Dynamic visual heatmap generation: The visual attention area is mapped to the spatial coordinate system of the video screen, and a dynamic visual heatmap with a time dimension is generated with gaze density and gaze duration as dual weights (0.5 each). The heatmap shows that the gaze density is the highest during the product display and the lowest during the discount explanation.

[0090] Step S3: Full-dimensional facial expression recognition and refined analysis of emotional state

[0091] Perform two-layer micro-expression analysis on facial video data at the frame level (33ms / frame) and the millisecond level (10ms / frame);

[0092] Sixty-eight core facial key points were extracted, and twelve facial action units were identified based on the displacement changes and motion speed of the key points. At the same time, the optical flow method was used to extract the millisecond-level motion features of twelve micro-expression key points.

[0093] Facial motion units and micro-expression features are input into a hybrid neural network model, which outputs the confidence scores of basic emotion and transient emotion. The confidence threshold for basic emotion is set to 0.5, and the confidence threshold for transient emotion is set to 0.4.

[0094] Emotion recognition results: During the face trial session, the audience's "joy" and "curiosity" emotions accounted for the highest proportion, with a mean confidence score of 0.75 for basic emotions and 0.62 for transient emotions; During the discount explanation session, the audience's "indifference" and "irritability" emotions accounted for the highest proportion, with a mean confidence score of 0.60 for basic emotions and 0.55 for transient emotions.

[0095] Step S4: Precise spatiotemporal alignment of multimodal data and fusion of spatiotemporal attention features

[0096] Precise spatiotemporal alignment: Employing a 0.1-frame precision timestamp sub-frame interpolation matching method, using video frame timestamps as a benchmark, linear interpolation is performed on eye-tracking data to ensure that each video frame corresponds to a complete set of eye-tracking data, basic emotion data, and micro-expression data, with a data deviation of ≤0.003s after alignment;

[0097] Feature extraction: For each frame, eye-tracking features, emotion features, and interaction features are extracted, resulting in a 32-dimensional feature vector.

[0098] Spatiotemporal attention feature fusion: The spatial attention layer assigns weights to features in different areas of the image (product display area weight 0.6, background area weight 0.1), and the temporal attention layer assigns weights to video temporal features (adjacent frame feature association weight 0.8). After 1000 sample iterations to optimize and determine the weights, the spatiotemporal attention mechanism is used to perform weighted fusion of the three types of features to generate a 64-dimensional fusion feature vector.

[0099] Step S5: Calculation of multi-dimensional evaluation indicators and analysis of group differences

[0100] Based on the fused feature vector, the six-dimensional evaluation index was calculated according to the calculation formula of this invention. Simultaneously, the index difference values ​​for the three age groups were calculated. The specific calculation results are as follows (index values ​​normalized to the 0-1 range):

[0101] Visual attention intensity: 0.72 (mean blink frequency correction factor 1.05, because the overall blink frequency of the audience is lower than the group average).

[0102] Emotional resonance index: 0.65 (The expected emotion of the video is "pleasure / curiosity", the average matching degree is 0.82, and the expected emotion weight is set to 1.1).

[0103] Cognitive processing depth: 0.68 =0.5、 =0.3、 =0.2, Δpupil diameter / average pupil diameter =0.15, average fixation duration / frame rate =0.012, saccade amplitude variation coefficient =0.2).

[0104] Emotion and attention synergy: 0.60 (Pearson correlation coefficient 0.55, Spearman correlation coefficient 0.68);

[0105] False expression detection coefficient: 0.25 (no obvious inconsistencies between basic emotion and micro-expressions, or mismatch between pupil diameter and emotion);

[0106] Group index difference value: 0.18 (the emotional resonance index difference between the 31-35 age group and the 20-25 age group is the largest, with a difference of 0.22).

[0107] Step S6: Fine-grained evaluation of video clip-level effects

[0108] Evaluation Unit Division: Combining video shot switching nodes and content semantic boundaries (semantic similarity threshold 0.7), the 60-second marketing video is divided into 8 evaluation units, with each unit lasting 3-8 seconds and no shots shorter than 3 seconds. Each unit corresponds to a different sub-section of the video.

[0109] Evaluation Vector Generation: Calculate the six-dimensional evaluation indicators for each evaluation unit and generate the effect evaluation vector for each unit. For example, the evaluation vector for the 3rd evaluation unit (details of face trial) is [0.85, 0.82, 0.78, 0.75, 0.20, 0.12], and the evaluation vector for the 8th evaluation unit (end of discount explanation) is [0.45, 0.38, 0.50, 0.40, 0.30, 0.25].

[0110] Step S7: Comprehensive effect evaluation of dynamic weighted fusion

[0111] Weight setting and correction: This video is a product-selling marketing video. An improved analytic hierarchy process was used to construct a judgment matrix and correct the weights. The final weights of each indicator were: visual attention intensity 0.4, emotional resonance index 0.35, cognitive processing depth 0.1, emotion and attention synergy 0.1, false expression identification coefficient 0.03, and group indicator difference value 0.02. The sum of the weights is 1.

[0112] Evaluation unit performance score calculation: The performance score of each evaluation unit is calculated according to the formula indicators. Unit 3 has the highest score (0.81) and Unit 8 has the lowest score (0.41).

[0113] Overall performance score calculation: Based on the formula for overall performance score, the overall performance score of this marketing video is calculated to be 0.66 (normalized to the 0-1 range, 0.6 and above is qualified, and 0.8 and above is excellent).

[0114] Step S8: Generate a multi-dimensional evaluation report and personalized optimization suggestions

[0115] Generate a multi-dimensional evaluation report: The report includes time change curves of six-dimensional evaluation indicators, visual heatmaps and emotional trajectory overlays of key frames such as product display / on-face trial, and indicator difference distribution maps of three age groups. It clarifies that the video's strengths are on-face trial and product display, and its weaknesses are the end of the discount explanation and the later stage of the makeup test.

[0116] Weak links were identified: Based on the preset thresholds trained from 100 sets of beauty product sales video samples, the visual attention intensity (0.45) and emotional resonance index (0.38) at the end of the discount explanation were lower than the threshold of 0.5, and the group indicator difference value (0.23) in the later stage of the makeup test was higher than the threshold of 0.2, which were identified as weak links in the effect.

[0117] Provide personalized optimization suggestions:

[0118] Visual layer: To address the issue of low visual attention intensity at the end of the discount explanation, it is recommended to add visual focal elements for the discount information (such as eye-catching price pop-ups and coupon animations), adjust the color contrast of the screen, and change the background of the discount explanation from a solid color to a real-life scene of the foundation's staying power.

[0119] Emotional layer: Regarding the issue of low emotional resonance at the end of the promotional explanation, it is recommended to adjust the emotional expression rhythm of the explanation, increase the empathy points of the target audience (women aged 20-35) such as "limited-time offer" and "exclusive benefits", and change to light and lively background music to improve the emotional matching with the visuals;

[0120] Content layer: In response to the issue of high differences in indicators among different groups in the later stages of makeup wear test, it is recommended to add content adaptation points for cross-age groups, such as showing the makeup wear effect for dry skin and oily skin at the same time, to adapt to the skin type needs of different age groups.

[0121] Predicted improvement in performance: By inputting the fused feature vectors into the pre-trained BP neural network model, it is predicted that after implementing the above optimization suggestions, the overall performance score of the video can be improved to 0.78, with the greatest improvement in visual attention intensity and emotional resonance index, at 0.20 and 0.25 respectively.

[0122] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A marketing video effectiveness evaluation method combining eye-tracking and emotion recognition, characterized in that, Includes the following steps: S1. Multimodal data synchronous acquisition: Play the marketing video to be evaluated, and simultaneously acquire the target audience's eye movement data and facial video data based on the frame timestamp; record the timestamps of the eye movement data and facial video data to ensure that the timelines of the two types of data are aligned; S2. Eye-tracking data processing and visual attention region extraction: Based on video frames, eye-tracking and emotion data are aligned, and data with misaligned timestamps are removed. Preprocess the eye-tracking data; extract the visual attention region for each frame of the video based on the gaze coordinates and gaze duration; The visual attention area is mapped onto the spatial coordinate system of the video frame to generate a dynamic visual heatmap; S3. Facial Expression Recognition and Emotional State Analysis: Frame-level analysis is performed on facial video data to extract facial motion units; based on the combination of facial motion units, the basic emotion category and its confidence level of each frame are identified; the basic emotion categories include joy, sadness, surprise, fear, anger, disgust, and neutrality; S4. Spatiotemporal Alignment and Feature Fusion of Multimodal Data: Eye-tracking data and facial expression data are precisely aligned according to timestamps to construct a spatiotemporally synchronized multimodal feature sequence; for each frame of the video, the following features are extracted: Eye movement characteristics: fixation point coordinates, fixation duration, saccade amplitude, and pupil diameter changes; Emotional features: emotion category probability vector, emotion intensity, facial action unit activation value; Interactive features: emotional response values ​​corresponding to visual attention areas, and the temporal relationship between emotional changes and gaze shifts; S5. Calculation of Multidimensional Evaluation Indicators: Based on the fused multimodal features, the following evaluation indicators are calculated: Visual attention intensity: Based on fixation density and fixation duration, it reflects the degree of visual engagement of the viewer with the video content; Emotional Resonance Index: Based on the degree of match between the emotion category and the expected emotion of the video, as well as the intensity of the emotion, it reflects the degree of emotional identification of the audience; Cognitive processing depth: Based on changes in pupil diameter and fixation duration, it reflects the depth of the viewer's cognitive processing of video content; Emotion and attention synergy: Based on the correlation between fixation shift and emotional change, it reflects the consistency between the viewer's emotional experience and visual attention; False Expression Detection Coefficient: Based on the abnormal matching degree between facial expressions and eye movement patterns, it identifies possible socially expected expressions; S6. Video Segment-Level Performance Evaluation: Divide the marketing video into multiple evaluation units based on scenes or shots; for each evaluation unit, calculate the above indicators and generate the performance evaluation vector for that unit. S7. Overall Effect Evaluation: Based on the indicator vectors of each evaluation unit, a weighted fusion method is used to calculate the overall video effect score; S8. Generate evaluation report and optimization suggestions: Generate video effect evaluation report, including time change curves of various indicators, visual heatmap overlay of key frames, emotion trajectory map, and provide content optimization suggestions for weak links.

2. The marketing video effectiveness evaluation method combining eye tracking and emotion recognition according to claim 1, characterized in that: Step S2, eye-tracking data preprocessing, includes the following sub-steps: S21. Use median filtering algorithm to remove high-frequency noise from eye-tracking data; S22. Identify blinking actions based on a blink detection algorithm and remove eye movement data during blinking; S23. Detect head movement using a head pose estimation algorithm. When the head pose fluctuation exceeds a preset threshold, remove the eye movement data for the corresponding time period. S24. Linear interpolation is used to supplement the missing data after removing invalid data to ensure the continuity of eye-tracking data.

3. The marketing video effectiveness evaluation method combining eye tracking and emotion recognition according to claim 1, characterized in that, In step S2, the method for extracting the visual attention region is as follows: set a gaze duration threshold of 0.1-0.3s, cluster gaze points whose gaze duration exceeds the threshold to form a visual attention region; the generation rule of the dynamic visual heatmap is: the higher the gaze density and the longer the gaze duration, the darker the heatmap color, where the gaze density is the number of gaze points per unit area.

4. The marketing video effectiveness evaluation method combining eye tracking and emotion recognition according to claim 1, characterized in that, In step S3, facial action unit extraction adopts a deep learning-based facial key point detection algorithm to extract 68 facial key points, and facial action units are identified based on the displacement changes of the key points; emotion recognition adopts a convolutional neural network model, inputs facial action unit combination features, and outputs the confidence of each basic emotion category. The confidence threshold is set to 0.

5. When the confidence of a certain emotion category exceeds the threshold, the emotion corresponding to the frame is determined to be of that category.

5. The marketing video effectiveness evaluation method combining eye tracking and emotion recognition according to claim 1, characterized in that, In step S4, the spatiotemporal alignment of multimodal data adopts the timestamp interpolation matching method. When there is a discrepancy between the timestamps of eye-tracking data and facial video data, the eye-tracking data is linearly interpolated based on the timestamp of the video frame, so that each video frame corresponds to a set of eye-tracking data and a set of emotion data. Feature fusion employs an attention mechanism, assigning different weights to eye-tracking features, emotion features, and interaction features. These weights are determined through iterative optimization using training samples.

6. The marketing video effectiveness evaluation method combining eye tracking and emotion recognition according to claim 1, characterized in that: Visual attention intensity Where T is the total number of video frames, and gaze density is the ratio of the number of gaze points in the current frame to the screen area; Emotional Resonance Index Where C is the set of emotion categories, and the matching degree is the cosine similarity between the current emotion and the expected emotion; Cognitive processing depth ,in , These are the weighting coefficients; Emotion and attention synergy = Pearson correlation coefficient (gaze shift sequence, emotion change sequence); False Expression Detection Coefficient The abnormal match is the confidence level when the emotion category is "neutral" but the pupil diameter changes significantly.

7. The marketing video effectiveness evaluation method combining eye tracking and emotion recognition according to claim 1, characterized in that: Overall performance score Where n is the number of indicators. Preset weights (e.g., visual attention intensity weight 0.3, emotional resonance index weight 0.4, cognitive processing depth weight 0.2, and emotion and attention synergy weight 0.1).

8. The marketing video effectiveness evaluation method combining eye tracking and emotion recognition according to claim 1, characterized in that, In step S6, the video evaluation unit is divided according to the following rules: video shot switching is used as the dividing node, the duration of a single evaluation unit is 3-10s, and shots with less than 3s are merged with adjacent shots into one evaluation unit; the effect evaluation vector of each evaluation unit is [visual attention intensity, emotional resonance index, cognitive processing depth, emotion and attention synergy, false expression recognition coefficient].

9. The marketing video effectiveness evaluation method combining eye tracking and emotion recognition according to claim 1, characterized in that, In step S7, the weights of the weighted fusion method are determined by the analytic hierarchy process. By constructing a judgment matrix, the weights of each evaluation index are calculated, and the sum of the weights is 1. The overall video effect score is the average of the weighted sum of the effect evaluation vectors of each evaluation unit and their corresponding weights.

10. The marketing video effectiveness evaluation method combining eye tracking and emotion recognition according to claim 1, characterized in that, In step S8, the criterion for determining the weak link is: a certain evaluation index is lower than a preset threshold, which is determined through training with a large number of samples; the optimization suggestions include video content adjustment, visual element optimization, and emotion guidance optimization, specifically: for units with low visual attention intensity, it is recommended to add visual focus elements; For segments with low emotional resonance, it is recommended to adjust the emotional expression of the content to better suit the target audience; for segments with high false expression detection rates, it is recommended to optimize the authenticity of the content and reduce suggestive hints.

Images