Audience analysis method and system for exhibition, medium and equipment
Through image acquisition and sentiment analysis technology, the facial expressions and postures of exhibition audiences are automatically analyzed, solving the problems of insufficient objective feedback and low response rate in the existing technology, and achieving efficient and accurate feedback data collection and analysis.
Patent Information
- Application Number
- CN202411006837.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-11-29
AI Technical Summary
The existing exhibition feedback system relies on the audience's active feedback, which has problems such as strong subjectivity and low response rate, resulting in the feedback data being not objective and accurate enough.
The image acquisition module, image preprocessing module, emotion analysis module and feedback report production module are adopted to automatically analyze the audience's facial expressions and body postures through computer vision and emotion computing technology to generate emotion analysis reports.
It improves the objectivity and accuracy of feedback, reduces the influence of individual subjective factors, and improves the feedback response rate, significantly improving the coverage and response rate of feedback.
Smart Images

Figure CN119049102A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an audience analysis method, in particular to an audience analysis method, system, medium and equipment for displaying exhibitions, belonging to the technical field of image processing. Background Art
[0002] In the prior art, the feedback system of exhibition display effect mainly relies on the active feedback of the audience, such as questionnaires, interviews and voting. Although these methods can reflect the opinions and feelings of the audience to a certain extent, they have the following main defects.
[0003] First, it is highly subjective. The audience's active feedback is often affected by personal subjective factors, and the feedback results may not be objective and accurate enough. Second, the response rate is low: methods such as questionnaires and interviews require the audience to invest a certain amount of time and energy, and many viewers are unwilling to participate, resulting in a low response rate and incomplete feedback data. Summary of the invention
[0004] The technical problem to be solved by the present invention is to provide a method, system, medium and device for analyzing audiences of exhibitions, which solve the problems of slow and inaccurate collection of audience feedback in the prior art.
[0005] The technical problem to be solved by the present invention is achieved by adopting the following technical solutions:
[0006] The present invention provides a method for analyzing audiences of exhibitions, comprising the following steps:
[0007] Step S1, deploying an image acquisition module in an exhibition area;
[0008] Step S2, transmitting the data acquired in step S1 to an image preprocessing module;
[0009] Step S3, the image preprocessing module performs data preprocessing, including denoising, enhancement and standardization, to generate processed data;
[0010] Step S4, the sentiment analysis module performs sentiment analysis on the processed data in step S3, and outputs the results to the feedback report production module and / or the data storage management module;
[0011] Step S5, the feedback report production module displays or outputs the sentiment analysis result, and / or the data storage management module stores or provides data of the sentiment analysis module.
[0012] As a preferred technical solution of the present invention, in step S4, the sentiment analysis performed by the sentiment analysis module includes the following steps:
[0013] Step S401, receiving and normalizing data, wherein the data to be normalized include facial features and posture features, using the following formula:
[0014]
[0015] Among them, x i is the original data, μ is the mean, and σ is the standard value;
[0016] Step S402, performing face detection, facial feature point extraction and face tracking;
[0017] Step S403, performing posture detection and posture feature extraction;
[0018] Step S404, performing sentiment calculation;
[0019] Step S405, output the result.
[0020] As a preferred technical solution of the present invention, in step S402,
[0021] Including using MTCNN for face detection, where MTCNN uses three cascaded neural networks: P-NetProposal Network, R-Net Refine Network, O-Net Output Network, P-Net generates candidate boxes, R-Net corrects candidate boxes, and O-Net outputs the final facial position and feature points;
[0022] Facial feature point extraction includes extracting feature points using a deep learning model;
[0023] Face tracking includes using the KLT optical flow method to track the movement of the face in the video;
[0024] In the step S403, posture detection is performed by Lite Pose, including feature extraction and key point detection, wherein the feature extraction is to input a depth image, use lightweight convolution to extract features features = LiteConv (x), and the key point detection uses depthwise separable convolution to detect key point positions keypoints = DepthwiseConv (features);
[0025] Posture feature extraction calculates the angle and distance between key points and outputs the posture feature vector. The angle calculation formula is:
[0026]
[0027] The distance calculation formula is:
[0028]
[0029] In step S404, sentiment calculation includes the following steps:
[0030] Step S4041, connect the standardized facial feature points and posture feature vectors, and fuse the facial feature points and posture feature vectors. The formula is as follows:
[0031] F 融合 =[F 面部 , F 姿态 ];
[0032] Step S4042, dynamic weight adjustment is performed. First, behavioral feature indicators are obtained based on facial features and posture features. The formula is as follows:
[0033]
[0034] Where, ΔF 面部,i and ΔF 姿态,i Respectively represent the changes of facial and posture features in the i-th frame, and n is the total number of frames;
[0035] Then, the modal data weight is dynamically adjusted according to the behavioral characteristic index to obtain the adaptively adjusted modal data weight. The formula is as follows:
[0036]
[0037] Then, the fused features are weighted using the dynamically adjusted weights to generate a weighted fused feature vector. The formula is as follows:
[0038] F 加权融合 =w 面部 ·F 面部 +w 姿态 ·F 姿态 ;
[0039] In step S4043, the multimodal deep learning model Multimodal Transformer is used to perform emotional state calculation, including model training and real-time emotional calculation, wherein:
[0040] Real-time emotion calculation includes inputting the weighted fusion feature vector received in real time into the trained emotion analysis model, performing emotion calculation, and outputting the audience's emotional state. The formula is as follows:
[0041]
[0042] Among them, f is the sentiment analysis model, is the predicted affective state;
[0043] In step S405, data encapsulation is performed before outputting the result.
[0044] As a preferred technical solution of the present invention, the step S5 further includes performing aggregate display, statistical analysis and anomaly detection based on the received data, wherein the aggregate display of the emotion state frequency data is as follows:
[0045]
[0046] Among them, f i is the frequency of emotional state i, n i is the number of occurrences of emotional state i, and N is the total amount of data;
[0047] Statistical analysis and display of the statistical characteristics of emotional states. The formula is as follows:
[0048]
[0049] Among them, μ is the mean, σ 2 is the variance, σ is the standard deviation, x i is the emotional state data;
[0050] Anomaly detection includes anomaly detection of emotional state data using the Isolation Forest algorithm, which isolates data points by building random trees to detect outliers.
[0051] As a preferred technical solution of the present invention, the step S5 further includes a data storage management module for data management and maintenance.
[0052] The present invention also provides an audience analysis system for displaying exhibitions, which has the aforementioned audience analysis method for displaying exhibitions, including an image acquisition module, an image preprocessing module, a sentiment analysis module, a feedback report production module and a data storage management module, wherein:
[0053] The image acquisition module includes an image information acquisition submodule and a depth information acquisition submodule;
[0054] The image preprocessing module receives the data from the image acquisition module and outputs it after processing;
[0055] The emotion analysis module calculates the emotion state information through the information output by the image preprocessing module, and outputs it to the feedback report production module and / or the data storage management module;
[0056] The feedback report production module is used to display or output the sentiment analysis results;
[0057] The data storage management module is used to store or provide the analysis results of the sentiment analysis module.
[0058] The present invention also provides a medium storing a computer program that can be loaded by a processor and execute the aforementioned audience analysis method for displaying an exhibition.
[0059] The present invention also provides a device, which is an electronic device, including a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the aforementioned audience analysis method for displaying an exhibition are implemented.
[0060] The beneficial effects of the present invention are: the present invention can improve the objectivity and accuracy of feedback, and by utilizing advanced computer vision and emotional computing technology to analyze the facial expressions and body postures of the audience, thereby obtaining objective and accurate feedback data and reducing the impact of personal subjective factors on the feedback results; the response rate of feedback is improved: the audience does not need to actively participate in questionnaires or interviews, and feedback data can be automatically collected, which significantly improves the coverage and response rate of feedback. BRIEF DESCRIPTION OF THE DRAWINGS
[0061] Figure 1 It is a structural schematic diagram of a system for analyzing audiences for displaying exhibitions according to the present invention;
[0062] In the figure: 1. Image acquisition module; 2. Image preprocessing module; 3. Sentiment analysis module; 4. Feedback report production module; 5. Data storage management module. DETAILED DESCRIPTION
[0063] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.
[0064] Example 1
[0065] This embodiment provides a method for analyzing audiences of an exhibition, comprising the following steps:
[0066] Step S1, deploying an image acquisition module 1 in an exhibition area;
[0067] Step S2, transmitting the data acquired in step S1 to the image preprocessing module 2;
[0068] Step S3, the image preprocessing module 2 performs data preprocessing, including denoising, enhancement and standardization, to generate processed data. Preferably, denoising uses a Gaussian filter to remove image noise, enhancement uses adjustment of contrast and brightness, and standardization uses standardization of image size and color, and outputs clear and standardized image data as processed data;
[0069] Step S4, the sentiment analysis module 3 performs sentiment analysis on the processed data in step S3, and outputs the result to the feedback report production module 4 and / or the data storage management module 5;
[0070] Step S5, the feedback report production module 4 displays or outputs the sentiment analysis result, and / or the data storage management module 5 stores or provides the data of the sentiment analysis module 3;
[0071] Specifically, regarding the generation of feedback reports (i.e., displaying or outputting sentiment analysis results), its typical implementation is to summarize the statistical analysis of sentiment states and anomaly detection results, generate a draft report containing sentiment state distribution, anomaly detection, etc., display sentiment analysis results according to a preset format, or output sentiment analysis results; preferably, its optimization process is to use incremental learning technology, update the Isolation Forest model online, process real-time data, and use the following formula for incremental update:
[0072] IF new =IF old +α(IF new_data -IF old )
[0073] Among them, α is the learning rate, IF new For the updated model, IF new_data Model trained on new data,
[0074] Improve the efficiency and accuracy of anomaly detection and achieve more efficient anomaly detection.
[0075] In step S4, the sentiment analysis performed by the sentiment analysis module 3 includes the following steps:
[0076] Step S401, receiving and normalizing data, wherein the data to be normalized include facial features and posture features, using the following formula:
[0077]
[0078] Among them, x i is the original data, μ is the mean, and σ is the standard deviation;
[0079] Step S402, performing face detection, facial feature point extraction and face tracking;
[0080] Step S403, performing posture detection and posture feature extraction;
[0081] Step S404, performing sentiment calculation;
[0082] Step S405, output the result.
[0083] In step S402,
[0084] Including the use of MTCNN for face detection, where MTCNN uses three cascaded neural networks: P-NetProposal Network, R-Net Refine Network, O-Net Output Network. P-Net generates candidate boxes, R-Net corrects candidate boxes, and O-Net outputs the final facial position and feature points. The detailed process is as follows:
[0085] P-Net:
[0086] 1. Input image and generate candidate boxes
[0087] 2. Filter candidate boxes and output the passed boxes
[0088] score i =σ(W i ·x+b i )
[0089] Among them, σ is the sigmoid function, W i is the weight, x is the input, b i Bias
[0090] R-Net:
[0091] 1. Receive candidate boxes output by P-Net for further screening and correction
[0092] 2. Output the corrected candidate box
[0093] bbox=reg(W·x+b)
[0094] O-Net:
[0095] 1. Receive R-Net output and accurately locate faces and landmarks
[0096] 2. Output facial position and feature points
[0097] landmarks=W landmark ·x+b landmark ;
[0098] Facial feature point extraction includes using a deep learning model to extract feature points and obtain accurate facial features. The formula is as follows:
[0099] Primary Network:
[0100] 1. Input facial image and preliminarily predict the location of feature points
[0101] landmarks0=W0·x+b0
[0102]
[0103] 1. Use a multi-level regression network to gradually correct the position of feature points
[0104] landmarks t+1 =landmarks t +Δlandmarks t
[0105] Among them, Δlandmarks t =W t ·features t ;
[0106] Facial tracking includes using the KLT optical flow method to track the movement of the face in the video. The algorithm steps are:
[0107] 1. Detect feature points in each frame
[0108] 2. Use the optical flow method to calculate the displacement of feature points in consecutive frames
[0109] Calculation results: motion trajectory of facial feature points;
[0110] In step S403, posture detection is performed by Lite Pose, including feature extraction and key point detection, wherein the feature extraction is to input a depth image, and use lightweight convolution to extract features features = LiteConv (x), and the key point detection uses depthwise separable convolution to detect key point positions keypoints = DepthwiseConv (features);
[0111] Posture feature extraction calculates the angle and distance between key points and outputs the posture feature vector. The angle calculation formula is:
[0112]
[0113] The distance calculation formula is:
[0114]
[0115] In step S404, sentiment calculation includes the following steps:
[0116] Step S4041, connect the standardized facial feature points and posture feature vectors, and fuse the facial feature points and posture feature vectors. The formula is as follows:
[0117] F 融合 =[F 面部 , F 姿态 ];
[0118] Step S4042, dynamic weight adjustment is performed. First, behavioral feature indicators are obtained based on facial features and posture features. The formula is as follows:
[0119]
[0120] Where, ΔF 面部,i and ΔF 姿态,i They represent the changes of facial and posture features in the ith frame, respectively, and n is the total number of frames;
[0121] Then, the modal data weight is dynamically adjusted according to the behavioral characteristic index to obtain the adaptively adjusted modal data weight. The formula is as follows:
[0122]
[0123] Then, the fused features are weighted using the dynamically adjusted weights to generate a weighted fused feature vector. The formula is as follows:
[0124] F 加权融合 =w 面部 ·F 面部 +w 姿态 ·F 姿态 ;
[0125] In step S4043, the multimodal deep learning model Multimodal Transformer is used to perform emotional state calculation, including model training and real-time emotional calculation, wherein:
[0126] Real-time emotion calculation includes inputting the weighted fusion feature vector received in real time into the trained emotion analysis model, performing emotion calculation, and outputting the audience's emotional state. The formula is as follows:
[0127]
[0128] Among them, f is the sentiment analysis model, is the predicted affective state;
[0129] Preferably, in step S4043, the steps of training the multimodal sentiment analysis model are:
[0130] 1. Collect and annotate a large amount of training data, including facial expression and posture data and corresponding emotion labels
[0131] 2. Input data into the multimodal sentiment analysis model for training
[0132] 3. Use the back propagation algorithm to update the model parameters. The formula is as follows:
[0133]
[0134] Among them, L is the loss function, y i is the true label, is the predicted label,
[0135] Finally, the trained sentiment analysis model is output;
[0136] In step S405, data encapsulation is performed before outputting the result.
[0137] Step S5 also includes performing aggregate display, statistical analysis and anomaly detection based on the received data, wherein the aggregate display of emotional state frequency data is as follows:
[0138]
[0139] Among them, f i is the frequency of emotional state i, n i is the number of occurrences of emotional state i, and N is the total amount of data;
[0140] Statistical analysis and display of the statistical characteristics of emotional states. The formula is as follows:
[0141]
[0142] Among them, μ is the mean, σ 2 is the variance, σ is the standard deviation, x i is the emotional state data;
[0143] Anomaly detection includes using the Isolation Forest algorithm to detect anomalies in emotional state data. Isolation Forest isolates data points by building random trees to detect outliers.
[0144] Specifically, the steps are as follows:
[0145] 1. Training the Isolation Forest model includes collecting and organizing sample data for training the Isolation Forest model, including normal and abnormal data points, training the Isolation Forest model data set, and then building a random tree for isolating data points. The operation is to randomly select a feature and randomly select a split point on the feature, divide the data into two parts, and recursively continue the above process for each split part until each data point is isolated separately. Each tree only uses a subset of the data set, and the depth of the tree is limited, usually log2(n), where n is the number of data points. The result is a forest composed of multiple random trees:
[0146] tree=build_tree(X,current_depth)
[0147] Among them, X is the data point, current_depth is the depth of the current tree;
[0148] Then calculate the path length of each data point in the random tree. For each data point, traverse the random tree and record the path length from the root node to the leaf node:
[0149] path_length(x)=traverse_tree(x,tree,0)
[0150] Among them, x is the data point, tree is the random tree, and 0 is the initial path length;
[0151] Get the average path length for each data point:
[0152]
[0153] Where h(x) is the average path length of data point x, n is the number of random trees, and T i is the i-th random tree;
[0154] 2. Perform outlier detection, including calculating the anomaly score using the average path length of the data points. The anomaly score reflects the difficulty of isolating the data point. The anomaly score of each data point is calculated as follows:
[0155]
[0156] Where s(x) is the anomaly score of data point x, h(x) is the average path length, and c(n) is the normalization coefficient
[0157]
[0158] Where H(n) is the nth harmonic number
[0159] Then, the data point is judged as an outlier based on the anomaly score. Specifically, a threshold is set. Data points with an anomaly score greater than the threshold are judged as outliers, and the outlier point set is obtained:
[0160] if s(x)>threshold, then x is an anomaly
[0161] Among them, threshold is a predefined threshold;
[0162] Regarding optimization, it includes model parameter optimization and algorithm optimization, among which,
[0163] Regarding model parameter optimization, parameter selection is to select the best parameters of the Isolation Forest model, and use grid search or random search techniques to select parameters, such as the number of trees (n_estimators) and the number of samples (max_samples), to obtain the best parameter combination; model evaluation, using cross-validation technology to evaluate model performance, calculate the model's accuracy, recall rate and F1 score, and obtain the best performing Isolation Forest model, so as to evaluate model performance and select the optimal model;
[0164] Regarding algorithm optimization, it includes using improved segmentation strategies, such as segmentation based on data density, using parallel computing to build and traverse random trees, improving the efficiency of tree structure construction and traversal, and obtaining a more efficient tree structure;
[0165] Regarding real-time anomaly detection (i.e., real-time detection of abnormal points in emotional state data), firstly, the emotional state data received in real time is input into the trained Isolation Forest model, and the abnormal score of each data point is output to obtain the detected abnormal emotional state data:
[0166]
[0167] Where s(x) is the anomaly score of data point x, h(x) is the average path length, and c(n) is the normalization coefficient.
[0168] In step S5, a data storage management module 5 is also included to perform data management and maintenance.
[0169] Preferably, the following is a typical implementation process:
[0170] Step S1: An image acquisition module is deployed in the exhibition area to collect the facial expression and body posture data of the audience in real time, including a high-definition camera and a depth camera to ensure that the facial expression and posture of the audience can be clearly captured.
[0171] Step S2: Transmit the data acquired in step S1 to the image preprocessing module.
[0172] Step S3: The image preprocessing module performs data preprocessing, performs denoising, enhancement and standardization, and generates processed data. The preprocessed data here will be used for subsequent facial and posture feature extraction;
[0173] Denoising: Use Gaussian filter I′=G(I) where:
[0174] Enhancements: Adjust contrast and brightness:
[0175] I″=αI′+β,
[0176] Among them, α is the contrast adjustment parameter, and β is the brightness adjustment parameter.
[0177] Normalization: Standardize image size and color
[0178] Among them, μ and σ are the mean and standard deviation of the image respectively;
[0179] Step S4: The sentiment analysis module performs sentiment analysis on the processed data in step S3. It includes the following sub-steps:
[0180] The execution steps of face detection, feature point extraction and tracking are:
[0181] 1. Face detection: MTCNN is used for face detection, which is divided into three cascaded neural networks:
[0182] P-Net generates candidate boxes: P(x)
[0183] R-Net corrects the candidate box: R(x)
[0184] O-Net outputs the final facial position and feature points: O(x)
[0185] The confidence of the candidate box output by P-Net is: C = σ(W p ·X+b p )
[0186] R-Net corrects the coordinates of the candidate box: R = W r ·C+b r
[0187] O-Net outputs the final facial position and feature points: F = W o ·R+b o
[0188] 2. Facial feature point extraction: Use deep learning model to extract feature points F(x)
[0189] 3. Face tracking: Use KLT optical flow method for face tracking to ensure the continuity of facial feature points in the video.
[0190] Feature point detection formula:
[0191] KLT optical flow method calculates the motion trajectory of feature points: x t+1 =x t +Δx where
[0192] The execution steps of posture detection and feature extraction are:
[0193] 1. Posture detection: Use Lite Pose for posture detection, which mainly includes:
[0194] Feature extraction: Lightweight convolution to extract features C(x)
[0195] Key point detection: Depthwise separable convolution detects key point position D(x)
[0196] Feature extraction formula: C(x)=W c ·X+b c
[0197] Key point detection formula: D(x) = σ(W d ·C(x)+b d )
[0198] 2. Posture feature extraction: calculate the angle and distance between key points to generate posture feature vectors.
[0199] Angle calculation formula:
[0200] Distance calculation formula:
[0201] Posture feature vector: V pose =[θ1,θ2,…,θ n ,d1,d2,…,d n ].
[0202] Steps for sentiment calculation and dynamic weight adjustment:
[0203] 1. Fusion of facial features and posture features: Connect the standardized facial feature points and posture feature vectors.
[0204] Fusion feature vector: V concat =[F norm ,P norm ]
[0205] 2. Dynamic weight adjustment: Dynamically adjust the modal data weight according to the behavioral characteristic indicators.
[0206] Behavioral characteristic indicators
[0207] Calculation formula: B = W f ·F+W p ·P Among them,
[0208] W f and W p is the weight matrix of facial features and posture features.
[0209] Dynamically adjust modal data weight: W = W0 (1 + αB)
[0210] Among them, W0 is the initial weight and α is the dynamic adjustment coefficient.
[0211] Weighted fusion feature vector: V weighted =W·V concat
[0212] Steps for calculating emotional state:
[0213] 1. Multimodal deep learning model calculation: Use Multimodal Transformer to calculate emotional state.
[0214] Emotional state calculation formula: S = T (V weighted )
[0215] Among them, T is the multimodal Transformer model.
[0216] Real-time emotion calculation formula: S real-time =T(V weighted )
[0217] Then execute step S5: the feedback report production module displays or outputs the sentiment analysis results, and the data storage management module stores or provides the data of the sentiment analysis module to ensure the long-term validity and security of the data. A database can be selected to store the data for easy retrieval.
[0218] Processing of aggregate display, statistical analysis and anomaly detection:
[0219] 1. Aggregate display: aggregate display of emotional state frequency data.
[0220] Aggregate formula:
[0221] 2. Statistical analysis: showing the statistical characteristics of emotional states.
[0222] Statistical formula:
[0223]
[0224]
[0225] 3. Anomaly detection: Use the Isolation Forest algorithm to perform anomaly detection on the emotional state data.
[0226] Train the IsolationForest model:
[0227] Construct a random tree T(x) Calculate the path length: L(x) Anomaly score calculation formula:
[0228] Where C is a normalization constant. Real-time anomaly detection formula:
[0229] The following is an example of actual operation combined with actual usage scenarios:
[0230] Scenario description
[0231] Exhibition activities: an art exhibition;
[0232] Number of participants: 1,000;
[0233] Data collection equipment: HD camera and depth camera;
[0234] Data types collected: facial expression data and body posture data;
[0235] Objective: Analyze the audience's emotional state and generate feedback reports;
[0236] Implementation steps and formula examples:
[0237] Step S1: Deploy image acquisition modules in the exhibition area
[0238] Assume that each camera captures 30 frames of images per second, the resolution of each frame is 1920x1080, and each visitor stays in the exhibition area for 10 minutes.
[0239] Step S2: Send the data of step S1 to the image preprocessing module;
[0240] Step S3: Data preprocessing:
[0241] Denoising: Use a Gaussian filter with σ set to 1.0 to perform denoising on each frame of the image.
[0242] Enhancement processing: adjust contrast and brightness I″=αI′+β, where α=1.2 and β=20.
[0243] Normalization: Standardize image size and color
[0244] Assume that the mean μ of the image is 128 and the standard deviation σ is 64.
[0245] Assuming that an average of 18,000 frames of image data are collected for each viewer (10 minutes x 60 seconds x 30 frames), a total of 1,000 x 18,000 = 18 million frames of image data are obtained.
[0246] Step S4: Sentiment Analysis
[0247] 1. Face detection, feature point extraction and tracking
[0248] P-Net generates candidate boxes: C = σ(W p ·X+b p )
[0249] Assuming that 5 faces are detected in each frame, a total of 90 million facial data are detected.
[0250] R-Net corrects the candidate box: R = W r ·C+b r
[0251] O-Net outputs the final facial position and feature points: F = W o ·R+b o
[0252] 2. Posture detection and feature extraction
[0253] Angle calculation formula: Assuming that 15 posture features are extracted from each frame image, a total of 270,000,000 posture features are extracted.
[0254] Distance calculation formula:
[0255] Posture feature vector: V pose =[θ1,θ2,…,θ n ,d1,d2,…,d n ]
[0256] 3. Emotional calculation and dynamic weight adjustment
[0257] Fusion feature vector: V concat =[F norm ,P norm ]
[0258] Dynamic weight adjustment:
[0259] Behavioral characteristic index calculation formula: B = W f ·F+W p ·P
[0260] Assume W f =0.5,W p =0.5.
[0261] Dynamically adjust modal data weight: W = W0 (1 + αB)
[0262] Assuming W0=1.0, α=0.1, then W=1.0·(1+0.1B).
[0263] Weighted fusion feature vector: V weighted =W·V concat
[0264] Assuming B = 0.8, then W = 1.08, V weighted =1.08 V concat .
[0265] 4. Emotional state calculation
[0266] Emotional state calculation formula:
[0267] S=T(V weighted )
[0268] Assume that the sentiment state S output by the multimodal Transformer model T is [0.6, 0.4, 0.5, ...].
[0269] Step S5: Feedback report production
[0270] 1. Aggregate display: aggregate display of emotional state frequency data
[0271]
[0272] Assume that a total of 18 million emotion state data are analyzed, and the aggregate emotion state result is [0.55, 0.15, 0.10, 0.05, 0.15], representing happiness, sadness, anger, surprise, and neutral, respectively.
[0273] 2. Statistical analysis: showing the statistical characteristics of emotional states
[0274] Assume that the mean of the affective state is 0.45, the variance is 0.0225, and the standard deviation is 0.15.
[0275] 3. Anomaly detection: Use the Isolation Forest algorithm to perform anomaly detection on emotional state data. Training the Isolation Forest model: Construct a random tree T(x)
[0276] Calculate the path length: L(x)
[0277] Anomaly score calculation formula:
[0278] Set C = 10, then the abnormal score
[0279] Assume that the path length of a certain emotional state is L = 20, then A(x) = 2 -2 =0.25.
[0280] Regarding the output results, the above calculation process yields:
[0281] 1. Audience emotional state distribution
[0282] Happiness: 55%, which means that most of the visitors expressed happy emotions during the exhibition.
[0283] Sadness: 15%. Some visitors expressed sad emotions during the exhibition.
[0284] Anger: 10%, a small number of viewers expressed anger.
[0285] Surprise: 5%, a small number of viewers expressed surprise.
[0286] Neutral: 15%. Some viewers have a relatively neutral emotional state, without obvious emotional fluctuations.
[0287] 2. Statistical characteristics
[0288] Mean of emotional state: used to measure the average level of overall emotional state. Assuming that we represent different emotional states with numerical values (such as happiness is 1, sadness is -1, anger is -2, surprise is 0, neutral is 0), the calculated mean is 0.45, indicating that the overall emotion is biased towards positive.
[0289] Standard Deviation: It is used to measure the fluctuation of emotional state. Assuming the standard deviation is 0.15, it means that the fluctuation of emotional state is small, and the emotional state of most viewers is concentrated around the average value.
[0290] 3. Anomaly detection results
[0291] The Isolation Forest algorithm detected 1,000 pieces of abnormal emotional state data, accounting for 0.0056% of the total data. These abnormal emotional states may be due to strong emotional reactions caused by certain exhibition contents.
[0292] Example 2
[0293] like Figure 1 As shown, this embodiment provides an audience analysis system for displaying exhibitions, which has an audience analysis method for displaying exhibitions in embodiment 1, including an image acquisition module 1, an image preprocessing module 2, a sentiment analysis module 3, a feedback report production module 4 and a data storage management module 5, wherein:
[0294] The image acquisition module 1 includes an image information acquisition submodule and a depth information acquisition submodule. The image information acquisition submodule can be a camera that collects the audience's facial expression and body posture data in real time. It can be installed at the audience's sight height in the exhibition area to ensure that the audience's facial expression can be clearly captured. Preferably, a high-definition camera is installed at the audience's sight height to cover the key areas of the exhibition area. The number of cameras is determined according to the size of the exhibition area and the audience flow to cover all key areas; preferably, the depth information acquisition submodule can be a depth camera, which is installed in parallel with the high-definition camera to ensure that the audience's body posture can be captured synchronously;
[0295] The image preprocessing module 2 receives the data from the image acquisition module 1, processes it and outputs it;
[0296] The emotion analysis module 3 calculates the emotion state information through the information output by the image preprocessing module 2, and outputs it to the feedback report production module 4 and / or the data storage management module 5;
[0297] Feedback report production module 4, used to display or output sentiment analysis results;
[0298] The data storage management module 5 is used to store or provide the analysis results of the sentiment analysis module 3.
[0299] Example 3
[0300] This embodiment provides a medium storing a computer program that can be loaded by a processor and execute a method for analyzing audiences of an exhibition as described in Embodiment 1.
[0301] Example 4
[0302] This embodiment provides a device, which is an electronic device, including a processor and a memory, wherein the memory stores programs or instructions that can be executed on the processor, and when the programs or instructions are executed by the processor, the steps of a method for analyzing audiences of exhibitions as described in Embodiment 1 are implemented.
[0303] The present invention can improve the objectivity and accuracy of feedback. By utilizing advanced computer vision and emotional computing technologies, the facial expressions and body postures of the audience are analyzed, thereby obtaining objective and accurate feedback data and reducing the impact of personal subjective factors on the feedback results. The feedback response rate is improved: the audience does not need to actively participate in questionnaires or interviews, and feedback data can be automatically collected, which significantly improves the coverage and response rate of feedback.
[0304] It also has the following advantages:
[0305] 1. Improve the objectivity and accuracy of feedback data: By automatically identifying and analyzing the audience's facial expressions, the system can provide objective data based on actual emotional responses, avoiding data bias and inaccuracy caused by subjective factors in traditional feedback methods.
[0306] 2. Improve feedback response rate: Using non-invasive expression recognition technology, feedback data can be automatically collected without the audience actively participating in questionnaires or interviews. This automated feedback collection method significantly improves the feedback response rate and ensures the breadth and representativeness of feedback data.
[0307] 3. Simplify the data analysis process: The system uses automated data processing and sentiment analysis technology to efficiently process large amounts of expression data and generate intuitive and easy-to-understand feedback reports. Compared with traditional methods, this approach significantly reduces the investment in manpower and time, and improves the efficiency and accuracy of data analysis.
[0308] 4. Support continuous improvement of exhibition effects: The system can continuously collect and analyze audience feedback data to provide a scientific basis for long-term optimization of exhibitions. Exhibition organizers can continuously improve the content and form of exhibitions based on system feedback to enhance the overall quality of the exhibition and audience experience.
[0309] 5. Wide scope of application: The present invention is not only suitable for traditional exhibition venues such as museums and art galleries, but can also be widely used in various exhibition activities, such as commercial exhibitions, educational exhibitions, theme parks, etc., and has broad application prospects.
[0310] The basic principles, main features and advantages of the present invention are shown and described above. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of the present invention is defined by the attached claims and their equivalents.
Claims
1. A method for analyzing audiences of exhibitions, It is characterized in that The steps include: Step S1, deploying an image acquisition module (1) in an exhibition area; Step S2, transmitting the data acquired in step S1 to the image preprocessing module (2); Step S3, the image preprocessing module (2) performs data preprocessing, including denoising, enhancement and standardization, to generate processed data; Step S4, the sentiment analysis module (3) performs sentiment analysis on the processed data in step S3, and outputs the result to the feedback report production module (4) and / or the data storage management module (5); Step S5, the feedback report production module (4) displays or outputs the sentiment analysis result, and / or the data storage management module (5) stores or provides the data of the sentiment analysis module (3).
2. A method for analyzing audiences of an exhibition according to claim 1, It is characterized in that In step S4, the sentiment analysis performed by the sentiment analysis module (3) comprises the following steps: Step S401, receiving and normalizing data, wherein the data to be normalized include facial features and posture features, using the following formula: Among them, x i is the original data, μ is the mean, and σ is the standard deviation; Step S402, performing face detection, facial feature point extraction and face tracking; Step S403, performing posture detection and posture feature extraction; Step S404, performing sentiment calculation; Step S405, output the result.
3. A method for analyzing audiences of an exhibition according to claim 2, It is characterized in that In step S402, Including using MTCNN for face detection, where MTCNN uses three cascaded neural networks: P-Net (Proposal Network), R-Net (Refine Network), O-Net (Output Network), P-Net generates candidate boxes, R-Net corrects the candidate boxes, and O-Net outputs the final facial position and feature points; Facial feature point extraction includes extracting feature points using a deep learning model; Face tracking includes using the KLT optical flow method to track the movement of the face in the video; In the step S403, posture detection is performed by Lite Pose, including feature extraction and key point detection, wherein the feature extraction is to input a depth image, use lightweight convolution to extract features features = LiteConv (x), and the key point detection uses depthwise separable convolution to detect key point positions keypoints = DepthwiseConv (features); Posture feature extraction calculates the angle and distance between key points and outputs the posture feature vector. The angle calculation formula is: The distance calculation formula is: In step S404, sentiment calculation includes the following steps: Step S4041, connect the standardized facial feature points and posture feature vectors, and fuse the facial feature points and posture feature vectors. The formula is as follows: F 融合 =[F 面部 ,F 姿态 ]; Step S4042, dynamic weight adjustment is performed. First, behavioral feature indicators are obtained based on facial features and posture features. The formula is as follows: Where, ΔF 面部,i and ΔF 资态,i They represent the changes of facial and posture features in the i-th frame, respectively, and n is the total number of frames; Then, the modal data weight is dynamically adjusted according to the behavioral characteristic index to obtain the adaptively adjusted modal data weight. The formula is as follows: Then, the fused features are weighted using the dynamically adjusted weights to generate a weighted fused feature vector. The formula is as follows: F 加权融合 =ω 面部 ·F 面部 +oh 姿态 ·F 姿态 ; In step S4043, the emotional state calculation is performed using a multimodal deep learning model (Multimodal Transformer), including model training and real-time emotional calculation, wherein: Real-time emotion calculation includes inputting the weighted fusion feature vector received in real time into the trained emotion analysis model, performing emotion calculation, and outputting the audience's emotional state. The formula is as follows: Among them, f is the sentiment analysis model, is the predicted emotional state. In step S405, data encapsulation is performed before outputting the result.
4. The method for analyzing audiences of an exhibition according to claim 2, It is characterized in that The step S5 also includes performing aggregate display, statistical analysis and anomaly detection based on the received data, wherein the aggregate display of emotional state frequency data is as follows: Among them, f i is the frequency of emotional state i, n i is the number of occurrences of emotional state i, and N is the total amount of data; Statistical analysis and display of the statistical characteristics of emotional states. The formula is as follows: Among them, μ is the mean, σ 2 is the variance, σ is the standard deviation, x i is the emotional state data; Anomaly detection includes anomaly detection of emotional state data using the Isolation Forest algorithm. IsolationForest isolates data points by building random trees to detect outliers.
5. The method for analyzing audiences of an exhibition according to claim 1, It is characterized in that The step S5 also includes a data storage management module (5) for data management and maintenance.
6. A system for analyzing audiences of exhibitions, comprising a method for analyzing audiences of exhibitions as claimed in any one of claims 1 to 5, It is characterized in that It includes an image acquisition module (1), an image preprocessing module (2), a sentiment analysis module (3), a feedback report production module (4) and a data storage management module (5), wherein: The image acquisition module (1) comprises an image information acquisition submodule and a depth information acquisition submodule; The image preprocessing module (2) receives the data from the image acquisition module (1), processes the data and outputs the data; The emotion analysis module (3) calculates the emotion state information through the information output by the image preprocessing module (2), and outputs the information to the feedback report generation module (4) and / or the data storage management module (5); The feedback report production module (4) is used to display or output the sentiment analysis results; The data storage management module (5) is used to store or provide the analysis results of the sentiment analysis module (3).
7. A medium, It is characterized in that The computer program is stored which can be loaded by a processor and executes the audience analysis method for displaying an exhibition as claimed in any one of claims 1 to 5.
8. A device, It is characterized in that An electronic device includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of a method for analyzing audiences for displaying exhibitions as described in any one of claims 1 to 5 are implemented.
Citation Information
Cited By
Method for analyzing experience degree of offline experience scene
CN119624527A
Lightweight algorithm for estimating 3D human body posture from video
CN120388395A
