A method for predicting the interest of live shopping based on eye movement features and DeepFM
By combining eye-tracking features and the DeepFM architecture, and utilizing automated algorithms to extract eye-tracking parameters and integrate a self-attention mechanism, the accuracy of existing live shopping interest assessments and their reliance on manual features are addressed, resulting in more efficient interest prediction.
Patent Information
- Application Number
- CN202210369501.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-09
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2042-04-09
AI Technical Summary
Existing methods for assessing interest in live shopping rely on viewers’ self-reports and click-through rate predictions, ignoring eye-tracking parameters and detailed product information. This leads to inaccurate assessments, dependence on manual feature engineering, limited sample sizes, and insufficient model learning capabilities.
We adopt a live shopping interest prediction method based on eye-tracking features and DeepFM. We extract eye-tracking parameters through eye-tracking data processing and the automated algorithm FDIMP. We combine the DeepFM architecture and self-attention mechanism, integrate user behavior and project knowledge, and use collaborative infographics for data encoding and prediction.
It improves the accuracy and reliability of predicting interest in live shopping, increases data dimensions, reduces the number of iterations, and enhances the model's learning ability and prediction accuracy.
Smart Images

Figure CN114706481B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of recommendation systems, and in particular to a method for predicting interest in live shopping based on eye movement features and DeepFM. Background Art
[0002] Online live shopping has become one of the channels for people to obtain information for daily shopping. Understanding the user's interest in the live broadcast process can not only improve the merchant's live broadcast strategy and increase the user's satisfaction with watching live broadcasts, but also help designers develop more humane live broadcast interaction methods and improve user experience. Therefore, studying the user's interest in watching shopping live broadcasts has great practical significance.
[0003] Eye movement characteristics refer to the data characteristics of the subjects' eyes when watching live broadcasts. Traditionally, eye movement technology is an analytical tool that can be used in different disciplines such as medicine, psychology, and marketing. In the process of visual evaluation, the method of combining eye tracking with some data processing methods can obtain fine-grained information in the individual cognitive process, and has achieved satisfactory results in various scene detections.
[0004] Currently, the evaluation of the interest in live shopping mostly adopts a "black box" research, that is, relying on the viewer's self-description to reflect the degree of interest in live broadcasting. This judgment of interest not only carries the subjective factors of the viewer, but is also affected by many objective factors such as environment, mood, etc., and it is difficult to truly reflect the influence of the viewer's interest in live shopping. With the development of neural networks, click-through rate prediction technology is also increasingly used in interest prediction models, but this kind of research ignores a lot of objective information, such as the level of detail of the goods in live shopping, eye movement parameters and other factors that may be very important. Therefore, it is necessary to introduce these factors into the live shopping interest model.
[0005] Eye tracking technology is used in many fields to complete recommendation or classification tasks. In recommendation tasks, the improvement of the AUC (Area Under Curve) indicator is mostly between 2% and 10%, and targeted conclusions or models have been drawn on respective research issues. However, most of its models are machine learning methods and the number of samples used is small, mostly from dozens to hundreds, which brings certain accidental factors to the experiment. The learning ability and accuracy of the model need to be further improved.
[0006] In terms of interest models, existing interest prediction models are mainly divided into two categories, namely click-through rate prediction models based on machine learning and click-through rate prediction models based on deep learning. Interest prediction models based on machine learning are mainly divided into two categories: single model and combination model prediction. In single models, logistic regression, decision tree, etc. are more common models. In terms of model combination, GBDT+LR, FwFM, etc. are more common models. However, interest prediction models based on machine learning rely more on manual processing of features, and a large amount of manual feature engineering is required in the early stage of model application. Interest prediction models based on deep learning have shown good results in the field of interest prediction by exploring high-order combinations between features. Among them, Wide&Deep, FGCNN, DeepFM, etc. are more common models.
[0007] In the research related to live broadcast interest estimation, eye movement data was not used as a data dimension in the model. Summary of the invention
[0008] The present invention overcomes the defects of existing methods and proposes a method for predicting the interest of live shopping based on eye movement features and DeepFM. The technology trains a model based on the user's historical browsing data (including eye movement data). Users can use this model to predict the user's interest in the live broadcast products, so that relevant personnel can adjust the live broadcast strategy and improve the user experience during the live broadcast.
[0009] A method for predicting the interest of live shopping based on eye movement features and DeepFM, comprising the following steps:
[0010] S1. Process the gaze time-related data of the live video output by the eye tracker; use the encapsulated tracking model to track the live sale items; establish a tracking frame for the user's viewpoint and the target area, and when the target area covers the user's viewpoint, it is determined that the overlap means that the user's viewpoint is focusing on the area during the corresponding time;
[0011] S2, based on step S1, obtaining the user's eye movement data, and introducing a collaborative information graph, encoding the user behavior and project knowledge into a unified relationship graph through the collaborative information graph;
[0012] S3, based on the DeepFM architecture, adds a self-attention mechanism to the deep neural network to improve the model's ability to learn key information;
[0013] S4. Output the results and judge the model accuracy through the binary cross entropy loss function Logloss and AUC.
[0014] The prediction method of the present invention can be divided into two stages, namely the eye movement data capture stage and the prediction stage. In the data capture stage, various parameters of the user's attention to a single area are obtained through an automated process and the eye tracker supporting software. In the prediction stage, user behavior information, self-information and eye movement information are spliced as input, passed through the prediction model and the predicted value is output. The FDIMP algorithm is proposed to solve the task of live video processing. On the basis of improving the tracking model's ability to distinguish targets and backgrounds and reducing the number of iterations, it provides an automated function for outputting required data from the video. After processing, the original data set is filled in preparation for the interest live broadcast model. Based on the DeepFM architecture, the eye movement data is embedded and encoded after the collaborative information graph is introduced. Adding a self-attention mechanism to the deep neural network improves the model's ability to learn key information.
[0015] Compared with the recommended method under the existing research, the advantages of the present invention are:
[0016] 1) For the first time, eye movement factors were applied to the study of live shopping, and the data dimension was increased by filling in information to the original data set.
[0017] 2) An automatic eye movement data collection algorithm FDIMP was designed to automatically extract eye movement parameters. A model for automatically extracting eye movement features and a live shopping interest model were developed, and an improved deep learning model was used to enhance the reliability and accuracy of model predictions.
[0018] 3) By integrating the DeepFM architecture with the multi-head attention mechanism, a live shopping interest prediction method based on DeepFM was established. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the required drawings:
[0020] Figure 1 This is a technical solution flow chart of an implementation case of the present invention, showing the approximate technical solution of the click-through rate prediction method based on deep feature fusion.
[0021] Figure 2 The collaborative information graph structure proposed for the present invention.
[0022] Figure 3 A multimodal information encoder used in the present invention.
[0023] Figure 4 It is a classic DNN model.
[0024] Figure 5 The impact of the number of fully connected layers on AUC DETAILED DESCRIPTION
[0025] The purpose of the present invention is to solve the problems existing in the prior art and provide a method for predicting the interest of live shopping based on eye movement features and DeepFM. The specific technical solutions adopted by the present invention are as follows:
[0026] like Figure 1 The technical solution flow chart is shown, and the solution specifically includes the following steps:
[0027] S1. Process the gaze time-related data of the live video output by the eye tracker. Use the encapsulated tracking model to track the live sale items. Establish a tracking frame for the user's viewpoint and the target area. When the target area covers the user's viewpoint, it is determined that the overlap means that the user's viewpoint is focusing on the area during the corresponding time.
[0028] Step S1 specifically includes the following steps:
[0029] S11. Use random samples in the video sequence for training, that is, extract three frames from a certain frame and the previous frame as the training set, and extract three frames from the frame after the frame as the test set. Pool the features of the extracted target area to obtain an initialized feature image, and generate a three-dimensional (4*4*n) feature filter. Use the initialized filter combined with the background information of the target area to optimize, and use an iterative method to obtain the optimized filter.
[0030] The filter is optimized with MSE. Since there are many negative samples and the labels of negative samples are collectively referred to as 0, this requires the model to be complex enough. In this case, discriminating negative samples will cause the model to tend to learn negative samples rather than distinguishing between negative and positive samples. In order to solve this problem, weights are added to the Loss, and with reference to HingeLoss in SVM, a large number of negative samples in the scoremap are filtered out. For the positive sample area, MSELoss is used, so the final residual function is as follows:
[0031] r(s,c)=v c ·(m c s+(1-m c )max(0,s)-y c )
[0032] In the formula, the subscript c represents the degree of dependence on the central point. c is the weight, m c (t)∈[0,1] is Mask, in the background area, m c ≈0, in the object corresponding area m c ≈1, HingeLoss is used in the background area, and MSELoss is used in the object area.
[0033] S12. Use the encapsulated data processing algorithm to track the live sale items. It is necessary to establish a tracking frame for the user's viewpoint and the target area. When the target area covers the user's viewpoint, it is determined to be overlapped, that is, the user's viewpoint is focusing on the area during the corresponding time. The above method is used to collect information for the demonstrated sale items, live broadcast anchor, background, comment area, and event coupon area.
[0034] S2. Based on step S1, the user's eye movement data is obtained and a collaborative information graph is introduced. The collaborative information graph is used to encode the user behavior and project knowledge into a unified relationship graph.
[0035] Step S2 specifically includes the following steps:
[0036] S21, such as Figure 2 As shown, we define a user-item bipartite graph {(e u ,y ui ,e i )|e u ∈U,e i ∈I}, where e u is a user entity, y ui Represents the link between user u and item i, e i represents the project entity, u and i represent the user and project set respectively. When there is interaction between the two, y ui is 1. The collaborative information graph incorporates the new data dimension, where each user's behavior can be represented as a triple (e u , Interact, ei). Interact = 1 means there is an additional interaction relationship e u and e i , so that the user information graph can be integrated with the newly added dimensions into a unified graph.
[0037] S22, such as Figure 3 The multimodal information encoder shown takes the newly added dimension entity and the original information entity as input, and encodes the relation behavior (relation) and project knowledge (item, entity) into a unified relation graph through the collaborative information graph. The collaborative information graph incorporates the newly added data dimension into it. The entity encoder and the attention layer are used to learn a new entity representation for each entity. The new entity representation aggregates the information of neighboring entities while retaining its own information. After being converted into a dense vector, it is embedded with the new entity representation in the following steps.
[0038] S3. Based on the Deepfm architecture, a self-attention mechanism is added to the deep neural network to improve the model's ability to learn key information.
[0039] Step S3 specifically includes the following steps:
[0040] S31. The prediction model introduces DNN to perform a fully connected layer cascade on the feature vector x after embedding encoding, and establishes a regression or classification model. Figure 4 A classic DNN model is given, in which the output of each neuron is the corresponding linear weighted value of the previous layer of neurons after nonlinear mapping, that is, for the l+1 layer of neurons, the corresponding output value is:
[0041]
[0042] Where W (l) , a (l) and b (l) They respectively represent the weight matrix of the lth layer, the output response of the lth neuron, and the bias value vector connecting the lth layer and the l+1th layer. It is a nonlinear mapping function, and the following Sigmoid function or ReLU function is commonly used.
[0043]
[0044]
[0045] S32. In the interest model, some data and some eye movement data of the user in the product features may be key factors. A single attention network is not enough to capture multiple aspects of representation. Use a multi-head attention network to jointly focus on information from different positions and different representation spaces to simulate user preferences from multiple interesting views. In the interest interaction layer, the sequence itself is used as the key and value vector of the data, and its output vector can be aggregated from the previous hidden output of the neural network.
[0046] S4. Output the results and evaluate the model accuracy using the binary cross entropy loss function Logloss and AUC (Area Under Curve). Logloss is defined as:
[0047]
[0048] N is the total number of samples in the test set, y (i) and are the true value of the category of the i-th sample in the test set and the probability value of the predicted user click;
[0049] AUC is defined as the area enclosed by the coordinate axis under the ROC curve:
[0050]
[0051] Among them, R is the threshold, fpr is the false positive rate; different classification thresholds can obtain the true positive rate curve under different false positive rates, namely ROC.
[0052] Furthermore, Table 1 shows the performance parameters of the mainstream interest estimation model in the present invention and the improved models after increasing the dimension of eye movement data and adding the self-attention mechanism.
[0053] Table 1 AUC values of the test set under different improvements
[0054]
[0055] Figure 5 The influence of the number of fully connected layers of the adaptive residual DNN on the AUC is given. It can be seen that when the convolution kernel is 4*4, the AUC value is as high as 0.8566, and the LOSS value is 0.5988. The experimental results show that the choice of the number of fully connected layers of the DNN has an important influence on the generalization performance of the model.
Claims
1. A method for predicting the interest of live shopping based on eye movement features and DeepFM, comprising the following steps: S1. Process the gaze time-related data of the live video output by the eye tracker; use the encapsulated tracking model to track the live sale items; establish a tracking frame for the user's viewpoint and the target area, and when the target area covers the user's viewpoint, it is determined that the overlap means that the user's viewpoint is focusing on the area during the corresponding time; S2, based on step S1, obtaining the user's eye movement data, and introducing a collaborative information graph, encoding the user behavior and project knowledge into a unified relationship graph through the collaborative information graph; specific The following steps are involved: S21. Define a user-item bipartite graph {(e u ,y ui ,e i )|e u ∈U,e i ∈I}, where e u is a user entity, y ui Represents the link between user u and item i, e i represents the project entity, u and i represent the user and project set respectively; when there is interaction between the two, y ui =1; the collaborative information graph incorporates the new data dimension into it, where each user’s behavior can be represented as a triple (e u , Interact, ei); Interact = 1 means there is an additional interaction relationship e u and ei, so that the user information graph and the newly added dimensions are integrated into a unified graph; S22, the multimodal information encoder takes the newly added dimension entity and the original information entity as input, and uses the entity encoder and the attention layer to learn a new entity representation for each entity. The new entity representation aggregates the information of adjacent entities while retaining its own information; S3, based on the DeepFM architecture, adds a self-attention mechanism to the deep neural network to improve the model's ability to learn key information; It includes the following sub-steps: S31. The prediction model introduces DNN to perform a fully connected layer cascade on the feature vector x after embedding coding, and establishes a regression or classification model; the output of each neuron in the DNN model is the corresponding linear weighted value of the previous layer of neurons after nonlinear mapping, that is, for the l+1 layer of neurons, its output corresponding value is: Where W (l) , a (l) and b (l) They represent the weight matrix of the lth layer, the output response of the lth neuron, and the bias value vector connecting the lth layer and the l+1th layer respectively; For nonlinear mapping function, Sigmoid function or ReLU function is used: S32. In the interest model, a multi-head attention network is used to jointly focus on information from different positions and different representation spaces to simulate user preferences from multiple interesting views; in the interest interaction layer, the sequence itself is used as the key and value vector of the data, and its output vector is aggregated from the previous hidden output of the neural network; S4. Output the results and judge the model accuracy through the binary cross entropy loss function Logloss and AUC.
2. The method for predicting interest in live shopping according to claim 1, Features The above step S1 specifically includes the following sub-steps: S11, using random samples in the video sequence for training, that is, extracting three frames from a certain frame and the previous frame as a training set, extracting three frames from the frame behind the frame as a test set, pooling the features of the extracted target area to obtain an initialized feature image, and generating a three-dimensional feature filter; optimizing the initialized filter in combination with the background information of the target area, and obtaining the optimized filter in an iterative manner; Weights are added to the loss, and referring to the Hinge Loss in SVM, a large number of negative samples in the score map are filtered out; for the positive sample area, MSE Loss is used, and the final residual function is as follows: r(s,c)=v c ·(m c s+(1-m c )max(0,s)-y c ) In the formula, the subscript c represents the degree of dependence on the center point; v c is the weight, m c (t)∈[0,1] is Mask, in the background area, m c ≈0, in the object corresponding area m c ≈1, Hinge Loss is used in the background area and MSE Loss is used in the object area; S12. Use the encapsulated data processing algorithm to track the items being sold during live broadcasting. It is necessary to establish a tracking frame for the user's viewpoint and the target area. When the target area covers the user's viewpoint, it is determined to be overlapped, that is, the user's viewpoint is focusing on the area during the corresponding time. Information is collected on the demonstrated items for sale, the live broadcast host, the background, the comment area, and the activity voucher area.
3. The method for predicting interest in live shopping according to claim 2, Features The above step S4 outputs the result and uses the binary cross entropy loss function Logloss and AUC to judge the model accuracy; where Logloss is defined as: N is the total number of samples in the test set, y (i) and are the true value of the category of the i-th sample in the test set and the probability value of the predicted user click; AUC is defined as the area enclosed by the coordinate axis under the ROC curve: Among them, R is the threshold, fpr is the false positive rate; different classification thresholds can obtain the true positive rate curve under different false positive rates, namely ROC.
