A method and system for monitoring Internet addiction behavior based on multimodal data fusion

Abstract

The present invention belongs to the field of computer vision and machine learning technology, and relates to a method and system for monitoring Internet addiction behavior based on multimodal data fusion. First, the multimodal data of the user is collected, including physiological signal data and behavioral posture data; then a multi-factor fusion judgment model based on deep learning is constructed, and the physiological signal data and behavioral posture data are input into the multi-factor fusion model for synchronous processing, and multimodal features are extracted and fused into a unified feature space. At the same time, a multi-task learning method is used to jointly train the physiological signal data and behavioral posture data; finally, a decision tree or random forest algorithm is used to classify and evaluate the user's addiction state. Compared with the existing Internet addiction behavior monitoring method, the present invention has shown significant advantages in accuracy, real-time, user privacy protection, environmental adaptability, user friendliness, technical implementation efficiency, generalization ability, comprehensive intervention ability, and end-side deployment ability.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and machine learning, and relates to a method and system for monitoring Internet addiction behavior based on multimodal data fusion. Background Art

[0002] With the popularization of the Internet and the abundance of digital entertainment content, Internet addiction has become a global social problem, especially among young people. Internet addiction not only affects academic performance and quality of life, but may also cause mental health problems. Traditional Internet addiction detection methods mainly rely on user behavior data analysis and psychological questionnaires, which have certain limitations:

[0003] Reliance on self-reports: Psychological questionnaires rely on users’ self-reports, which may be affected by subjective will and honesty and cannot accurately reflect the user’s addiction status.

[0004] Lack of real-time performance: Behavioral data analysis is usually based on historical records and cannot monitor users’ network usage behavior and psychological state in real time.

[0005] Privacy issues: Collecting user behavior data may involve privacy violations, and users may be resistant to data collection.

[0006] To overcome these limitations, we began to explore Internet addiction monitoring technologies based on physiological signals and behavioral characteristics. These technologies evaluate users' Internet usage status in a more objective way by analyzing their physiological reactions and facial expressions. However, existing implementations usually only focus on single-modal data, such as heart rate or facial expressions, and lack comprehensive analysis of multimodal data.

[0007] The closest implementation is:

[0008] Heart rate-based monitoring systems: Some studies use heart rate variability (HRV) to assess users' stress and concentration levels, thereby inferring Internet addiction. These systems monitor users' heart rates through wearable devices, but are limited by the portability of the devices and user acceptance.

[0009] Facial expression recognition system: By analyzing the changes in the user's facial expression to identify the emotional state, and then infer network usage behavior. Such systems usually require high-quality video input and complex image processing algorithms, and require high computing resources.

[0010] Eye tracking technology: Analyze attention distribution by tracking the user's eye movements. This method works well in laboratory environments, but is affected by ambient light and user head movements in actual applications. Summary of the invention

[0011] In order to solve the above technical problems existing in the prior art, the present invention proposes a method and system for monitoring Internet addiction behavior based on multimodal data fusion, and its specific technical solution is as follows:

[0012] A method for monitoring Internet addiction behavior based on multimodal data fusion, firstly collecting multimodal data of users, including physiological signal data and behavioral posture data;

[0013] Then, a multi-factor fusion judgment model based on deep learning is constructed, and the physiological signal data and the behavioral posture data are input into the multi-factor fusion model for synchronous processing, and multi-modal features are extracted and fused into a unified feature space. At the same time, a multi-task learning method is used to jointly train the physiological signal data and the behavioral posture data;

[0014] Finally, a decision tree or random forest algorithm is used to classify and evaluate the user's addiction status.

[0015] Further, the behavior posture data includes: data of eye gaze image and head posture image;

[0016] The multi-factor fusion model includes: a gaze estimation model and a posture estimation model;

[0017] The gaze estimation model extracts gaze direction information features of the eye gaze image using a self-adversarial learning method, and reconstructs the image based on the extracted gaze direction information features to obtain an image that retains the gaze direction information, and estimates the user's gaze direction at the same time;

[0018] The posture estimation model is used to extract facial key point information from the head posture image and estimate the facial key point heat map.

[0019] Furthermore, the gaze estimation model includes: a Backbone network, a SA-Module network and a multi-layer perceptron KAN network, wherein the Backbone network is responsible for extracting gaze direction information features of the image, the multi-layer perceptron KAN network is used to learn and estimate the user's gaze direction in the image, and the SA-Module network uses the extracted gaze direction information features to reconstruct the image, while the Backbone network uses adversarial loss to minimize the reconstruction ability of the SA-Module network, that is, the Backbone network learns to extract features related to gaze while removing features related to image reconstruction during the adversarial process.

[0020] Furthermore, the SA-Module network uses reconstruction loss for training optimization; the Backbone network uses the ResNet-50 network.

[0021] Furthermore, the posture estimation model is an HRnet model, which adopts a multi-scale fusion strategy, including multiple high-to-low resolution sub-networks and information exchange units connected in parallel. The multiple sub-networks process the input head posture image in parallel to extract feature maps of different resolutions, and perform repeated multi-scale information fusion through the information exchange unit to generate a rich high-resolution representation and obtain a facial key point heat map.

[0022] Furthermore, the HRnet model adopts a SLPT network, and the SLPT network includes: an inherent relationlayers layer and a prediction heads layer;

[0023] In the inherent relation layers, the relationship between landmarks is modeled. Each layer consists of three blocks: multi-head self-attention MSA block, multi-head cross-attention MCA block and multi-layer perceptron MLP block, and a layer normalization LayerNorm is applied before each block. Through the self-attention mechanism in the MSA block, the information of the query interacts to adaptively learn the query-query intrinsic relationship; the MCA block aggregates the representation of facial landmarks through the cross-attention mechanism to learn the adaptive representation-query relationship. By utilizing cross-attention, each landmark can make consistent predictions using neighboring landmarks, and the position of the occluded landmark can be predicted based on the representation of the visible landmark; the MLP block is used to further process the feature representation;

[0024] Two prediction heads are used in the prediction heads layer: a key point heat map prediction head and a key point offset prediction head. The key point heat map prediction head is used to predict the heat map of each landmark, where each pixel represents the probability of the landmark; the key point offset prediction head is used to predict the offset of each landmark relative to the center of its support patch.

[0025] Furthermore, the SLPT network introduces structured coding to retain the relative position information of facial key points, and encodes the distance between key points, that is, the distance between landmarks in the normal facial structure, by learning the similarity between the codings.

[0026] Furthermore, the HRnet model introduces the ELA local attention mechanism.

[0027] Furthermore, the decision tree algorithm adopts the LightGBM algorithm.

[0028] A network addiction behavior monitoring system based on multimodal data fusion, comprising:

[0029] Data acquisition module: including camera, heart rate sensor, blood pressure monitor and blood oxygen saturation sensor, used to collect the user's physiological and behavioral data in real time; using the camera to capture the user's facial image video, and using the sensor to monitor the heart rate, blood pressure and blood oxygen saturation in real time;

[0030] Data processing module: including preprocessing unit and feature extraction unit, used to process raw image data and extract effective features, preprocess video data, including noise reduction, illumination correction and facial key point detection, and filter and normalize physiological signal data;

[0031] Deep learning module: including convolutional neural network (CNN) for physiological signal analysis and recurrent neural network (RNN) for behavioral feature analysis. The neural network model is trained end-to-end using labeled training data sets, and cross-validation and hyperparameter optimization techniques are used to improve the generalization ability of the model.

[0032] Fusion and decision module: including feature fusion algorithm unit and decision tree or random forest algorithm unit, which are used to comprehensively analyze features and judge the user's addiction status; the extracted multimodal features are integrated through feature fusion algorithm, and the user's addiction status is classified and evaluated using decision tree or random forest algorithm model;

[0033] User Interface: A display interface that provides real-time feedback and intervention suggestions.

[0034] Compared with the prior art, the present invention has the following significant advantages:

[0035] 1. Multimodal data integration: By integrating multiple physiological signals and behavioral characteristics, it can provide a more comprehensive user status analysis and more accurately assess the user's addiction status compared to the monitoring method of a single data source.

[0036] 2. Real-time monitoring capability: The design focuses on real-time data processing, which can quickly respond to changes in user behavior and provide users with timely feedback and intervention, which is lacking in many existing technologies.

[0037] 3. High-precision evaluation: Using deep learning models to extract and fuse features from multimodal data improves the accuracy of addictive behavior identification and reduces the possibility of misjudgment and missed judgment.

[0038] 4. Privacy protection: User privacy is fully considered during the design, avoiding the collection of user sensitive data. At the same time, effective addiction monitoring is achieved, resolving the contradiction between privacy protection and behavior monitoring.

[0039] 5. Strong environmental adaptability: It optimizes the adaptability to environmental changes, can work stably under different lighting, noise and user movement conditions, and improves the reliability of monitoring.

[0040] 6. Good generalization ability: Training and testing on diverse data sets improves the application effect of the model in different populations and environments, and enhances the practicality of the system.

[0041] 7. End-side deployment capability: The models and algorithms are optimized to run on mobile devices or in home environments, making it easier for users to conduct self-monitoring and management anytime and anywhere.

[0042] To sum up, through technological innovation, the present invention has demonstrated significant advantages in accuracy, real-time, user privacy protection, environmental adaptability, user friendliness, technical implementation efficiency, generalization capability, comprehensive intervention capability and terminal deployment capability, providing a better solution for monitoring and intervening in Internet addiction behavior. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a schematic diagram of the framework structure of the gaze estimation model according to an embodiment of the present invention;

[0044] Figure 2 is a schematic diagram of a KAN network architecture according to an embodiment of the present invention;

[0045] Figure 3 is a schematic diagram of the SA-Module network structure of an embodiment of the present invention;

[0046] Figure 4 is a schematic diagram of a plurality of sub-network structures of different resolutions in the HRnet model of an embodiment of the present invention;

[0047] Figure 5 It is a schematic diagram of the SLPT network framework structure of an embodiment of the present invention;

[0048] Figure 6 is a diagram of a network structure of an SE block according to an embodiment of the present invention;

[0049] Figure 7 is a CA block network structure diagram of an embodiment of the present invention;

[0050] Figure 8 4 is a diagram of the ELA block network structure of an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical scheme and technical effect of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0052] The present invention aims at the logical causal relationship between physiological signals and the state of addiction, and analyzes based on existing data conditions: when an individual is addicted to online activities, he may experience higher levels of stress and excitement, which will lead to the activation of the sympathetic nervous system, thereby increasing the heart rate. Therefore, a high heart rate may be a sign of addictive behavior because it reflects the body's physiological response to continuous cognitive load and emotional stimulation. At the same time, individuals in an addicted state may experience an increase in psychological stress, which may cause blood pressure to rise to a certain extent. Continuous monitoring of blood pressure can be used to assess the stress level of individuals during Internet use. Long-term Internet use may cause individuals to maintain static postures, affecting the efficiency of the respiratory system, thereby reducing blood oxygen saturation. Therefore, a decrease in blood oxygen levels may be a physiological indicator of the body's response to long-term static behavior and hypoventilation. For eye gaze and head posture, users who are addicted to online activities will stare at the screen for a long time, which will cause tension in the neck and eye muscles. Therefore, the persistence of eye gaze and the stability of head posture can be used as indicators to assess individual concentration and the intensity of Internet use.

[0053] A single physiological signal may not be sufficient to accurately assess addictive behavior. Therefore, based on the above data analysis, it can be evaluated by integrating multiple physiological signals and behavioral characteristics. For example, an individual may show increased heart rate, increased blood pressure, decreased blood oxygen saturation, and fixed eye gaze and head posture during long-term Internet use. These multi-dimensional physiological and behavioral characteristics can confirm each other and more accurately judge the individual's addiction state.

[0054] Therefore, in order to achieve a comprehensive assessment of adolescent Internet addiction behavior, the present invention proposes a method for monitoring Internet addiction behavior based on multimodal data fusion. First, the multimodal data of the user is collected, including: physiological signals such as heart rate, blood pressure, blood oxygen, and eye gaze and head posture data in facial videos; then, by constructing a multi-factor fusion judgment model, the collected user multimodal data is synchronously processed through a unified timestamp to ensure the consistency of the multimodal data in the time dimension, and the data of these different modalities are extracted by using a deep learning network. The data is fused into a unified feature space through a feature fusion algorithm, and the user's addiction state is classified and evaluated using a decision tree or a random forest algorithm. Multimodal data fusion helps to capture the complex behavior patterns of users and improve the robustness of the model. Considering the complexity of addictive behavior, the present invention adopts a multi-task learning method to jointly train multiple tasks such as heart rate, blood pressure, blood oxygen, eye gaze, head posture, etc. By sharing model parameters, multi-task learning can effectively utilize the relevant information between different tasks and improve the prediction performance of each task. At the same time, the mutual constraints between tasks are utilized to enhance the model's ability to distinguish addictive behavior. During the model training process, cross-validation, data enhancement and other technologies are used to improve the generalization ability of the model. Through hyperparameter tuning and model selection, the structure and parameters of the model are optimized to ensure its stability and accuracy in different environments and conditions. In addition, the transfer learning method is combined with existing data sets and pre-trained models to further improve the performance of the model. After completing the model training and optimization, the user's addictive behavior is monitored in real time. The model comprehensively evaluates the user's addiction status based on the user's multimodal data, and outputs specific evaluation results and risk levels. The results can be used in a variety of application scenarios such as personal self-management, family supervision, and school intervention, providing a scientific basis for the identification and intervention of addictive behavior.

[0055] For the evaluation of the addiction state, the present invention uses a machine learning method to discriminate the addiction behavior. This embodiment uses multiple machine learning algorithm models to conduct experiments, and the experiments are compared through the mean square error (MSE) results after training, as shown in the experimental results in Table 1 below. MSE is a commonly used performance evaluation indicator in regression problems, which measures the average of the squares of the differences between the model prediction value and the actual value. The lower the MSE, the closer the model's prediction result is to the actual value, and the better the model's performance is.

[0056] Table 1:

[0057]

[0058] For the performance overview of the models in Table 1: First is LightGBM: MSE is 0.026, which is the best performance among the models. Following LightGBM is XgBoost, which has an MSE of 0.030 and performs quite well. RandomForestRegressor, GradientBoostingRegressor, and ExtraTreesRegressor have MSEs of 0.031, 0.033, and 0.032, respectively, which are slightly weaker than the first two.

[0059] Both LightGBM and XgBoost are optimization algorithms based on the gradient boosting framework. They both perform well in many regression problems, but LightGBM is slightly better in this comparison, probably because it is more efficient when processing large data and has better parallel processing capabilities.

[0060] RandomForestRegressor is an ensemble learning method that improves prediction accuracy by building multiple decision trees and taking the average of their prediction results. Its performance is slightly inferior to LightGBM and XgBoost, but it is still very effective in dealing with nonlinear relationships and feature interactions. Given that LightGBM performs best, the LightGBM model is selected, and the addiction factors are analyzed based on the feature influence results obtained by LightGBM ensemble learning to identify addiction behaviors.

[0061] For the above-mentioned eye gaze task learning, the existing methods that estimate the gaze direction by building geometric models of the eyes and head usually require accurate facial feature point annotations, such as the corners of the eyes, the center of the pupil, etc. This method performs well in a controlled environment, but in practical applications it may be affected by factors such as lighting and occlusion. Among the rich information in facial images, the features that are truly related to gaze only correspond to slight changes in the eye area, while other features that are not related to gaze, such as lighting, personal appearance, and even facial expressions, may affect learning in unexpected ways.

[0062] Therefore, the present invention designs a gaze estimation model to perform adversarial reconstruction on the gaze direction to eliminate the influence of noise on the gaze direction in actual situations. The gaze estimation model divides the gaze task into three subtasks, and the three subtasks correspond to a Backbone network for feature extraction, a multi-layer perceptron (KAN) for gaze estimation, and a SA-Module network for image reconstruction.

[0063] Specifically, the main framework of the gaze estimation model is to improve the accuracy of gaze estimation through two adversarial tasks, such as Figure 1As shown in Figure 2, the first task is to preserve the gaze information, and the second task is to remove the general image information from the extracted features. This self-adversarial learning allows the network to implicitly purify the gaze features during the optimization process without explicitly defining which features are irrelevant to the gaze.

[0064] A new neural network architecture KAN (Kolmogorov-Arnold Networks) is used in this gaze estimation model. Through this innovative alternative to improve the traditional MLP multi-layer perceptron, KAN is superior to MLP in accuracy and interpretability, and can outperform MLP with a large number of parameters with a small number of parameters. The core idea of ​​KAN is to move the activation function from the node to the edge, and these activation functions are learnable rather than fixed.

[0065] The design of the KAN is based on the Kolmogorov-Arnold representation theorem, which states that any continuous function can be approximated by a series of simpler functions through addition and nonlinear transformations. The design goal of KAN is to improve the expressiveness, accuracy, and interpretability of neural networks while reducing the number of parameters required. In traditional MLPs, activation functions are located on nodes, i.e. neurons, while in KANs, activation functions are placed on edges, i.e. weights. At the same time, the activation functions in KANs are learnable, which means that they can be adjusted based on the data, rather than using fixed activation functions like MLPs.

[0066] KAN has a multi-layer structure, and each layer consists of multiple 1D learnable functions. Layers are realized through the combination of these functions. The depth of the network can be expanded arbitrarily, while the width depends on the dimension of the input and its combination structure. Following the Kolmogorov-Arnold representation theorem, the input constructs the network through the combination of multiple 1D functions. By increasing the depth of the layers, the network can better learn complex function representations.

[0067] like Figure 2 As shown in Figure 1, the KAN network architecture has two main branches: the B-spline branch and the shortcut branch. The B-spline branch utilizes the B-spline activation function, while the shortcut branch is a combination of nonlinear activation and linear transformation. The forward equation of KAN can be expressed as In the B-spline branch, the parameters of each B-spline function are learned together with other network parameters, while the forward equation of MLP has the same form as the B-spline branch formula in KAN, with the main difference being the use of nonlinear functions. Each B-spline function in KAN has a set of learnable parameters that are updated during training through the back-propagation algorithm.

[0068] Specifically, the KAN network will train and fit the optimal activation function through the following steps:

[0069] First, each activation function of KAN is parameterized as a B-spline curve, and the coefficients of these splines are learnable. During the training process, the shape and parameters of the spline curve are constantly adjusted according to the data. During training, KAN uses a loss function to evaluate the gap between the model output and the true target. Commonly used loss functions are mean square error (MSE) or other loss functions suitable for specific tasks.

[0070] Through the back-propagation algorithm, the gradient of each parameter including the coefficient of the spline to the loss function is calculated, which allows KAN to update its parameters to minimize the loss. The training process includes calculating the network output, calculating the value of the loss function, calculating the gradient of the loss to each spline parameter through the chain rule, and finally updating the parameters.

[0071] KAN can dynamically adjust the grid of the spline, which means that during the training process, the spline can be made more refined, thereby improving the accuracy of the fit. This grid expansion allows the model to better capture complex features in the data when needed. To avoid overfitting, KAN can apply regularization techniques, such as L1 or L2 regularization. These techniques help control the complexity of the spline function so that it can fit the training data while maintaining good generalization capabilities.

[0072] Through multiple forward and back propagation, the parameters of KAN will gradually converge to the optimal value, and finally obtain the optimal activation function. This process will continue until the preset stopping criteria are reached, such as the value of the loss function no longer decreases significantly. Through the above steps, KAN can effectively train activation functions suitable for specific tasks, thereby improving the model's fitting ability and performance. This flexible parameterization method gives KAN a stronger advantage in dealing with complex function fitting problems.

[0073] At the same time, KAN can effectively overcome the "curse of dimensionality" in high-dimensional data through its structure. It can better learn the combinatorial structure in the data and can process high-dimensional data more effectively than MLP. Since the activation function of KAN is a learnable 1D function, these functions can be visualized as curves, providing an intuitive interpretation tool, which gives KAN a significant advantage in scientific discovery and physical modeling.

[0074] By introducing the KAN model architecture to replace the common MLP model, the model is more efficient in terms of parameters, and can achieve similar or better performance than MLP with fewer parameters. At the same time, the design of the KAN network takes into account the generalization ability of features, which also helps to improve the performance of the line of sight estimation method in unknown target domains, especially when evaluating across domains or across datasets. And based on the interpretability of the KAN network, the line of sight can be better learned and estimated, especially when debugging and improving the model, which can greatly save time.

[0075] The main purpose of the SA-Module is to reconstruct the original image as much as possible from the extracted features. If the image can be reconstructed from the features, it means that these features contain specific information about the image. Therefore, the design of the SA-Module aims to force the network to remove this non-attention-related information.

[0076] like Figure 3 As shown in the figure, SA-Module consists of multiple layers, including upsampling layers and convolution layers, which upsample the feature maps and gradually reconstruct the image. The model structure uses a total of five layers of SA-Module, which contain different numbers of feature maps, 256, 128, 64, 32, and 16 from the bottom to the top, and finally outputs a reconstructed image of the same size as the original image through a 1x1 convolution layer.

[0077] There is an adversarial relationship between the SA-Module network and the Backbone network. The Backbone network is responsible for extracting features, and the SA-Module tries to reconstruct the image using these features. The Backbone network learns how to extract features related to gaze while removing those related to image reconstruction in this adversarial process. To train the SA-Module, a reconstruction loss, such as pixel-level mean square error loss, is used to optimize the reconstruction task. At the same time, the Backbone network uses adversarial loss to minimize the reconstruction ability of the SA-Module, which forces the features extracted by the Backbone network to contain as little information as possible required for image reconstruction. This design helps to improve the generalization ability of the model in unknown target domains, thereby improving the performance of cross-domain gaze estimation.

[0078] The Backbone network retains gaze information while avoiding extracting image information when extracting features. The SA-Module attempts to reconstruct the image from the extracted features, while the Backbone network attempts to prevent such reconstruction. Through this adversarial learning, the features extracted ultimately remove general image information and only retain information about gaze direction, which can effectively improve gaze accuracy.

[0079] In a comparative experiment of this embodiment, the performance of different backbone networks such as ResNet-18 and ResNet-50, combined with different multi-layer perceptrons MLP and Kolmogorov-Arnold Networks KAN, on the MPiiGaze and Gaze360 datasets was evaluated. The experimental results are shown in Table 2 below. The model using ResNet-50 as the backbone generally outperforms the model using ResNet-18, which indicates that a deeper network structure helps to extract more complex features, thereby improving the accuracy of gaze estimation.

[0080] Table 2:

[0081]

[0082] On the MPiiGaze dataset, the gaze estimation error of Resnet-18-MLP is 11.8°. On the Gaze360 dataset, the error is 9.8°. Resnet-18-KAN uses KAN instead of MLP as the model for gaze estimation. On the MPiiGaze dataset, its error is 11.4°, slightly lower than the version using MLP. In this experiment, the Resnet-18-KAN network was not tested based on the Gaze360 dataset. Nevertheless, the performance improvement of ResNet-18-KAN on MPiiGaze suggests that the KAN structure may be more effective in feature extraction.

[0083] Resnet-50-MLP is a model that uses ResNet-50 as the backbone network and combines it with MLP. On the MPiiGaze dataset, its gaze estimation error is 11.3°, which is lower than Resnet-18-MLP. On the Gaze360 dataset, the error is 7.5°. Similar to Resnet-50-MLP, Resnet-50-KAN uses KAN instead of MLP. On the MPiiGaze dataset, its gaze estimation error is 10.8°, the lowest among all listed models.

[0084] In addition, the embodiment of the present invention also tests three existing gaze estimation models: Full-Face, CA-Net and ADL. Among them, the error of Full-Face on the Gaze360 dataset is 11.1°, which is higher than all models using ResNet. The error of CA-Net on the MPiiGaze dataset is 18.8°, which is significantly higher than ResNet-50-MLP and ResNet-50-KAN. The error of ADL on the MPiiGaze dataset is 11.3°, which is comparable to ResNet-50-MLP.

[0085] Therefore, it can be concluded that the model using ResNet-50 as the Backbone network generally has better performance than the model using ResNet-18, indicating that a deeper network structure helps to improve the accuracy of gaze estimation. The version using KAN as the gaze estimation model has better performance on the MPiiGaze dataset than the version using MLP, indicating that KAN is more effective in feature extraction.

[0086] Compared with existing model methods, ResNet-50-KAN shows lower error on the MPiiGaze dataset, which may indicate that it has stronger generalization ability.

[0087] In view of the excellent performance of ResNet-50-KAN on the MPiiGaze dataset, ResNet-50 will be given priority as the Backbone network in the gaze estimation task in the future, and KAN will be combined as the gaze estimation model.

[0088] The process of acquiring head posture angles usually includes: detection of facial key points and calculation of head posture angles. The deep learning network HRnet developed by scientists at Microsoft Research Asia is selected. The core feature of HRnet is that it always maintains high-resolution feature representation throughout the entire network structure and enhances the feature expression capability through multi-scale fusion.

[0089] The key points of the head in the image are obtained through the facial key point detection model, such as the corners of the eyes, the tip of the nose, the corners of the mouth, etc. These key points are used to describe the position and shape of the face in the image. The 2D image points are mapped to the 3D model points, and accordingly, the positions of these points are marked in the image as 2D image points. Correction is performed through the camera matrix, and the PnP algorithm is used to match the 3D model points with the 2D image points to solve the rotation vector and translation vector of the head. The rotation vector is then converted into a rotation matrix, and further converted into Euler angles. The final Pitch, Yaw, and Roll angles represent the rotation angles of the head on the X, Y, and Z axes, respectively. Pitch represents the angle of the head tilting up and down, Yaw represents the angle of the head turning left and right, and Roll represents the angle of the head tilting left and right.

[0090] The core design of HRNet is to maintain high-resolution representation throughout the network, rather than restoring from low resolution to high resolution as in traditional methods. This design enables the network to generate more accurate and spatially precise keypoint heatmaps.

[0091] like Figure 4As shown in the figure, HRNet adopts a multi-scale fusion strategy to construct multiple high-to-low resolution sub-networks, which are connected in parallel, processed in parallel by sub-networks of different resolutions in the network, and feature representations are enhanced through repeated multi-scale fusion. This design allows each high-resolution representation to repeatedly receive information from other parallel representations, thereby enriching the high-resolution representation. Each sub-network processes feature maps of different resolutions and fuses information through specific information exchange units. This parallel processing method helps to maintain high-resolution information while enhancing features using low-resolution information.

[0092] The information exchange unit is one of the key components of HRNet, which allows information exchange between sub-networks of different resolutions. The information exchange unit adjusts the size and number of channels of the feature map through upsampling, downsampling and 1x1 convolution operations to achieve effective information fusion between different resolutions. HRNet estimates the key point heat map directly from the high-resolution representation output by the network. This design simplifies the network structure, avoids the complex upsampling process, and improves the accuracy of key point positioning.

[0093] At the same time, HRNet provides network structures with different width configurations, such as HRNet-W32 and HRNet-W48, where the numbers 32 and 48 represent the number of channels of the high-resolution sub-network in the network. This design allows the network to be adjusted under different computing resources and performance requirements. Finally, it was tested on multiple benchmark datasets, including the COCO key point detection dataset and the MPII human pose dataset. The experimental results proved its effectiveness and superiority in the task of human pose estimation.

[0094] In addition to human pose estimation, the design of HRNet can also be extended to other dense prediction tasks, such as semantic segmentation, object detection, etc. Overall, HRNet provides an efficient and accurate solution in the field of human pose estimation through its innovative network structure and multi-scale fusion strategy.

[0095] like Figure 5 As shown in the figure, the backbone network based on HRnet is modified to obtain the network Sparse Local Patch Transformer (SLPT) of the new face alignment method. The SLPT network draws on the method of ViT (Vision Transformer), divides the image or feature map into a grid of size Pr×Pu, and cuts a fixed-size local patch for each key point as its support. These patches are adjusted to K×K size through linear interpolation and mapped to vectors through the CNN layer.

[0096] In order to preserve the relative position information of facial key points, SLPT introduces structural encoding, which encodes the distance between key points, that is, the distance between landmarks in the normal facial structure, by learning the similarity between encodings.

[0097] SLPT includes: inherent relation layers fixed structure layer and prediction heads prediction head layer. In the inherent relation layers, the relationship between landmarks is modeled. Each layer consists of three blocks: multi-head self-attention (MSA) block, multi-head cross attention (MCA) block and multi-layer perceptron (MLP) block, and a layer normalization LayerNorm (LN) is applied before each block. Through the self-attention mechanism in the MSA block, the information of the query interacts to adaptively learn the query-query intrinsic relationship. The MCA block aggregates the representation of facial landmarks through the cross attention mechanism to learn the adaptive representation-query relationship. By utilizing cross attention, each landmark can make consistent predictions using neighboring landmarks, and the position of the occluded landmark can be predicted based on the representation of the visible landmark. The MLP block is used to further process the feature representation. Two prediction heads are used in the prediction heads layer: the key point heat map prediction head and the key point offset prediction head. The key point heat map prediction head is used to predict the heat map of each landmark, where each pixel represents the probability of the landmark; the key point offset prediction head is used to predict the offset of each landmark relative to the center of its support patch. These two prediction heads are used together to achieve accurate positioning of key points.

[0098] In the experimental part, multiple public datasets were used for extensive experimental evaluation. The experimental results show that the SLPT network achieves excellent performance in the key point detection task, surpassing other advanced methods, verifying the effectiveness and robustness of the network. By combining the advantages of ViT and HRnet, using local and global feature information, modeling the relationship between landmarks, learning the intrinsic relationship between different key points on the face, generating a representation of each key point, and aggregating them through adaptive intrinsic relationships. Then, accurate key point positioning is achieved by predicting the sub-pixel coordinates of each key point in its corresponding local area.

[0099] In order to improve the performance of deep convolutional neural networks, an embodiment of the present invention introduces the Efficient Local Attention (ELA) local attention mechanism. Existing methods have difficulties in utilizing spatial information, or if they can be utilized, they will reduce the channel dimension or increase the complexity of the neural network. In order to solve these problems, by analyzing the limitations of the Coordinate Attention method, it was found that the generalization ability of batch normalization is insufficient, the adverse effect of dimensionality reduction on channel attention, and the complexity of the attention generation process. In order to overcome these challenges, the 1D convolution and GroupNormalization feature enhancement techniques are combined, and it is found that two 1D position feature maps can be effectively encoded to accurately locate the area of ​​interest without dimensionality reduction, while completing a lightweight implementation.

[0100] based on Figure 6 The SE block network structure diagram shown in Figure 7 The CA block network structure diagram shown in the figure, Figure 8 The ELA block network structure diagram shown in the figure can be used to compare and analyze the differences between the ELA block and the SE block (Squeeze-and-Excitation block) and the CA block (Coordinate Attention). Specifically, the analysis is carried out from the following aspects:

[0101] In terms of structural simplification and lightweight: the SE block compresses the spatial dimension to the channel dimension through 2D global pooling to enhance feature learning, but it only encodes the information between channels and ignores the spatial position information in the feature map; the CA block allows the mobile network to accurately capture long-distance spatial relationships by embedding spatial position information into the channel attention, but the design process of the CA block is relatively complex, involving multiple separation and merging of feature maps, and reducing the channel dimension during the 2D convolution process; ELA effectively encodes two 1D position feature maps through 1D convolution and group normalization (GN) feature enhancement technology without reducing the dimension, while maintaining the lightweight implementation of the attention module.

[0102] When encoding spatial information, SE blocks and CA blocks have limitations in processing spatial information. SE blocks do not consider spatial position information, while CA blocks consider spatial position information but reduce the channel dimension when processing. ELA processes feature vectors in the horizontal and vertical directions independently through 1D convolution, and then combines the attention predictions in these two directions through multiplication operations to accurately locate the spatial position of the region of interest.

[0103] In terms of the normalization strategy, the CA block uses Batch Normalization (BN), which may affect the generalization ability of the model in the case of small batch sizes. ELA uses Group Normalization (GN), which can maintain good performance in small batch sizes and has better generalization ability.

[0104] In addition, ELA carefully designs three hyperparameters and produces four different versions: ELA-T, ELA-B, ELA-S, and ELA-L to meet the specific needs of different visual tasks, such as image classification, object detection, and semantic segmentation. In contrast, the hyperparameter settings of SE blocks and CA blocks are relatively fixed and not flexible enough.

[0105] Overall, ELA provides an efficient and effective attention mechanism in multiple visual tasks through its innovative structure and hyperparameter design, significantly improving the performance of deep learning models.

[0106] In the comparative experiment of this embodiment, the performance of various facial key point detection models on the WFLW and 300W datasets is evaluated. These models include LAB, ADNet, SDL, SDFL, SAN, SLPT, HR-ELA-T and HR-ELA-B. The performance of all models is expressed as the mean mean error (NME) percentage, as shown in the experimental results in Table 3 below.

[0107] Table 3:

[0108]

[0109] LAB has an NME of 5.27% on the WFLW dataset and 2.98% on the 300W dataset. This indicates that LAB performs better on the 300W dataset, probably because the 300W dataset provides more diverse training samples, which helps the model generalize. ADNet has an NME of 4.14% on the WFLW dataset and 2.53% on the 300W dataset. ADNet outperforms LAB on both datasets, which may be attributed to ADNet's more effective feature extraction and key point localization strategy.

[0110] SDL achieves a NME of 4.21% on the WFLW dataset and 2.62% on the 300W dataset. SDL performs on par with ADNet but is slightly inferior to ADNet on the 300W dataset. SDFL achieves a NME of 4.35% on the WFLW dataset and 2.88% on the 300W dataset. SDFL performs slightly worse than SDL, which suggests that SDFL’s robustness in handling complex facial expressions and posture changes needs to be improved.

[0111] SAN has an NME of 5.22% on the WFLW dataset and 3.34% on the 300W dataset. SAN performs relatively poorly among all models, which may be related to its feature fusion strategy or keypoint localization algorithm. SLPT has an NME of 4.20% on the WFLW dataset and 2.75% on the 300W dataset. SLPT performs comparable to SDL, but outperforms SDL on the 300W dataset, showing better generalization ability.

[0112] Based on HRnet and combining four models with different parameters of SLPT attention mechanism and ELA attention mechanism, in the experimental results, HR-ELA-T has a NME of 4.14% on the WFLW dataset and a NME of 2.61% on the 300W dataset. The performance of HR-ELA-T is comparable to ADNet, but slightly inferior to ADNet on the 300W dataset. The NME of HR-ELA-B is 4.11% on the WFLW dataset and 2.55% on the 300W dataset. HR-ELA-B performs best among all models, indicating that it has a more efficient feature extraction and key point positioning mechanism. In addition, the two attention mechanisms of ELA-S and ELA-L did not show good performance in the experiment, so they were not shown in the control experiment.

[0113] From the above conclusions, we can see that HR-ELA-B shows lower NME on both datasets, indicating that it has higher accuracy and robustness in the facial key point detection task. ADNet and HR-ELA-B perform similarly on the 300W dataset, but HR-ELA-B performs slightly better than ADNet on the WFLW dataset, indicating that HR-ELA-B has better generalization ability when dealing with more diverse samples.

[0114] Finally, the multi-factor fusion judgment model of the present invention is deployed in the actual application system, and laboratory tests and field tests are carried out to verify its accuracy and stability in different environments and conditions. At the same time, user data is continuously collected and fed back to further optimize and improve the model to ensure its effectiveness and reliability in practical applications.

[0115] In combination with the above method, the present invention also proposes a network addiction behavior monitoring system based on multimodal data fusion, comprising:

[0116] Data acquisition module: including camera, heart rate sensor, blood pressure monitor and blood oxygen saturation sensor, used to collect the user's physiological and behavioral data in real time. Use high frame rate camera to capture the user's facial video, including eye gaze and head posture; use non-invasive sensors to monitor heart rate, blood pressure and blood oxygen saturation in real time.

[0117] Data processing module: including preprocessing unit and feature extraction unit, used to process raw data and extract effective features. Preprocess video data, including noise reduction, illumination correction and facial key point detection. Filter and normalize physiological signal data to reduce environmental interference.

[0118] Deep learning module: including convolutional neural network (CNN) for physiological signal analysis and recurrent neural network (RNN) for behavioral feature analysis. The neural network model is trained end-to-end using labeled training data sets, and cross-validation and hyperparameter optimization techniques are used to improve the generalization ability of the model.

[0119] Fusion and decision module: including feature fusion algorithm unit and decision tree or random forest algorithm unit, which is used to comprehensively analyze features and judge the user's addiction status. The extracted multimodal features are integrated through feature fusion algorithm, and the user's addiction status is classified and evaluated using decision tree or random forest algorithm model.

[0120] User Interface: A display interface that provides real-time feedback and intervention suggestions.

[0121] When a young user is playing games for a long time, the system captures his facial expressions and eye movements through the camera, while monitoring his heart rate and blood pressure changes. By analyzing these data through a deep learning model, the system determines that the user may be highly addicted, and prompts the user to take a break on the user interface and provides relaxation suggestions.

[0122] The above is only a preferred implementation case of the present invention and does not limit the present invention in any form. Although the implementation process of the present invention is described in detail above, for those familiar with the art, they can still modify the technical solutions recorded in the above examples, or replace some of the technical features therein with equivalents. All modifications, equivalent replacements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for monitoring Internet addiction behavior based on multimodal data fusion, characterized by: First, collect the user's multimodal data, including physiological signal data and behavioral posture data; Then, a multi-factor fusion model based on deep learning is constructed, and the physiological signal data and the behavioral posture data are input into the multi-factor fusion model for synchronous processing, multi-modal features are extracted and fused into a unified feature space, and a multi-task learning method is used to jointly train the physiological signal data and the behavioral posture data; Finally, the decision tree or random forest algorithm is used to classify and evaluate the user's addiction status; The behavior posture data includes: data of eye gaze image and head posture image; The multi-factor fusion model includes: a gaze estimation model and a posture estimation model; The gaze estimation model extracts gaze direction information features of the eye gaze image using a self-adversarial learning method, and reconstructs the image based on the extracted gaze direction information features to obtain an image that retains the gaze direction information, and estimates the user's gaze direction at the same time; The posture estimation model is used to extract facial key point information from the head posture image and estimate the facial key point heat map; The gaze estimation model includes: a Backbone network, an SA-Module network and a multi-layer perceptron KAN network. The Backbone network is responsible for extracting gaze direction information features of the image. The multi-layer perceptron KAN network is used to learn and estimate the user's gaze direction in the image, and the SA-Module network uses the extracted gaze direction information features to reconstruct the image. At the same time, the Backbone network uses adversarial loss to minimize the reconstruction ability of the SA-Module network, that is, the Backbone network learns to extract features related to gaze while removing features related to image reconstruction during the adversarial process.

2. The method for monitoring Internet addiction behavior according to claim 1, characterized in that: The SA-Module network uses reconstruction loss for training optimization; The Backbone network uses the ResNet-50 network.

3. The method for monitoring Internet addiction behavior according to claim 1, characterized in that: The posture estimation model includes an SLPT network and an HRnet model. The HRnet model adopts a multi-scale fusion strategy, including multiple parallel-connected high-to-low resolution sub-networks and information exchange units. The multiple sub-networks process the input head posture image in parallel to extract feature maps of different resolutions, and perform repeated multi-scale information fusion through the information exchange unit to generate a rich high-resolution representation and obtain a facial key point heat map.

4. The method for monitoring Internet addiction behavior according to claim 3, characterized in that: The SLPT network includes: Inherent relation layers and prediction heads layers; In the inherent relation layers, the relationship between landmarks is modeled. Each layer consists of three blocks: multi-head self-attention MSA block, multi-head cross-attention MCA block and multi-layer perceptron MLP block, and a layer normalization LayerNorm is applied before each block; through the self-attention mechanism in the MSA block, the information of the query interacts to adaptively learn the query-query intrinsic relationship; the MCA block aggregates the representation of facial landmarks through the cross-attention mechanism to learn the adaptive representation-query relationship. By utilizing cross-attention, each landmark can make consistent predictions using neighboring landmarks, and the position of the occluded landmark can be predicted based on the representation of the visible landmark; the MLP block is used to further process the feature representation; Two prediction heads are used in the prediction heads layer: a key point heat map prediction head and a key point offset prediction head. The key point heat map prediction head is used to predict the heat map of each landmark, where each pixel represents the probability of the landmark; the key point offset prediction head is used to predict the offset of each landmark relative to the center of its support patch.

5. The method for monitoring Internet addiction behavior according to claim 4, characterized in that: The SLPT network introduces structured coding to retain the relative position information of facial key points, and encodes the distance between key points, that is, the distance between landmarks in the normal facial structure, by learning the similarity between the codings.

6. The method for monitoring Internet addiction behavior according to claim 3, characterized in that: The pose estimation model introduces the ELA local attention mechanism.

7. The method for monitoring Internet addiction behavior according to claim 3, characterized in that: The decision tree algorithm adopts the LightGBM algorithm.

8. A network addiction behavior monitoring system based on multimodal data fusion using the method described in any one of claims 1 to 7, characterized in that: include: Data collection module: including camera, heart rate sensor, blood pressure monitor and blood oxygen saturation sensor, used to collect users' physiological and behavioral data in real time; Use the camera to capture the user's facial image video, and use sensors to monitor heart rate, blood pressure and blood oxygen saturation in real time; Data processing module: including preprocessing unit and feature extraction unit, used to process raw image data and extract effective features, preprocess video data, including noise reduction, illumination correction and facial key point detection, and filter and normalize physiological signal data; Deep learning module: including convolutional neural network (CNN) for physiological signal analysis and recurrent neural network (RNN) for behavioral feature analysis. The neural network model is trained end-to-end using labeled training data sets, and cross-validation and hyperparameter optimization techniques are used to improve the generalization ability of the model. Fusion and decision module: It includes a feature fusion algorithm unit and a decision tree or random forest algorithm unit, which are used to comprehensively analyze features and judge the user's addiction state; the extracted multimodal features are integrated through the feature fusion algorithm, and the user's addiction state is classified and evaluated using a decision tree or random forest algorithm model; User Interface: A display interface that provides real-time feedback and intervention suggestions.

Images