Data collection method and system for building a virtual digital platform

Abstract

The present invention discloses a data collection method and system for building a virtual digital platform, which specifically relates to the field of virtual digital interaction, and is used to solve the problems of user interest capture and cognitive fatigue. It accurately captures users' long-term interests in scenarios such as virtual exhibitions and online shopping platforms, and effectively copes with short-term cognitive fatigue. Through the fusion strategy of time-varying covariance matrix and graph neural network, the user's multimodal behavior data is monitored in real time, the modal weights are dynamically adjusted, and multidimensional features are integrated. Combining time series analysis with clustering algorithms, the system compares current behavior with historical baselines to accurately extract deviation data. Through deep learning models, the user's cognitive fatigue state is evaluated in real time, high-risk and low-risk feedback signals are generated, and recommendation optimization based on long-term interests is triggered. This technical solution ensures that the pushed content is consistent with the user's long-term needs, avoids the cognitive burden caused by information overload, improves the accuracy of recommendations and user experience, and reduces the churn rate.

Description

Technical Field

[0001] The present invention relates to the field of virtual digital interaction, and more specifically, to a data collection method and system for building a virtual digital platform. Background Art

[0002] In virtual digital scenarios such as virtual exhibitions and online shopping platforms, capturing user interests is crucial. Platforms usually rely on data such as user browsing time, click behavior, and interaction frequency to determine user interests and then push personalized content. When users browse exhibits and product details in a virtual exhibition hall, the system collects key data such as user stay time, click feedback, and interaction records to build a user interest model and recommend relevant content based on this model. However, the amount of information in virtual exhibitions is huge, and users are easily impacted by the visual and content of different exhibits, resulting in short-term cognitive fatigue, leading to inconsistency between behavior and interest expression. In this case, when the system recommends content in real time, it is easy to infer the user's interests based on short-term behavioral data, thereby ignoring their long-term needs and reducing the accuracy of personalized recommendations and user experience.

[0003] The current interest capture technology for virtual exhibitions or digital platforms usually makes judgments based on simple behavioral data. For example, a longer browsing time is considered to be a strong interest, and a high click frequency is inferred to be a strong demand for the user. The shortcoming of this technology is that it cannot effectively identify the user's short-term cognitive fatigue phenomenon. Specifically, when faced with information overload, users may show a short stay or repeatedly click on an exhibit. This behavior does not necessarily mean that the user is really interested, but is a change in behavioral habits due to visual or cognitive fatigue. Existing systems often cannot distinguish between this fatigue feedback and real interest, resulting in too frequent and in-depth push content, which not only fails to help users find their actual needs, but may also increase the user's cognitive burden and increase the churn rate.

[0004] In order to solve the above problems, a technical solution is now provided. Summary of the invention

[0005] In order to overcome the above-mentioned defects of the prior art, the embodiment of the present invention provides a data collection method and system for building a virtual digital platform. Through the adaptive fusion strategy of the time-varying covariance matrix and the graph neural network, the multimodal behavior data of the user is monitored in real time, the modal weight is dynamically adjusted and the multidimensional features are integrated to generate a unified user behavior feature vector. On this basis, the time series analysis and clustering algorithm are used to dynamically compare the current behavior with the historical baseline, accurately extract the user's behavior deviation, and combine the deep learning model to analyze the user's cognitive fatigue state in real time to generate high-risk and low-risk feedback signals. When the user has a behavioral deviation caused by cognitive fatigue, the recommendation optimization based on long-term interests is triggered by the high-risk signal to avoid the interference of short-term behavior, ensure that the pushed content not only meets the long-term needs of the user, but also avoids the cognitive burden caused by information overload on the user. The technical solution improves the accuracy of content recommendation, improves the overall experience of the user, effectively reduces the negative impact of cognitive fatigue on the user's interest expression and platform interaction effect, and thus improves the user's retention rate and satisfaction, so as to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] Collect multimodal data from users;

[0008] Generate a unified user behavior feature vector through the dynamic adaptive fusion strategy of the time-varying covariance matrix combined with the graph neural network;

[0009] Based on time series analysis and clustering algorithms, dynamic comparison of current behavior with historical baselines is performed to extract deviation data;

[0010] Extract user cognitive fatigue features, use deep learning models for real-time training and recognition, analyze user cognitive fatigue status changes in real time, and generate corresponding feedback signals, including high-risk signals and low-risk signals;

[0011] After obtaining high-risk signals, refer to long-term interest optimization recommendations and adjust push strategies.

[0012] In a preferred embodiment, the multimodal data of the user is collected, including the following:

[0013] In a virtual exhibition, users interact through devices and use the timestamp mechanism to synchronously mark the data collected by each device.

[0014] In a preferred embodiment, a unified user behavior feature vector is generated by combining a time-varying covariance matrix with a dynamic adaptive fusion strategy of a graph neural network, including the following contents:

[0015] After the multimodal data is synchronously collected, multimodal data fusion is performed to integrate data from different sources through a preset fusion strategy;

[0016] Use feature extraction algorithms to extract key features such as the duration of gaze and the frequency of gaze point switching from eye movement data; extract expression change features related to cognitive fatigue from facial expression data; and extract movement delays and irregular movement trajectories from hand movement data;

[0017] The adaptive fusion method based on dynamic context combines the time-varying covariance matrix to capture the temporal correlation of multimodal data, identifies the implicit feature interactions between modalities through graph neural networks, and dynamically adjusts the modal weights. The covariance matrix and GNN results are integrated through the weight adjustment formula, and the multimodal features are fused using a nonlinear weighting function to finally generate a user behavior feature vector.

[0018] In a preferred embodiment, based on time series analysis and clustering algorithms, the current behavior is dynamically compared with the historical baseline to extract deviation data, including the following:

[0019] The obtained user behavior feature vector is used as the current behavior data, and the behavior baseline model is constructed through long-term monitoring of the user's historical interaction behavior;

[0020] After building the behavior baseline, use the time series similarity measurement method to dynamically compare the current user behavior with the historical behavior baseline;

[0021] Based on the results of time series similarity measurement, the K-means clustering algorithm is used to perform cluster analysis on current behavior and historical behavior patterns;

[0022] When it is detected that the current behavior deviates from the historical baseline or a cluster center, the deviation data is extracted to provide real-time feedback on the degree and type of abnormal behavior;

[0023] Deviation data indicates the distance between the current behavior and the historical behavior baseline or cluster center, including behavioral feature deviation data and time series deviation data; the extracted deviation data is standardized.

[0024] In a preferred embodiment, extracting user cognitive fatigue features includes the following:

[0025] Covariance information and partial-ladder information are obtained based on the deviation data, and are used as input data into the long short-term memory network LSTM for modeling. The covariance information includes the time-frequency covariance index, and the partial-ladder information includes the local weighted deviation gradient.

[0026] In a preferred embodiment, the acquisition logic of the time-frequency co-variation index is:

[0027] First, the behavioral feature deviation data Perform short-time Fourier transform to obtain the spectrum information at different time points and obtain the time-frequency matrix: ;in, is the time-frequency matrix, is a sliding window function, is the frequency, is the time step; in the time-frequency matrix, select the main frequency components , calculate the covariability of these frequency components, the covariance index The calculation is based on the relative rate of change between the covariance of the time-frequency matrix and the frequency. The formula is as follows: ;in, represents the covariance of two frequency components in the time-frequency matrix, Indicates frequency The corresponding standard deviation, Indicates the weight adjustment caused by frequency difference; Indicates the total number of main frequency components selected in time-frequency analysis; and Indicates the first frequency component in the selected and frequency components.

[0028] In a preferred embodiment, the logic for obtaining the local weighted deviation gradient is:

[0029] Time series bias data Divide into length In each window, the deviation gradient change is calculated and each gradient value is weighted. The calculation formula is: ; Weight each local gradient value, and the weighted gradient calculation formula is: ;in, , that is, weighted according to the square value of the local deviation; Represents the index in the current local time window; calculates the volatility of the weighted gradient in the window, and the volatility calculation formula is: .

[0030] In a preferred embodiment, a deep learning model is used for real-time training and recognition, and real-time analysis of the user's cognitive fatigue state changes is performed to generate corresponding feedback signals, including the following:

[0031] The obtained time-frequency co-variation index and local weighted deviation gradient are input into the deep learning model for real-time training and recognition. The long short-term memory network is used for modeling, and then the recognition discrimination score is output. When the recognition discrimination score is greater than the discrimination threshold, a high-risk signal is generated; when the recognition discrimination score is less than or equal to the discrimination threshold, a low-risk signal is generated.

[0032] In a preferred embodiment, after obtaining a high-risk signal, the push strategy is adjusted with reference to the long-term interest optimization recommendation, including the following contents:

[0033] When a high-risk signal is obtained, based on long-term interest data, collaborative filtering algorithms or content-based recommendation methods are used to prioritize content related to the user's long-term interests.

[0034] The data collection system built for the virtual digital platform includes: data collection module, feature fusion module, deviation extraction module, cognitive analysis module and recommendation adjustment module;

[0035] Data collection module: collects multimodal data of users and passes the collected data to the feature fusion module;

[0036] Feature fusion module: Through the dynamic adaptive fusion strategy of the time-varying covariance matrix combined with the graph neural network, it monitors and analyzes multimodal data in real time, dynamically adjusts the modal weights, nonlinearly fuses features, generates a unified user behavior feature vector, and passes the obtained user behavior feature vector to the deviation extraction module;

[0037] Deviation extraction module: Based on time series analysis and clustering algorithms, it dynamically compares current behavior with historical baselines, extracts deviation data, and passes the deviation data to the cognitive analysis module;

[0038] Cognitive analysis module: extracts user cognitive fatigue characteristics, uses deep learning models for real-time training and recognition, analyzes changes in user cognitive fatigue status in real time, generates corresponding feedback signals, including high-risk signals and low-risk signals, and passes high-risk signals to the recommendation adjustment module;

[0039] Recommendation adjustment module: After obtaining high-risk signals, optimize recommendations based on long-term interests and adjust push strategies.

[0040] The technical effects and advantages of the data collection method and system for building a virtual digital platform of the present invention are as follows:

[0041] The present invention uses an adaptive fusion strategy of a time-varying covariance matrix and a graph neural network to monitor the user's multimodal behavior data in real time, dynamically adjust the modal weights and fuse multidimensional features to generate a unified user behavior feature vector. On this basis, time series analysis and clustering algorithms are used to dynamically compare current behaviors with historical baselines, accurately extract user behavior deviations, and combine deep learning models to analyze the user's cognitive fatigue state in real time, generating high-risk and low-risk feedback signals. When a user has behavioral deviations caused by cognitive fatigue, a high-risk signal is used to trigger recommendation optimization based on long-term interests to avoid interference from short-term behaviors, ensuring that the pushed content meets the user's long-term needs while avoiding the cognitive burden on the user caused by information overload. This technical solution improves the accuracy of content recommendations, improves the overall user experience, and effectively reduces the negative impact of cognitive fatigue on user interest expression and platform interaction effects, thereby improving user retention and satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A flow chart of a data collection method for a virtual digital platform according to the present invention.

[0043] Figure 2 It is a structural schematic diagram of the data acquisition system built for the virtual digital platform of the present invention. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] Embodiment 1: Figure 1 The present invention provides a data collection method for building a virtual digital platform, including:

[0046] Collect multimodal data from users.

[0047] Through the dynamic adaptive fusion strategy of the time-varying covariance matrix combined with the graph neural network, multimodal data can be monitored and analyzed in real time, modal weights can be dynamically adjusted, and nonlinear fusion features can be generated to generate a unified user behavior feature vector.

[0048] Based on time series analysis and clustering algorithms, current behavior is dynamically compared with historical baselines to extract deviation data.

[0049] Extract user cognitive fatigue characteristics, use deep learning models for real-time training and recognition, analyze changes in user cognitive fatigue status in real time, and generate corresponding feedback signals, including high-risk signals and low-risk signals.

[0050] After obtaining high-risk signals, refer to long-term interest optimization recommendations and adjust push strategies.

[0051] Collect multimodal data from users, including the following:

[0052] In virtual exhibitions, users interact through virtual reality helmets, hand controllers, cameras and other devices. First, it is necessary to clarify the source of multimodal data collection. These data include but are not limited to:

[0053] Eye movement data: Through the eye tracking sensors in the virtual reality helmet (such as infrared eye tracking technology), the user's gaze focus, gaze time and eye movement speed are monitored in real time.

[0054] Facial expression data: The camera is used to capture the user's facial expression changes, especially micro-expressions, such as eyebrow raising, eyelid drooping, etc. Convolutional neural network (CNN) is used to identify and analyze facial feature points.

[0055] Hand motion data: Capture the speed, direction, and frequency of the user’s hand motions through a hand controller or somatosensory device (such as Leap Motion).

[0056] In order to ensure the synchronization and accuracy of the data, multimodal data synchronization acquisition technology needs to be adopted, and the data collected by each device should be synchronized and marked using the timestamp mechanism to ensure the time consistency between different data sources.

[0057] Timestamp recording: Each data collection device (such as eye tracking sensors, cameras, etc.) generates behavioral data with timestamps in real time, ensuring that the system can synchronously process input information from multiple devices.

[0058] Preprocessing steps: Noise filtering and abnormal data removal are performed on the collected raw data. For jitter in eye movement trajectories and occlusion data in facial expressions, low-pass filtering algorithms and abnormality detection models are used to remove invalid information to ensure data accuracy.

[0059] Through the time-varying covariance matrix combined with the dynamic adaptive fusion strategy of the graph neural network, a unified user behavior feature vector is generated, including the following:

[0060] After the multimodal data is synchronously collected, multimodal data fusion is performed to integrate data from different sources through a preset fusion strategy in order to more accurately analyze the user's status.

[0061] Feature extraction algorithm: Use feature extraction algorithms (such as PCA principal component analysis) to extract key features such as gaze dwell time and gaze point switching frequency from eye movement data; extract expression change features related to cognitive fatigue from facial expression data; extract movement delays and irregular movement trajectories from hand movement data.

[0062] Data fusion strategy: Adaptive fusion method based on dynamic context, combined with time-varying covariance matrix to capture the temporal correlation of multimodal data, identify implicit feature interactions between modalities through graph neural network (GNN), and dynamically adjust modal weights. The covariance matrix and GNN results are integrated through the weight adjustment formula, and multimodal features are fused using nonlinear weighting functions to finally generate user behavior feature vectors. This fusion method ensures that the temporal correlation and nonlinear characteristics of multimodal data are fully integrated, improving the flexibility and accuracy of the multimodal fusion process. It includes the following:

[0063] a1, assuming that multimodal data have modality data sources (e.g., eye movements, facial expressions, hand movements, etc.), for the time window, the acquired modal data can be expressed as a matrix , where each column represents a data sequence of a different mode. In order to obtain the correlation between modes, we first need to construct a time-varying covariance matrix : ;

[0064] in:

[0065] It is samples in the time window Modal data within;

[0066] It is the time window Mean vector of modal data within ;

[0067] is the number of samples in the window;

[0068] Is the size of The covariance matrix describes the coordinated changes between the modal data.

[0069] Each element in the covariance matrix Representing modality and modal In the time window Through the covariance matrix, the temporal correlation between modalities is determined, providing a basis for the subsequent fusion weights.

[0070] a2, constructing a graph from multimodal data ,in Represents a modal node, Represents the associated edge between modes, and the initial edge weight is determined by the correlation in the covariance matrix Setting. Capture the implicit correlation between modal features through graph neural networks.

[0071] The propagation and node update rules of GNN are: ;

[0072] in:

[0073] Representation Node In the Feature vectors in layer propagation.

[0074] Is a node The set of neighbor nodes.

[0075] It is Layer, node Its neighbor nodes The attention weight between is calculated by the following formula: ;

[0076] It is The learnable parameter matrix of the layer.

[0077] is the weight vector in the attention mechanism.

[0078] is the activation function (such as ReLU).

[0079] Representation and Node Another neighboring node that is different.

[0080] After multiple layers of propagation, GNN generates the final feature vectors of each modal node. These vectors contain the implicit association information between modal data, and finally obtain the GNN implicit association features. .

[0081] a3, through the time-varying covariance matrix And GNN implicit correlation features , dynamically adjust the fusion weight for each modality data. The specific weight adjustment formula is as follows: ;

[0082] in:

[0083] is modal In the time window The final fusion weight within.

[0084] , There are two adjustment parameters that control the influence ratio of the covariance matrix and GNN.

[0085] is modal The final feature vector after propagation in the GNN.

[0086] It is a learnable parameter matrix used to convert GNN feature vectors into fusion weights.

[0087] The Sigmoid function is used to normalize the results of GNN to the interval [0,1] to ensure the rationality of the fusion weights.

[0088] Finally, the adaptive weights of each modality in different time windows are obtained by nonlinear weighted fusion of the time-varying covariance matrix and GNN features.

[0089] a4, with adaptive weights , nonlinearly fuse the features of each modality to generate a unified user behavior feature vector : ;

[0090] in:

[0091] is modal In the time window The original feature vector in .

[0092] is modal Nonlinear feature conversion functions (such as multi-layer perceptron MLP) are used to extract high-order features.

[0093] The function is used to activate the final feature vector to ensure that the fused feature vector remains within the appropriate range.

[0094] Through the processing of steps a1-4, the weights of multimodal data can be dynamically adjusted, and a unified user behavior feature vector can be generated in a nonlinear way, ensuring that the fusion process can fully capture the complex correlations between modalities and changes in behavior patterns.

[0095] After multimodal data fusion, the user's behavior status is dynamically monitored based on the real-time collected data, and the user's behavior pattern is continuously updated through a real-time analysis feedback mechanism.

[0096] Real-time behavior model update: The user behavior model is updated in real time through online learning algorithms (such as recurrent neural networks (RNNs)) to ensure that every user behavior is immediately reflected in the system's behavior analysis results.

[0097] By collecting multimodal data of users, it is possible to monitor the changes in user behavior in real time and accurately capture key data such as user eye movement trajectories, facial expressions, and hand movements to ensure comprehensive perception of user status. The dynamic adaptive fusion strategy that combines the time-varying covariance matrix and the graph neural network can not only identify the implicit associations between modal data, but also dynamically adjust the fusion weights of modal data through nonlinear weighting functions to achieve comprehensive integration of user behavior characteristics. After processing and fusing multimodal data, it is possible to effectively identify user behavior deviations and cognitive fatigue states, and update user behavior models in real time through online learning algorithms, thereby achieving high-precision behavior monitoring and accurate feedback in interactive scenarios such as virtual exhibitions, improving user experience while ensuring the intelligence and flexibility of the system.

[0098] Based on time series analysis and clustering algorithms, the current behavior is dynamically compared with the historical baseline to extract deviation data, including the following:

[0099] The obtained user behavior feature vector is used as the current behavior data, and the behavior baseline model is constructed through long-term monitoring of the user's historical interaction behavior. The historical behavior baseline extracts the user's typical behavior pattern through time series analysis and constructs a reference baseline with time series characteristics. Historical data can be divided by the sliding window method, and the behavior characteristics in each time window are stored as a sequence for subsequent dynamic comparison. The size of the time window is dynamically adjusted according to the interaction frequency and complexity of different scenarios to ensure that the sensitivity of the behavior feature changes is consistent with the baseline.

[0100] After building the behavior baseline, the time series similarity measurement method is used to dynamically compare the current user behavior with the historical behavior baseline. The Dynamic Time Warping (DTW) algorithm is used to measure the similarity between the current behavior and the historical behavior sequence. DTW measures the similarity of two sequences by calculating the minimum cumulative distance between different time steps, thereby avoiding errors caused by small offsets on the time axis.

[0101] Based on the results of time series similarity measurement, the K-means clustering algorithm is used to perform cluster analysis on the current behavior and historical behavior patterns. First, the historical behavior data is divided into multiple behavior patterns, each of which represents the user's interactive behavior in a specific scenario. Then, the current behavior characteristics are compared with the cluster center, and the clustering results are used to identify whether the current behavior deviates from the historical pattern. If the current behavior vector is too far from a cluster center, it indicates that the current behavior is abnormal or deviated, and the corresponding deviation information is further extracted.

[0102] When it is detected that the current behavior deviates from the historical baseline or a cluster center, the degree and type of abnormal behavior are fed back in real time by extracting deviation data.

[0103] Deviation data represents the distance between the current behavior and the historical behavior baseline or cluster center. It is an important input for identifying user cognitive fatigue and includes behavioral feature deviation data and time series deviation data. First, the extracted deviation data is standardized to ensure that the deviation values ​​in different time windows are compared on the same scale.

[0104] Extract user cognitive fatigue features, including the following:

[0105] When extracting the cognitive fatigue characteristics of users, traditional time series analysis methods can only identify basic behavioral deviations and have difficulty capturing the multidimensional changes in user behavior in time and frequency. The manifestation of cognitive fatigue is usually not a single behavioral deviation, but is accompanied by coordinated changes in the time and frequency dimensions, as well as the volatility of local behavior. Therefore, when extracting fatigue characteristics, it is necessary to introduce characteristic indicators suitable for describing such coordinated changes and local fluctuations.

[0106] The time-frequency covariation index can reflect the coordinated changes in user behavior between different frequencies and time periods, revealing the fluctuation trend of user behavior across frequency dimensions. The local weighted deviation gradient describes the local fluctuation characteristics of behavior by analyzing the change rate and volatility of user behavior in a short-term time window. These two features can more comprehensively reflect the changes in user behavior patterns under cognitive fatigue.

[0107] Using these features as input data can provide the LSTM model with richer behavioral information, enabling it to effectively identify user fatigue trends when processing time series. This feature extraction method not only enhances the accuracy of detecting cognitive fatigue states, but also provides a more detailed understanding of user behavior changes.

[0108] Based on the deviation data, covariation information and partial ladder information are obtained, and the covariation information and partial ladder information are used as input data to the long short-term memory network LSTM for modeling to capture the gradual changes of user cognitive fatigue. Among them, the covariation information includes the time-frequency covariation index, and the partial ladder information includes the local weighted deviation gradient.

[0109] The time-frequency co-variation index is used to quantify the synchronous fluctuations of user behavior characteristics in the time domain and frequency domain, and to capture the frequency change trend of behavioral deviations across time periods.

[0110] The local weighted deviation gradient is used to measure the gradient change rate of user behavior deviation within a short-term time window, which can capture the local nonlinear changes of behavior fluctuations.

[0111] The logic for obtaining the time-frequency covariation index is:

[0112] First, the behavioral feature deviation data Perform short-time Fourier transform (STFT) to obtain spectrum information at different time points and obtain the time-frequency matrix: ;

[0113] in, is the time-frequency matrix, is a sliding window function, is the frequency, is the time step.

[0114] In the time-frequency matrix, select the main frequency components , calculate the covariability of these frequency components. Covariance index The calculation is based on the relative rate of change between the covariance of the time-frequency matrix and the frequency. The formula is as follows: ;

[0115] in, represents the covariance of two frequency components in the time-frequency matrix, Indicates frequency The corresponding standard deviation, Indicates the weight adjustment caused by frequency difference.

[0116] Represents the total number of main frequency components selected in the time-frequency analysis. Represents the number of frequencies used to calculate the covariation index. That is, the number of frequency components selected in the entire time-frequency matrix.

[0117] and Indicates the first frequency component in the selected and frequency components. It is used to represent the index of two different frequency components in the time-frequency matrix. In the process of calculating the covariance index, and Used to compare the covariance and standard deviation between two frequency components and calculate co-variation.

[0118] The time-frequency co-variation index can capture the degree of cross-time co-variation of user behavior at different frequencies. A higher time-frequency co-variation index value indicates that the cross-frequency co-variation of behavioral deviations is more significant, and is suitable as an LSTM input for identifying fatigue trends.

[0119] The logic for obtaining the local weighted deviation gradient is:

[0120] Time series bias data Divide into length In each window, the deviation gradient change is calculated and each gradient value is weighted. Deviation gradient within the window The calculation formula is: ;

[0121] Weight each local gradient value, weight It is determined by the squared difference of the deviation within the window, and the moment with larger deviation is given higher weight. The weighted gradient calculation formula is: ;

[0122] in, , that is, weighting is performed according to the square value of the local deviation to ensure that the time point with a larger deviation value contributes more to the gradient; Represents the index within the current local time window.

[0123] The volatility of the weighted gradient within the window is further calculated to measure the severity of the deviation change. The volatility calculation formula is: ;

[0124] Local weighted deviation gradient It can reflect the degree of dramatic changes in short-term behavioral deviations. A larger value indicates a stronger fluctuation in the behavioral deviation within a local time period. It is suitable as an LSTM input for predicting user cognitive fatigue.

[0125] Use deep learning models for real-time training and recognition, analyze the user's cognitive fatigue status changes in real time, and generate corresponding feedback signals, including the following:

[0126] The obtained time-frequency co-variation index and local weighted deviation gradient are input into the deep learning model for real-time training and recognition. The long short-term memory network (LSTM) is used for modeling. LSTM has the ability to process long sequence dependencies and can capture the gradual changes in user cognitive fatigue.

[0127] The hidden layer of LSTM can store the long-term and short-term trend of time series through memory units.

[0128] The output layer of LSTM generates a recognition discrimination score, which determines in real time whether the user is in a state of cognitive fatigue based on the cumulative effect of time series features.

[0129] After the LSTM model outputs the recognition and distinction score, different feedback strategies are set for different recognition and distinction scores. When the score is high, rest suggestions or simplified interface interactions can be pushed. When the recognition and distinction score is in a critical state, the system will dynamically adjust the content presentation format, such as reducing the push of deep content and reducing the complexity of information, to reduce the cognitive burden on users.

[0130] The fatigue feedback process can be described as:

[0131] When the recognition distinction score is greater than the distinction threshold, it indicates that the user is in a highly deviant state and may be severely fatigued or have abnormal behavior, generating a high-risk signal, indicating that appropriate operational adjustments need to be taken immediately.

[0132] When the identification and differentiation score is less than or equal to the differentiation threshold, it indicates that the user behavior is normal without obvious deviation, generating a low-risk signal, and maintaining the current interaction status without additional adjustments.

[0133] By calculating the time-frequency co-variation index and local weighted deviation gradient based on deviation data, the user's cognitive fatigue characteristics can be extracted. This can capture the multi-dimensional changes in user behavior from the time and frequency dimensions, and identify the coordinated fluctuations and local drastic changes in user behavior deviations. Combined with the modeling capabilities of long-term and short-term memory networks, real-time analysis of user fatigue status and capture of progressive changes are achieved. By generating feedback signals for different recognition and differentiation scores, the interactive content is dynamically adjusted to help users reduce cognitive burden and improve the intelligence level of the overall interactive experience. This combination of feature extraction and real-time feedback mechanism ensures accurate perception and efficient feedback of user behavior status, which helps optimize user experience.

[0134] After obtaining high-risk signals, refer to the long-term interest optimization recommendations and adjust the push strategy, including the following:

[0135] When a high-risk signal is obtained, the long-term interest features in the user's historical interaction data are called. Long-term interests are usually extracted through analysis of users' long-term behavioral data, mainly including users' stable preferences in past interactions, frequently visited content categories, and long-term topics of concern. Long-term interests are obtained through time aggregation analysis of behavioral features to avoid interference from short-term behavioral deviations.

[0136] Based on long-term interest data, collaborative filtering algorithms or content-based recommendation methods are used to prioritize content related to users’ long-term interests. This recommendation mechanism is accomplished through the following two steps:

[0137] Based on the user's historical behavior and the behavior of other similar users, recommend the content that similar users have been paying attention to for a long time. The calculation formula of collaborative filtering is as follows: ;

[0138] in, Indicates user About content Interest score prediction, is the overall score mean, and are the deviation values ​​of users and content respectively, Is a user With users The similarity of is the rating of the content by similar users.

[0139] Based on the content categories that users have paid attention to in the past, content features (such as keywords, topics, etc.) are extracted to recommend content of similar categories to users. The content recommendation formula is: ;

[0140] in, It is the content and The similarity between It is the content In Features The feature weights on is the weight of the feature.

[0141] Under high-risk signals, in order to avoid information overload and create additional cognitive burden on users, the pushed content needs to be simplified and the push rhythm adjusted. Simplification strategies include reducing the depth and complexity of information, giving priority to lightweight and easy-to-understand content, and extending the interval between pushes to prevent users from receiving too much information in a short period of time.

[0142] After adjusting the content push, dynamic feedback tracking is performed based on the user's real-time interactive behavior. By monitoring the user's clicks, browsing time, interaction frequency and other behaviors, the content presentation method is further adjusted to ensure that the pushed content is in line with the user's current cognitive state and long-term interest preferences. For example, if the user shows a high degree of engagement with the simplified content pushed, the current strategy is maintained; if the user shows difficulty in processing information, the content is further simplified or the push frequency is reduced.

[0143] By adjusting the push strategy based on high-risk signals and long-term interests, we can effectively avoid the user's cognitive burden caused by information overload, while ensuring that the push content is highly consistent with the user's long-term preferences. By simplifying the push content and extending the push rhythm, the user's instant information processing pressure is reduced and the interactive experience is optimized. The dynamic content adjustment mechanism based on real-time feedback ensures that the push strategy can flexibly adapt to the user's current status, further improving the accuracy of information recommendations and user satisfaction. This strategy not only maintains the user's long-term interest, but also improves the acceptability of the push content and the smoothness of interaction.

[0144] Embodiment 2: Figure 2 The present invention provides a data acquisition system for building a virtual digital platform, including: a data acquisition module, a feature fusion module, a deviation extraction module, a cognitive analysis module and a recommendation adjustment module.

[0145] Data collection module: collects multimodal data of users and passes the collected data to the feature fusion module.

[0146] Feature fusion module: Generates a unified user behavior feature vector through the time-varying covariance matrix combined with the dynamic adaptive fusion strategy of the graph neural network, and passes the obtained user behavior feature vector to the deviation extraction module.

[0147] Deviation extraction module: Based on time series analysis and clustering algorithms, it dynamically compares current behavior with historical baselines, extracts deviation data, and passes the deviation data to the cognitive analysis module.

[0148] Cognitive analysis module: extracts user cognitive fatigue characteristics, uses deep learning models for real-time training and recognition, analyzes changes in user cognitive fatigue status in real time, generates corresponding feedback signals, including high-risk signals and low-risk signals, and passes high-risk signals to the recommendation adjustment module.

[0149] Recommendation adjustment module: After obtaining high-risk signals, optimize recommendations based on long-term interests and adjust push strategies.

[0150] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0151] The above description is only by way of illustration of certain exemplary embodiments of the present invention. It is undoubted that those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

[0152] It should be noted that, in this article, if there are relational terms such as first and second, etc., they are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including a..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0153] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data collection method for building a virtual digital platform, characterized in that: Includes steps: Collect multimodal data from users; Generate a unified user behavior feature vector through the dynamic adaptive fusion strategy of the time-varying covariance matrix combined with the graph neural network; Based on time series analysis and clustering algorithms, dynamically compare current behavior with historical behavior baselines to extract deviation data; Extract user cognitive fatigue features, use deep learning models for real-time training and recognition, analyze user cognitive fatigue status changes in real time, and generate corresponding feedback signals, including high-risk signals and low-risk signals; Extract user cognitive fatigue features, including the following: Covariance information and partial-ladder information are obtained based on deviation data, and the covariance information and partial-ladder information are used as input data to the long short-term memory network LSTM for modeling, wherein the covariance information includes the time-frequency covariance index, and the partial-ladder information includes the local weighted deviation gradient; The logic for obtaining the time-frequency covariation index is: First, the behavioral feature deviation data Perform short-time Fourier transform to obtain the spectrum information at different time points and obtain the time-frequency matrix: ;in, is the time-frequency matrix, is a sliding window function, f is the frequency, n is the time step; in the time-frequency matrix, select the main frequency components , calculate the covariability of these frequency components, the covariance index The calculation is based on the relative rate of change between the covariance of the time-frequency matrix and the frequency. The formula is as follows: ;in, represents the covariance of two frequency components in the time-frequency matrix, Indicates frequency The corresponding standard deviation, Indicates the weight adjustment caused by frequency difference; Indicates the total number of main frequency components selected in time-frequency analysis; and Indicates the first frequency component in the selected frequency component and frequency components; The logic for obtaining the local weighted deviation gradient is: Time series bias data Divide into short-term windows of length W. In each window, calculate the deviation gradient change and weight each gradient value. The calculation formula is: ; Weight each local gradient value, and the weighted gradient calculation formula is: ;in, , that is, weighted according to the square value of the local deviation; Represents the index in the current local time window; calculates the local weighted deviation gradient in the window, and the calculation formula is: ; Use deep learning models for real-time training and recognition, analyze the user's cognitive fatigue status changes in real time, and generate corresponding feedback signals, including the following: The obtained time-frequency covariation index and local weighted deviation gradient are input into the deep learning model for real-time training and recognition, and the long short-term memory network is used for modeling, and then the recognition and discrimination score is output; when the recognition and discrimination score is greater than the discrimination threshold, a high-risk signal is generated; when the recognition and discrimination score is less than or equal to the discrimination threshold, a low-risk signal is generated; After obtaining high-risk signals, refer to long-term interest optimization recommendations and adjust push strategies.

2. The data collection method for building a virtual digital platform according to claim 1, characterized in that: Collect multimodal data from users, including the following: In a virtual exhibition, users interact through devices and use the timestamp mechanism to synchronously mark the data collected by each device.

3. The data collection method for building a virtual digital platform according to claim 2, characterized in that: Through the dynamic adaptive fusion strategy of the time-varying covariance matrix combined with the graph neural network, a unified user behavior feature vector is generated. Includes the following: After the multimodal data is synchronously collected, multimodal data fusion is performed to integrate data from different sources through a preset fusion strategy; Use feature extraction algorithms to extract key features such as gaze dwell time and gaze point switching frequency from eye movement data; Extracting expression change features related to cognitive fatigue from facial expression data; Extracting motion delays and irregular motion trajectories from hand motion data; The adaptive fusion method based on dynamic context combines the time-varying covariance matrix to capture the temporal correlation of multimodal data, identifies the implicit feature interactions between modalities through graph neural networks, and dynamically adjusts the modal weights; The covariance matrix and GNN results are integrated through the weight adjustment formula, and the multimodal features are fused using the nonlinear weighting function to finally generate the user behavior feature vector.

4. The data collection method for building a virtual digital platform according to claim 3, characterized in that: Based on time series analysis and clustering algorithms, the current behavior is dynamically compared with the historical behavior baseline to extract deviation data, including the following: The obtained user behavior feature vector is used as the current behavior data, and the behavior baseline model is constructed through long-term monitoring of the user's historical interaction behavior; After building the behavior baseline model, use the time series similarity measurement method to dynamically compare the current user behavior with the historical behavior baseline; Based on the results of time series similarity measurement, the K-means clustering algorithm is used to perform cluster analysis on current behavior and historical behavior patterns; When it is detected that the current behavior deviates from the historical behavior baseline or a cluster center, the deviation data is extracted to provide real-time feedback on the degree and type of abnormal behavior; Deviation data indicates the distance between the current behavior and the historical behavior baseline or cluster center, including behavioral feature deviation data and time series deviation data; the extracted deviation data is standardized.

5. The data collection method for building a virtual digital platform according to claim 1, characterized in that: After obtaining high-risk signals, refer to the long-term interest optimization recommendations and adjust the push strategy, including the following: When a high-risk signal is obtained, based on long-term interest data, collaborative filtering algorithms or content-based recommendation methods are used to prioritize content related to the user's long-term interests.

6. A data collection system for building a virtual digital platform, used to implement the data collection method for building a virtual digital platform as described in any one of claims 1 to 5, characterized in that: include: Data collection module, feature fusion module, deviation extraction module, cognitive analysis module and recommendation adjustment module; Data collection module: collects multimodal data of users and passes the collected data to the feature fusion module; Feature fusion module: Through the dynamic adaptive fusion strategy of the time-varying covariance matrix combined with the graph neural network, it monitors and analyzes multimodal data in real time, dynamically adjusts the modal weights, nonlinearly fuses features, generates a unified user behavior feature vector, and passes the obtained user behavior feature vector to the deviation extraction module; Deviation extraction module: Based on time series analysis and clustering algorithms, it dynamically compares current behavior with historical behavior baselines, extracts deviation data, and passes the deviation data to the cognitive analysis module; Cognitive analysis module: extracts user cognitive fatigue characteristics, uses deep learning models for real-time training and recognition, analyzes changes in user cognitive fatigue status in real time, generates corresponding feedback signals, including high-risk signals and low-risk signals, and passes high-risk signals to the recommendation adjustment module; Recommendation adjustment module: After obtaining high-risk signals, optimize recommendations based on long-term interests and adjust push strategies.

Images