Online behavior detection method based on multi-modal time sequence feature deep fusion and related device

Abstract

The invention discloses an online behavior detection method based on multi-modal time sequence feature deep fusion and a related device, and the method comprises the steps: collecting multi-modal perception data in an online activity process of a user, and carrying out the feature extraction of the preprocessed multi-modal perception data, and obtaining a single-modal time sequence feature and a multi-modal combined feature; performing cross attention fusion on the single-mode time sequence features and performing depth representation extraction on the multi-mode joint features by using a joint cross attention depth fusion model to obtain cross-mode depth representation of the single-mode time sequence features and depth representation of the multi-mode joint features; performing deep fusion on the cross-modal depth representation of the single-modal time sequence feature and the depth representation of the multi-modal joint feature by adopting a selective fusion strategy capable of learning weight to obtain a multi-modal depth fusion representation; and inputting the multi-modal deep fusion representation into a classifier to generate a prediction probability of each behavior category. According to the invention, on the premise of protecting the privacy of the user, the online behavior can be accurately detected.

Description

Technical Field

[0001] This invention belongs to the field of user behavior detection technology, specifically relating to an online behavior detection method and related apparatus based on deep fusion of multimodal temporal features. Background Technology

[0002] With the rapid development of information technology and network infrastructure, people are increasingly relying on computers and various smart terminals to perform complex information processing tasks in network environments. These activities are typically characterized by being online, time-consuming, multi-stage, and highly interactive. User behavior is mainly reflected in the continuous interaction with interface elements and the comprehensive use of multiple input devices.

[0003] In complex information task scenarios like these, relying solely on task configuration, foreground application type, or coarse-grained operation logs makes it difficult for the system to accurately determine the user's online behavioral state at any given moment. For example, it's hard to distinguish whether the user is effectively operating around the current task or has shifted to browsing or operating on topics less related to the task. Without an objective detection and identification mechanism for user online behavioral states, related systems and researchers often have to make rough judgments based on experience or general statistics, making it difficult to obtain fine-grained, quantifiable behavioral labels for analyzing the relationship between different behavioral patterns and task results. This also hinders the provision of objective behavior-based feedback or necessary external supervision for users. Therefore, it is necessary to automatically detect and model user behavioral states in complex and dynamic online task environments, distinguishing between different types of task-related and non-task behaviors. This provides a foundation for subsequent behavioral pattern analysis, effect evaluation, and applications such as supervision or self-management.

[0004] Currently, most existing online behavior detection methods rely primarily on video data, inferring users' behavioral states by analyzing facial expressions, posture changes, and environmental context. However, in practical applications, users are often unwilling to turn on or upload camera video data due to privacy concerns, making it difficult to widely deploy and use such video-based behavior detection methods in real-world scenarios.

[0005] To mitigate privacy concerns, some existing studies attempt to identify online behavior using non-video, multimodal perception data such as eye tracking, mouse and keyboard operations, touch swipes, or application / webpage switching records. While these data offer advantages in terms of privacy friendliness, current methods often employ shallow multimodal fusion strategies such as simple feature concatenation or weighted summation, failing to fully exploit complementary information and temporal dependencies between modalities. Consequently, the accuracy and robustness of online behavior detection remain unsatisfactory, making it difficult to meet the practical needs of complex and ever-changing online scenarios.

[0006] Therefore, without relying on video data, how to effectively and deeply fuse multimodal temporal sensing data, fully utilize the complementary information between single-modal behavior information and multimodal data, and improve the accuracy and generalization ability of online behavior detection is a technical problem that urgently needs to be solved. Summary of the Invention

[0007] The purpose of this invention is to address the problems in the prior art by providing an online behavior detection method and related apparatus that uses deep fusion of multimodal temporal features. This method constructs a joint cross-attention deep fusion model, performs cross-modal deep representation learning on multimodal perception data sequences, and selectively fuses the joint features of cross-modal representation and multimodal interaction behavior to achieve accurate detection of user online behavior while protecting user privacy.

[0008] To achieve the above objectives, the present invention provides the following technical solution: Firstly, an online behavior detection method based on deep fusion of multimodal temporal features is provided, including: Collect and preprocess multimodal perception data during users' online activities; Feature extraction is performed on the preprocessed modal sensing data to obtain single-modal temporal features and multimodal joint features; Using a pre-built joint cross-attention deep fusion model, cross-attention fusion is performed on single-modal temporal features and deep representation extraction is performed on multi-modal joint features, respectively, to obtain cross-modal deep representations of single-modal temporal features and deep representations of multi-modal joint features; Based on the potential differences in predicted contributions, a selective fusion strategy with learnable weights is adopted to deeply fuse the cross-modal deep representation of single-modal temporal features and the deep representation of multimodal joint features to obtain a multimodal deep fusion representation. The multimodal deep fusion representation is input into the classifier to generate the predicted probability of each behavior category.

[0009] As a preferred embodiment, the multimodal perception data during the user's online activity includes eye-tracking behavior, mouse movement, keystroke behavior, and touch swipe data, expressed as coordinates. Recorded with timestamps, represented as .

[0010] As a preferred approach, the preprocessing steps for multimodal sensing data include: Outlier filtering is performed on the sensing data for each modality. Interpolation is performed on each modal sensing data sequence after outlier filtering to fill in missing data points and maintain the temporal continuity and length consistency of each modal sensing data sequence. Align the modal sensing data sequences within the same time period based on timestamps to ensure that under the same label or the same time window, each modal sensing data sequence corresponds to the same time range; Resample each modal sensing data sequence to ensure that the sampling frequency of different modal sensing data sequences is consistent on the time axis.

[0011] As a preferred approach, the extraction method for the single-modal temporal features is as follows: For length of n A single-modal sensing data sequence, in the first n The following features are extracted from the sequence points:

[0012] In the formula, They represent the first n The sequence points are in x shaft and y Coordinates on the axis Indicates the first n The movement speed of each sequence point Indicates the first n The direction of movement of each sequence point.

[0013] As a preferred embodiment, the multimodal joint features are used to characterize the correlation between different modalities in terms of spatial location and dynamic behavior; For two types of modal sensing data, the multimodal joint features are extracted according to the following expression:

[0014] In the formula, These represent the data sequence points of the two modes at... x direction and y Coordinate difference in direction The Euclidean distance between points in the two modal sequences. It represents the angle between the motion directions of two modal data sequence points; when there are more than two modalities, it is expanded to the distance and direction relationship features between pairs of sensing data of each modality, comprehensively reflecting the multimodal interaction behavior pattern.

[0015] As a preferred approach, the steps of using a pre-built joint cross-attention deep fusion model to perform cross-attention fusion on single-modal temporal features to obtain a cross-modal deep representation of the single-modal temporal features include: For two modal sensing data, the temporal feature sequences of the two modalities are respectively input into an encoding network with a ResNet as the backbone to obtain their respective latent representation vectors. Let the outputs of the first and second modalities after passing through the first two residual modules of the ResNet be respectively... and , and This represents the latent feature matrix of the corresponding mode; Attention scores between the first and second modalities are calculated using the dot product between the query and the key to characterize the correlation between the two modalities at different time points; the query for the first modality is... The key of the second mode is Then the cross-attention weight matrix A of the first mode to the second mode is expressed as follows:

[0016] In the formula, and It is a weight matrix. The dimension of the key vector; After calculating the cross-attention weights for the first and second modalities respectively, the latent feature matrix is ​​weighted and aggregated according to the following formula to obtain the modal features after the attention region is enhanced:

[0017]

[0018] in, and These represent the attention weighting matrices for the first and second modes, respectively. Will and The inputs to the last two residual modules and the global average pooling GAP layer of ResNet are then non-linearly mapped to the ReLU activation function through a fully connected layer. Finally, the attention features of the two modalities are concatenated to obtain the cross-modal representation vector.

[0019] Using a learnable weight matrix Mapping the concatenated features to a new representation space yields a cross-modal deep representation: .

[0020] As a preferred approach, the steps of extracting deep representations of multimodal joint features using a pre-built joint cross-attention deep fusion model to obtain the deep representations of multimodal joint features include: taking the multimodal joint features as input, performing deep representation learning through a deep network structure to obtain the deep representations of the multimodal joint features. Deep representation of multimodal joint features This approach comprehensively characterizes the spatial and temporal patterns of coordinated changes among multimodal sequences, providing supplementary information for behavior classification.

[0021] As a preferred embodiment, the step of using a selective fusion strategy with learnable weights to deeply fuse the cross-modal deep representation of single-modal temporal features and the deep representation of multimodal joint features based on the potential differences in predicted contributions to obtain a multimodal deep fused representation includes: Cross-modal depth representation of unimodal temporal features and deep representation of multimodal joint features To splice:

[0022] By performing a nonlinear mapping between a fully connected layer and the ReLU activation function, a multimodal deep fusion representation is obtained:

[0023] In the formula, The fusion weight matrix is ​​learnable; The step of inputting the multimodal deep fusion representation into the classifier to generate the predicted probability of each behavior category includes: Multimodal deep fusion representation The input is fed into a multilayer perceptron (MLP) classifier to predict the activity label for each input behavior sequence. The MLP classifier consists of an input layer, a hidden layer, and an output layer. The number of units in the output layer corresponds to the number of online behavior categories to be identified. Each layer uses ReLU as the activation function, and the output layer uses the Softmax function to generate the predicted probability of each behavior category. The entire process is trained in an end-to-end manner, and the loss function is cross-entropy loss.

[0024] Secondly, an online behavior detection system based on deep fusion of multimodal temporal features is provided, including: The data acquisition and preprocessing module is used to collect and preprocess multimodal perception data during users' online activities. The feature extraction module is used to extract features from the preprocessed modal sensing data to obtain single-modal temporal features and multimodal joint features; The joint cross-attention deep fusion module is used to perform cross-attention fusion on single-modal temporal features and extract deep representations of multimodal joint features using a pre-built joint cross-attention deep fusion model, so as to obtain cross-modal deep representations of single-modal temporal features and deep representations of multimodal joint features. The learnable weight selective fusion module is used to perform deep fusion of the cross-modal deep representation of single-modal temporal features and the deep representation of multimodal joint features based on the potential differences in prediction contributions using a learnable weight selective fusion strategy, so as to obtain a multimodal deep fusion representation. The behavior category prediction module is used to input multimodal deep fusion representations into the classifier to generate predicted probabilities for each behavior category.

[0025] Thirdly, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being executed by a processor in an electronic device to implement the online behavior detection method of deep fusion of multimodal temporal features as described in the first aspect.

[0026] Compared with the prior art, the present invention has at least the following beneficial effects: This invention presents an online behavior detection method based on deep fusion of multimodal temporal features. By collecting multimodal perception data during users' online activities, it eliminates the reliance on camera video data, offering better privacy protection. It tracks user online behavior using non-intrusive, non-video perception methods such as eye movement, mouse movement, keyboard clicks, and touch swipes, making it suitable for complex information processing and online work scenarios with privacy and compliance requirements. Based on a cross-attention mechanism, this invention constructs a joint cross-attention deep fusion model. It establishes explicit correlation modeling between different modalities for multimodal temporal features, calculates attention weights between different modalities through cross-attention, and extracts cross-modal deep representations. Compared to existing shallow fusion methods such as simple feature concatenation, this method can more fully exploit the complementarity and consistency information between modalities, effectively improving the accuracy and robustness of behavior detection. This invention, based on the extraction of cross-modal deep representations of unimodal data, introduces deep representations of multimodal joint features. It selectively fuses the two representations using learnable weights, enabling the model to adaptively adjust the contribution ratio of unimodal and multimodal joint information in the final deep fused representation. This fully utilizes unimodal behavioral information and multimodal complementary information, thereby improving the model's ability to distinguish complex online behavior patterns. The online behavior detection method based on deep fusion of multimodal temporal features in this invention achieves stable online behavior detection performance under different detection time windows by windowing and overlapping sampling of multimodal temporal data. Experimental results show that, compared to single-modal methods, this invention achieves higher detection accuracy and F1 scores under different time windows, reaching a maximum F1 score of approximately 94.87%. Furthermore, it performs excellently in multiple online behavior recognition tasks, demonstrating good generalization ability and practical value. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a schematic diagram of the online behavior detection method based on deep fusion of multimodal temporal features according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the joint cross-attention deep fusion model structure according to an embodiment of the present invention; Figure 3 This is a comparative evaluation chart of the overall performance of embodiments of the present invention under different detection windows; Figure 4 This is a performance comparison and evaluation chart of the embodiments of the present invention under different modal combinations; Figure 5 This is a classification performance evaluation graph for different online behavior categories in an embodiment of the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, those skilled in the art can obtain other embodiments without creative effort.

[0030] Please see Figure 1 This invention proposes an online behavior detection method based on deep fusion of multimodal temporal features, which achieves high-precision identification of online behavior while protecting user privacy. The method mainly includes the following steps: S1. Collect and preprocess multimodal perception data during users' online activities; S2. Extract features from the preprocessed modal sensing data to obtain single-modal temporal features and multimodal joint features; S3. Using a pre-built joint cross-attention deep fusion model, cross-attention fusion is performed on single-modal temporal features and deep representation extraction is performed on multi-modal joint features to obtain cross-modal deep representations of single-modal temporal features and deep representations of multi-modal joint features; S4. Based on the potential differences in predicted contributions, a selective fusion strategy with learnable weights is adopted to deeply fuse the cross-modal deep representation of single-modal temporal features and the deep representation of multimodal joint features to obtain a multimodal deep fusion representation. S5. Input the multimodal deep fusion representation into the classifier to generate the predicted probability of each behavior category.

[0031] In one possible implementation, the present invention eliminates the reliance on video data and employs a non-intrusive, non-video-based perception method to track user online behavior. The multimodal perception data during the user's online activity in step S1 includes data such as eye movement, mouse movement, keystrokes, and touch swipes. This raw data can all be derived from coordinates... Recorded with timestamps, represented as .

[0032] Further steps for preprocessing multimodal sensing data include: (1) Filtering: Perform outlier filtering on each modal perception data to remove obviously unreasonable or invalid data points. For example, for eye-tracking data, fixation points that are outside the screen range or invalid fixation points caused by blinking can be removed; for mouse data, abnormal jump points can be removed.

[0033] (2) Interpolation completion: Interpolation is performed on the modal sensing data sequences after outlier filtering to fill in missing data points and maintain the temporal continuity and length consistency of the sequences. Interpolation strategies such as mean interpolation or spline interpolation can be adopted according to the characteristics of different modal sensing data.

[0034] (3) Time alignment: Based on the timestamp, the data sequences of each modality perception within the same time period are aligned to ensure that under the same label or the same time window, each modality data sequence corresponds to the same time range, so that the subsequent multimodal fusion is based on a unified time semantic.

[0035] (4) Resampling: Since the sampling frequency of different modal sensing data may differ, it is necessary to resample each modal sensing data sequence to make the sampling frequency of different modal sensing data on the time axis consistent, so as to facilitate the formation of a unified multimodal time sequence input.

[0036] In one possible implementation, the extraction method of the single-modal temporal features in step S2 is as follows: For length of n For single-modal sensing data sequences, basic temporal feature extraction is performed, and in the first... n The following features are extracted from the sequence points:

[0037] In the formula, They represent the first n The sequence points are in x shaft and y Coordinates on the axis Indicates the first n The movement speed of each sequence point Indicates the first n The direction of movement of each sequence point.

[0038] The multimodal joint features described in step S2 are used to characterize the correlation between different modalities in terms of spatial location and dynamic behavior; for the two modal perception data, the multimodal joint features are extracted according to the following expression:

[0039] In the formula, These represent the data sequence points of the two modes at... x direction and y Coordinate difference in direction The Euclidean distance between points in the two modal sequences. It represents the angle between the motion directions of the data sequence points of the two modalities; when there are more than two modalities, it is expanded to the distance and direction relationship features between the pairwise combinations of the sensing data of each modality, so as to more comprehensively reflect the multimodal interaction behavior pattern.

[0040] Please see Figure 2 In one possible implementation, this embodiment of the invention employs a joint cross-attention deep fusion model with a ResNet backbone. Taking two modal perception data as an example, step S3 utilizes the pre-built joint cross-attention deep fusion model to perform cross-attention fusion on the single-modal temporal features to obtain a cross-modal deep representation of the single-modal temporal features, including: The temporal feature sequences of the two modalities are respectively input into an encoding network with a ResNet residual network as its backbone to obtain their respective latent representation vectors. Let the outputs of the first and second modalities after passing through the first two residual modules of the ResNet be respectively... and , and This represents the latent feature matrix of the corresponding mode; Similar to traditional attention mechanisms, the attention score between the first and second modalities is calculated by the dot product between the query and the key, thus characterizing the correlation between the two modalities at different time points; the query of the first modality is... The key of the second mode is Then the cross-attention weight matrix A of the first mode to the second mode is expressed as follows:

[0041] In the formula, and It is a weight matrix. The dimension of the key vector; this type of fusion method analyzes the correlation between the first modality and the second modality at different time positions through similarity scores, thereby identifying shared features or consistent patterns that contribute significantly to the final prediction in the cross-modal deep representation.

[0042] After calculating the cross-attention weights for the first and second modalities respectively, the latent feature matrix is ​​weighted and aggregated according to the following formula to obtain the modal features after the attention region is enhanced:

[0043]

[0044] in, and These represent the attention weighting matrices for the first and second modes, respectively. A can be split or constructed symmetrically as needed.

[0045] Will and The inputs are fed into the last two residual modules and the global average pooling GAP layer of ResNet to further extract high-level cross-modal features. After a non-linear mapping through a fully connected layer and the ReLU activation function, the attention features of the two modalities are concatenated to obtain the cross-modal representation vector.

[0046] Using a learnable weight matrix Mapping the concatenated features to a new representation space yields a cross-modal deep representation: .

[0047] In one possible implementation, step S3 utilizes a pre-built joint cross-attention deep fusion model to extract a deep representation of the multimodal joint features, obtaining a deep representation of the multimodal joint features, including: By taking the multimodal joint features as input, deep representation learning is performed through ResNet or other deep network structures to obtain the deep representation of the multimodal joint features. Deep representation of multimodal joint features This approach comprehensively characterizes the spatial and temporal patterns of coordinated changes among multimodal sequences, providing supplementary information for behavior classification.

[0048] In one possible implementation, to fully account for the potential differences in the contributions of the two types of representations to the final prediction, step S4 includes: Cross-modal depth representation of unimodal temporal features and deep representation of multimodal joint features To splice:

[0049] By performing a nonlinear mapping between a fully connected layer and the ReLU activation function, a multimodal deep fusion representation is obtained:

[0050] In the formula, The fusion weight matrix is ​​learnable; Furthermore, step S5 represents the multimodal deep fusion. The input is fed into a multilayer perceptron (MLP) classifier to predict the activity label for each input behavior sequence. The MLP classifier includes an input layer with 512 units, a hidden layer with 256 units, and an output layer. The number of units in the output layer corresponds to the number of online behavior categories to be identified.

[0051] Each layer of the classifier uses ReLU as the activation function, and the output layer uses the Softmax function to generate the predicted probabilities of each behavior category. The entire system is trained in an end-to-end manner, and the loss function is cross-entropy loss.

[0052] The following is an experimental evaluation of the effectiveness of the online behavior detection method based on deep fusion of multimodal temporal features according to embodiments of the present invention.

[0053] In the experiments of this invention embodiment, a dataset containing multiple typical online behavioral states was constructed against the backdrop of general complex information processing and online work scenarios to experimentally evaluate the method proposed in this invention. Eight categories of online behavioral states were predefined in the experiments to distinguish between task-related and non-task-related behaviors, different types of task-related behaviors, and common distraction or wandering behaviors, such as: document writing and editing related to the current task, information retrieval and document review related to the task, browsing of content related to the task, and browsing of websites or content unrelated to the task. Twenty volunteers with daily computer and online application usage experience were selected as subjects. They completed various online activities on computing devices according to preset instructions, and multimodal interaction data was collected during the execution of different activities, including eye-tracking data and mouse data. In actual implementation, to verify the effectiveness of the method of this invention under fewer modal conditions, eye-tracking data and mouse movement data were mainly used for subsequent modeling and evaluation experiments.

[0054] The experiment employed a "leave-one-subject cross-validation" strategy for evaluation, meaning that data from 19 subjects were used for training each time, while data from the remaining subject was used for testing. The final result was the average of 20 experiments. To investigate the impact of different time windows on detection performance, this embodiment set four detection window sizes: 15 seconds, 30 seconds, 45 seconds, and 60 seconds, and used a 50% window overlap rate to divide continuous multimodal data sequences, thereby generating training and testing samples.

[0055] For each round of "one subject left out" partitioning, the subject data is divided into a training set and a test set. Then, the multimodal data sequences of each subject in each activity in the training set and test set are segmented according to the detection window and overlap rate mentioned above. The model is trained using the stochastic gradient descent (SGD) optimization algorithm in an end-to-end manner, with a weight decay coefficient set to 0.001 and a batch size of 64. It is trained for 50 iterations at an empirically set initial learning rate.

[0056] See Figure 3 , Figure 4 , Figure 5 The experimental evaluation results show that: (1) Under different detection windows (15 seconds, 30 seconds, 45 seconds, 60 seconds), simply fusing the two perception modalities can achieve significantly better detection results compared to the single modal method. The detection accuracy of multimodal fusion under each time window is better than that of using only single modal data.

[0057] (2) After adopting the joint cross-attention deep fusion model proposed in this invention, it is possible to utilize and fuse information from different modalities more effectively, thereby further improving the model accuracy, which can reach an F1 score of approximately 94.87%.

[0058] (3) In the performance evaluation of class-by-class recognition of eight types of online activities, the online behavior detection method of the present invention outperforms the comparison method in each type of activity, showing better stability and generalization ability.

[0059] The comprehensive experimental results show that the online behavior detection method based on deep fusion of multimodal temporal features of this invention can effectively integrate multiple perceptual data such as eye movement and mouse movements without relying on video data, significantly improving the detection accuracy of online behavior activities and showing good application prospects.

[0060] Another embodiment of the present invention proposes an online behavior detection system based on deep fusion of multimodal temporal features, comprising: The data acquisition and preprocessing module is used to collect and preprocess multimodal perception data during users' online activities. The feature extraction module is used to extract features from the preprocessed modal sensing data to obtain single-modal temporal features and multimodal joint features; The joint cross-attention deep fusion module is used to perform cross-attention fusion on single-modal temporal features and extract deep representations of multimodal joint features using a pre-built joint cross-attention deep fusion model, so as to obtain cross-modal deep representations of single-modal temporal features and deep representations of multimodal joint features. The learnable weight selective fusion module is used to perform deep fusion of the cross-modal deep representation of single-modal temporal features and the deep representation of multimodal joint features based on the potential differences in prediction contributions using a learnable weight selective fusion strategy, so as to obtain a multimodal deep fusion representation. The behavior category prediction module is used to input multimodal deep fusion representations into the classifier to generate predicted probabilities for each behavior category.

[0061] Another embodiment of the present invention provides an electronic device comprising: A memory for storing at least one instruction; and a processor for executing the instructions stored in the memory to implement the online behavior detection method of deep fusion of multimodal temporal features.

[0062] Another embodiment of the present invention provides a computer-readable storage medium storing at least one instruction, which is executed by a processor in an electronic device to implement the online behavior detection method of deep fusion of multimodal temporal features.

[0063] For example, the instructions stored in the memory can be divided into one or more modules / units. These modules / units are stored in a computer-readable storage medium and executed by the processor to complete the online behavior detection method of multimodal temporal feature deep fusion according to the present invention. The one or more modules / units can be a series of computer-readable instruction segments capable of performing specific functions, which describe the execution process of the computer program on the server.

[0064] The electronic device may be a smartphone, laptop, PDA, or cloud server, among other computing devices. It may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the electronic device may also include more or fewer components, or combinations of certain components, or different components; for example, it may also include input / output devices, network access devices, buses, etc.

[0065] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0066] The memory can be an internal storage unit of the server, such as a hard drive or RAM. It can also be an external storage device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or FlashCard. Furthermore, the memory can include both internal and external storage units. The memory is used to store computer-readable instructions and other programs and data required by the server. It can also be used to temporarily store data that has been output or will be output.

[0067] It should be noted that the information interaction and execution process between the above-mentioned module units are based on the same concept as the method embodiment. For details on their specific functions and technical effects, please refer to the method embodiment section. They will not be repeated here.

[0068] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0069] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0070] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0071] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. An online behavior detection method based on deep fusion of multimodal temporal features, characterized in that, include: Collect and preprocess multimodal perception data during users' online activities; Feature extraction is performed on the preprocessed modal sensing data to obtain single-modal temporal features and multimodal joint features; Using a pre-built joint cross-attention deep fusion model, cross-attention fusion is performed on single-modal temporal features and deep representation extraction is performed on multi-modal joint features, respectively, to obtain cross-modal deep representations of single-modal temporal features and deep representations of multi-modal joint features; Based on the potential differences in predicted contributions, a selective fusion strategy with learnable weights is adopted to deeply fuse the cross-modal deep representation of single-modal temporal features and the deep representation of multimodal joint features to obtain a multimodal deep fusion representation. The multimodal deep fusion representation is input into the classifier to generate the predicted probability of each behavior category.

2. The online behavior detection method based on deep fusion of multimodal temporal features according to claim 1, characterized in that, The multimodal perception data during the user's online activities includes eye movement, mouse movement, keystrokes, and touch swipe data, expressed as coordinates. Recorded with timestamps, represented as .

3. The online behavior detection method based on deep fusion of multimodal temporal features according to claim 1, characterized in that, The steps for preprocessing multimodal sensing data include: Outlier filtering is performed on the sensing data for each modality. Interpolation is performed on each modal sensing data sequence after outlier filtering to fill in missing data points and maintain the temporal continuity and length consistency of each modal sensing data sequence. Align the modal sensing data sequences within the same time period based on timestamps to ensure that under the same label or the same time window, each modal sensing data sequence corresponds to the same time range; Resample each modal sensing data sequence to ensure that the sampling frequency of different modal sensing data sequences is consistent on the time axis.

4. The online behavior detection method based on deep fusion of multimodal temporal features according to claim 1, characterized in that, The extraction method for the single-modal temporal features is as follows: For length of n A single-modal sensing data sequence, in the first n The following features are extracted from the sequence points: In the formula, They represent the first n The sequence points are in x shaft and y Coordinates on the axis Indicates the first n The movement speed of each sequence point Indicates the first n The direction of movement of each sequence point.

5. The online behavior detection method based on deep fusion of multimodal temporal features according to claim 1, characterized in that, The multimodal joint features are used to characterize the correlation between different modalities in terms of spatial location and dynamic behavior; For two types of modal sensing data, the multimodal joint features are extracted according to the following expression: In the formula, These represent the data sequence points of the two modes at... x direction and y Coordinate difference in direction The Euclidean distance between points in the two modal sequences. It represents the angle between the motion directions of two modal data sequence points; when there are more than two modalities, it is expanded to the distance and direction relationship features between pairs of sensing data of each modality, comprehensively reflecting the multimodal interaction behavior pattern.

6. The online behavior detection method based on deep fusion of multimodal temporal features according to claim 1, characterized in that, The steps for performing cross-attention fusion on single-modal temporal features using a pre-built joint cross-attention deep fusion model to obtain a cross-modal deep representation of the single-modal temporal features include: For two modal sensing data, the temporal feature sequences of the two modalities are respectively input into an encoding network with a ResNet as the backbone to obtain their respective latent representation vectors. Let the outputs of the first and second modalities after passing through the first two residual modules of the ResNet be respectively... and , and This represents the latent feature matrix of the corresponding mode; Attention scores between the first and second modalities are calculated using the dot product between the query and the key to characterize the correlation between the two modalities at different time points; the query for the first modality is... The key of the second mode is Then the cross-attention weight matrix A of the first mode to the second mode is expressed as follows: In the formula, and It is a weight matrix. The dimension of the key vector; After calculating the cross-attention weights for the first and second modalities respectively, the latent feature matrix is ​​weighted and aggregated according to the following formula to obtain the modal features after the attention region is enhanced: in, and These represent the attention weighting matrices for the first and second modes, respectively. Will and The inputs to the last two residual modules and the global average pooling GAP layer of ResNet are then non-linearly mapped to the ReLU activation function through a fully connected layer. Finally, the attention features of the two modalities are concatenated to obtain the cross-modal representation vector. Using a learnable weight matrix Mapping the concatenated features to a new representation space yields a cross-modal deep representation: 。 7. The online behavior detection method based on deep fusion of multimodal temporal features according to claim 1, characterized in that, The steps for extracting deep representations of multimodal joint features using a pre-built joint cross-attention deep fusion model include: taking the multimodal joint features as input, performing deep representation learning through a deep network structure to obtain the deep representations of the multimodal joint features. Deep representation of multimodal joint features This approach comprehensively characterizes the spatial and temporal patterns of coordinated changes among multimodal sequences, providing supplementary information for behavior classification.

8. The online behavior detection method based on deep fusion of multimodal temporal features according to claim 1, characterized in that, The step of using a selective fusion strategy with learnable weights to deeply fuse the cross-modal deep representation of single-modal temporal features and the deep representation of multimodal joint features based on the potential differences in predicted contributions to obtain a multimodal deep fused representation includes: Cross-modal depth representation of unimodal temporal features and deep representation of multimodal joint features To splice: By performing a nonlinear mapping between a fully connected layer and the ReLU activation function, a multimodal deep fusion representation is obtained: In the formula, The fusion weight matrix is ​​learnable; The step of inputting the multimodal deep fusion representation into the classifier to generate the predicted probability of each behavior category includes: Multimodal deep fusion representation The input is fed into a multilayer perceptron (MLP) classifier to predict the activity label for each input behavior sequence. The MLP classifier consists of an input layer, a hidden layer, and an output layer. The number of units in the output layer corresponds to the number of online behavior categories to be identified. Each layer uses ReLU as the activation function, and the output layer uses the Softmax function to generate the predicted probability of each behavior category. The entire process is trained in an end-to-end manner, and the loss function is cross-entropy loss.

9. An online behavior detection system based on deep fusion of multimodal temporal features, characterized in that, include: The data acquisition and preprocessing module is used to collect and preprocess multimodal perception data during users' online activities. The feature extraction module is used to extract features from the preprocessed modal sensing data to obtain single-modal temporal features and multimodal joint features; The joint cross-attention deep fusion module is used to perform cross-attention fusion on single-modal temporal features and extract deep representations of multimodal joint features using a pre-built joint cross-attention deep fusion model, so as to obtain cross-modal deep representations of single-modal temporal features and deep representations of multimodal joint features. The learnable weight selective fusion module is used to perform deep fusion of the cross-modal deep representation of single-modal temporal features and the deep representation of multimodal joint features based on the potential differences in prediction contributions using a learnable weight selective fusion strategy, so as to obtain a multimodal deep fusion representation. The behavior category prediction module is used to input multimodal deep fusion representations into the classifier to generate predicted probabilities for each behavior category.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, which is executed by a processor in an electronic device to implement the online behavior detection method of deep fusion of multimodal temporal features as described in any one of claims 1 to 8.

Images