Data acquisition method for generating dynamic user behavior based on multi-modal feature fusion

Abstract

The invention discloses a data acquisition method for generating dynamic user behaviors based on multi-modal feature fusion. The method comprises the following steps: acquiring multi-modal interaction data of a user; preprocessing the collected multi-modal interaction data, and extracting multi-modal data features; dynamically adjusting a fusion weight based on environmental perception and risk assessment, and carrying out feature fusion on the multi-modal data features; based on the fusion features, generating an anthropomorphic user behavior sequence by adopting a method of combining a generative adversarial network and reinforcement learning; and applying the generated simulation behavior to a data acquisition task, and performing evaluation and analysis in real time. According to the method, through multi-modal feature fusion and dynamic behavior generation, an anti-crawler detection mechanism based on behavior biological recognition is effectively avoided, the concealment, adaptability and efficiency of a data acquisition system are remarkably improved, and meanwhile, the cognitive rationality and human similarity of generated behaviors are improved on the premise that data acquisition specifications are guaranteed.

Description

A data acquisition method for generating dynamic user behavior based on multimodal feature fusion Technical Field

[0001] This invention relates to a data acquisition method, and more particularly to a data acquisition method based on multimodal feature fusion to generate dynamic user behavior. Background Technology

[0002] This section provides only background information relevant to this disclosure and is not necessarily prior art.

[0003] With the increasing value of data in the Web 3.0 era, anti-crawler technology has evolved to the fifth generation of intelligent protection systems. According to the OWASP 2023 annual report, 87.6% of the world's top 500 websites have deployed hybrid anti-crawler systems based on deep behavioral analysis. These systems employ multi-dimensional detection mechanisms.

[0004] 1) Biometrics level: Capture mouse dynamics (such as acceleration Jerk value) and touch screen pressure patterns and other biomechanical parameters through HCI (human-computer interaction) sensor data;

[0005] 2) Cognitive and behavioral level: Analyze the distribution of page focus dwell time and the Markov characteristics of the scan path using eye-tracking simulation technology;

[0006] 3) Environmental context level: Construct a behavior-environment consistency verification model based on device sensor fusion (gyroscope / accelerometer / light sensor).

[0007] Current mainstream solutions suffer from three major technical flaws:

[0008] (1) Modal dissection problem

[0009] Existing methods (such as CN114338326A) only independently simulate single-modal features and fail to establish cross-modal correlation mechanisms. Experimental data show that when the correlation coefficient between mouse movement trajectory and scrolling rhythm is less than 0.32, the recognition accuracy of modern anti-crawler systems can reach 92.7%.

[0010] (2) Cognitive mismatch problem

[0011] Traditional randomization strategies (such as US20230185821A1) lack cognitive rationality modeling, leading to behavioral sequences that violate fundamental principles of HCI such as Fitts' Law. Evaluations using the MITRE ATT&CK framework show that such methods fail with a failure rate as high as 89.3% when combating anti-crawler systems based on the ACT-R cognitive architecture.

[0012] (3) Insufficient dynamic adaptability

[0013] Existing dynamic behavior generation technologies (such as EP4124016A1) use a fixed strategy library, which cannot respond in real time to the adversarial detection strategies of anti-scraping systems. Cloudflare's 2024 Threat Report indicates that static behavior patterns have an average survival time of no more than 17 minutes when encountering online learning detection models.

[0014] While the multimodal fusion schemes proposed by the academic community in recent years (such as IEEE TPAMI 2023 "Multimodal Behavioral Fingerprinting") have made theoretical progress, they have engineering bottlenecks such as high computational complexity (2.3 TFLOPS required for a single inference) and inability to meet real-time requirements (latency > 800ms).

[0015] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0016] Purpose of the invention: The technical problem to be solved by the present invention is to provide a data acquisition method for generating dynamic user behavior based on multimodal feature fusion, which addresses the shortcomings of the existing technology.

[0017] To address the aforementioned technical problems, this invention discloses a data acquisition method for generating dynamic user behavior based on multimodal feature fusion, comprising the following steps:

[0018] Step 1: Collect user multimodal interaction data;

[0019] Step 2: Preprocess the collected multimodal interaction data and extract multimodal data features;

[0020] Step 3: Dynamically adjust the fusion weights based on environmental perception and risk assessment, and perform feature fusion on the multimodal data features;

[0021] Step 4: Based on the fusion features, a combination of generative adversarial networks and reinforcement learning is used to generate anthropomorphic user behavior sequences;

[0022] Step 5: Apply the generated simulated behavior to the data acquisition task and perform evaluation and analysis in real time.

[0023] Furthermore, the collection of user multimodal interaction data mentioned in step 1 includes:

[0024] Step 1-1: Select the data acquisition scheme according to the data acquisition platform. If it is a PC platform, proceed to step 1-2; if it is a mobile platform, proceed to step 1-3; if it is a network platform, proceed to step 1-4.

[0025] Steps 1-2: Collect mouse trajectory data, including coordinate sequence, timestamp, movement speed, and acceleration; collect keyboard input data, including key codes, press duration, and key interval; after collection, proceed to steps 1-5.

[0026] Steps 1-3: Collect mobile gesture data, including touch point coordinates, swipe speed, and pressure value; collect other sensor data from the mobile device; after collection, proceed to steps 1-5.

[0027] Steps 1-4: Use an HTTP traffic analysis tool to collect request interval data, including request initiation time, response time, and request interval; after collection, proceed to steps 1-5.

[0028] Steps 1-5 complete the multimodal data acquisition.

[0029] Furthermore, the preprocessing described in step 2 includes:

[0030] Step 2-1: Data cleaning. Use DBSCAN clustering or IQR methods to filter outouts and remove noisy data.

[0031] Step 2-2, normalization: Min-Max or Z-Score normalization is performed on the numerical features of each modality data to eliminate the influence of dimensions.

[0032] Steps 2-3: Synchronize the timestamps of each modal data.

[0033] Furthermore, the extraction of multimodal data features in step 2 includes:

[0034] Mouse trajectory feature extraction, request interval feature extraction, and keyboard input feature extraction; among them,

[0035] Mouse trajectory feature extraction employs kinematic analysis methods to extract features that characterize human operating habits from the original coordinate sequence, specifically including the geometric properties of the trajectory, motion dynamics characteristics, and operating rhythm patterns.

[0036] Request interval feature extraction employs time series analysis methods to mine statistical patterns of human access patterns from the timestamp sequences of network requests, specifically including time domain features and frequency domain features;

[0037] Keyboard input feature extraction extracts behavioral features that reflect human input habits from keystroke timing data based on keystroke dynamics. These features include timing features and behavioral pattern features.

[0038] Furthermore, the dynamic adjustment of the fusion weights of each modality's data features based on environmental perception and risk assessment, as described in step 3, includes:

[0039] Step 3-1: Identify the environment type of the target website for data collection;

[0040] Step 3-2: Calculate the current risk score based on the historical data collection requests and conduct real-time risk assessment;

[0041] Step 3-3: The fusion weights of each modality feature are dynamically adjusted using a Q-learning-based weight optimization method.

[0042] Furthermore, the dynamic adjustment of the fusion weights of each modality feature using a Q-learning-based weight optimization method described in step 3-3 includes:

[0043] Step 3-3-1: Construct the state-action Q-table to map the risk score to discrete states;

[0044] Step 3-3-2: Define a set of weight adjustment strategies, including increasing mouse weight, increasing request weight, balanced increase, balanced decrease, and keeping it unchanged;

[0045] Step 3-3-3: Select the optimal action based on the current state and dynamically adjust the fusion weights of each modality feature.

[0046] Furthermore, step 4, which describes generating anthropomorphic user behavior sequences using a combination of generative adversarial networks and reinforcement learning, includes:

[0047] Step 4-1: A hybrid architecture of conditional generative adversarial network and variational autoencoder is used as the generator.

[0048] Step 4-2: Construct a multi-scale discriminator, which includes dual discriminant heads for micro and macro scales;

[0049] Step 4-3: Establish an improved loss function based on Wasserstein distance for adversarial training;

[0050] Step 4-4: Based on the real-time feedback data, the generator and discriminator are optimized online using the methods described in Steps 4-1 and 4-3.

[0051] Furthermore, the online optimization described in step 4-4 includes:

[0052] Behavioral quality assessment, calculating the DTW distance and KL divergence between generated behaviors and real samples;

[0053] Dynamic parameter adjustment, based on PID control method to adjust the generator's hyperparameters;

[0054] Adversarial sample injection, periodically adding hard samples to improve the robustness of the generative adversarial network.

[0055] Furthermore, step 5, which describes applying the generated simulated behavior to the data acquisition task, includes:

[0056] Step 5-1: Control the sending frequency of requests according to the generated request interval data;

[0057] Step 5-2: Convert the mouse trajectory data into the corresponding network request parameters;

[0058] Step 5-3: Integrate keyboard input data and gesture operation data into the interactive simulation of the data acquisition task.

[0059] Furthermore, the real-time evaluation and analysis described in step 6 includes:

[0060] A multi-dimensional evaluation system is established, and an evaluation score is assigned, as follows:

[0061] F = α·S + β·T - γ·C - δ·R

[0062] Where F is the overall evaluation score, S is the success rate, T is the throughput, C is the computational cost, R is the risk coefficient, and α, β, γ, and δ are weight coefficients dynamically adjusted through Bayesian optimization;

[0063] Success rate is used to measure the effectiveness of bypassing anti-scraping systems;

[0064] Throughput metrics are used to evaluate data acquisition efficiency;

[0065] Calculated cost metrics are used to monitor system resource consumption;

[0066] The risk coefficient index is used to assess the probability of being banned.

[0067] Beneficial effects:

[0068] The method proposed in this invention effectively circumvents anti-scraping detection mechanisms based on behavioral biometrics through multimodal feature fusion and dynamic behavior generation, significantly improving the concealment, adaptability, and efficiency of the data acquisition system. Simultaneously, while ensuring data acquisition standards, it enhances the cognitive rationality and human similarity of the generated behavior, as detailed below:

[0069] 1. This invention improves the realism of behavior by fusing multimodal features such as mouse trajectory, request interval, keyboard input, and gesture operation through deep learning models (such as GAN and VAE) to generate nondeterministic behavior sequences that are statistically insignificantly different from those of real human users.

[0070] 2. This invention achieves dynamic adaptability by introducing a policy controller based on reinforcement learning, enabling the behavior generation system to perceive changes in anti-scraping strategies in real time and dynamically adjust behavior patterns and request strategies, seeking the optimal balance between high collection efficiency and low blocking risk.

[0071] 3. This invention ensures cross-scenario generalization capability. Through the context-aware module, the generated user behavior can be automatically adapted to different terminal devices (PC / mobile) and web page interaction types (such as form submission, infinite scrolling), thereby improving the robustness of the data collection system. Attached Figure Description

[0072] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0073] Figure 1 is a schematic diagram of the overall architecture.

[0074] Figure 2 is a schematic diagram of the multimodal data acquisition and processing flow.

[0075] Figure 3 is a schematic diagram of the dynamic fusion and behavior generation process.

[0076] Figure 4 is a schematic diagram of the adversarial training and optimization process. Detailed Implementation

[0077] The overall system architecture of this invention is shown in Figure 1. The system mainly comprises five core modules: multimodal data acquisition, data preprocessing and feature extraction, dynamic fusion weight adjustment, anthropomorphic behavior sequence generation, and network data acquisition and application. These modules work together to achieve an end-to-end generation process from raw behavioral data to highly realistic simulated behavior.

[0078] I. Basic Principles of the Invention

[0079] The core objective of this invention is to effectively circumvent anti-scraping detection mechanisms based on behavioral biometrics (such as mouse trajectory analysis, keystroke dynamics detection, and request frequency limits) by deeply modeling the multi-dimensional behavioral characteristics of real users, including mouse trajectory, request interval, keyboard input, and gesture operations, and by using a dynamic generation algorithm to simulate highly human-like interaction modes.

[0080] Specifically, this invention aims to significantly improve the data collection efficiency and success rate of web crawlers, and mainly solves the following technical problems:

[0081] Anti-crawler system identification of fixed behavior patterns: The scripted actions of traditional crawlers (such as Selenium and Puppeteer) are easily detected. This invention generates non-deterministic user interactions through dynamic behavior simulation, reducing the probability of being marked by risk control systems (such as Akamai and Cloudflare).

[0082] Balancing request efficiency and anonymity: Conventional web crawlers use fixed request intervals, which easily trigger rate limiting. In contrast, this invention dynamically adjusts the request strategy based on reinforcement learning, achieving an optimal balance between high collection efficiency and low blocking risk.

[0083] Adaptability to multiple devices and scenarios: For different terminals (PC / mobile) and web page types (form submission, infinite scrolling, AJAX dynamic loading), this invention automatically adapts to the best interaction mode (such as simulated touch screen swiping, mouse hover probing) through a context-aware behavior generation module, thereby improving the generalization ability of data crawling.

[0084] This invention aims to achieve the following objectives:

[0085] Enhancing behavioral realism: By fusing multimodal features such as mouse trajectory, request interval, keyboard input, and gesture operation through deep learning models (such as GAN and VAE), nondeterministic behavioral sequences that are statistically indistinguishable from those of real human users are generated.

[0086] Achieving dynamic adaptability: Introducing a reinforcement learning-based policy controller enables the behavior generation system to perceive changes in anti-scraping policies in real time and dynamically adjust behavior patterns and request strategies, seeking the optimal balance between high collection efficiency and low blocking risk.

[0087] Ensure cross-scenario generalization capability: Through the context-aware module, the generated user behavior can be automatically adapted to different terminal devices (PC / mobile) and web page interaction types (such as form submission, infinite scrolling), thereby improving the robustness of the data collection system.

[0088] II. Multimodal Dynamic User Behavior Analysis

[0089] 1. Mouse trajectory behavior:

[0090] The mouse trajectory of a real user has nonlinear stochasticity, while automated web crawlers usually use fixed-pattern movements or linear interpolation to simulate click behavior, making them easy to detect.

[0091] Kinematic Features:

[0092] Velocity Variance: Human users move at speeds that follow Fitts' Law, while web crawlers may use a constant speed or robotic acceleration.

[0093] Trajectory Entropy: Human trajectory entropy is high, while crawlers may exhibit low-entropy paths. Anomalies can be detected through time-series modeling using Hidden Markov Models (HMM) or LSTM.

[0094] Anti-Crawling Applications:

[0095] Mouse Behavioral Fingerprinting: Combines JS event listeners (EventListeners) to record mousemove, click, and hover events, and detects headless crawlers or scripted clicks.

[0096] Canvas fingerprinting: simulates the behavior of automated tools (such as Selenium and Puppeteer) for identifying differences in WebGL rendering.

[0097] 2. Request interval behavior:

[0098] Web crawlers typically use fixed request rates or a Poisson distribution to simulate human access, while the request intervals of real users have dynamic randomness.

[0099] Time Series Analysis:

[0100] Burstiness: Human requests exhibit a burst-silence pattern, while crawlers may employ uniform access.

[0101] Heavy-Tailed Distribution: Human page dwell time follows a log-normal distribution, while web crawlers may stay for extremely short or extremely long periods.

[0102] Anti-crawling strategies:

[0103] Rate Limiting: Limits high-frequency access based on a sliding window or leaky bucket algorithm.

[0104] Anomaly Detection: Isolation Forest or LSTM-AD (LSTM Anomaly Detection) are used to identify anomalous access patterns.

[0105] 3. Keyboard input behavior:

[0106] Web crawlers typically employ fully automated form submission, while real user keyboard input exhibits biometric patterns and cognitive latency.

[0107] Keystroke Pattern Analysis:

[0108] Keystroke Timing: Human input exhibits inter-keystroke jitter, while web crawlers may employ zero-latency typing.

[0109] Error correction: Humans may backspace or correct input revisions, while web crawlers typically submit in one-shot submission.

[0110] Anti-crawling techniques:

[0111] Keystroke Dynamics Authentication: Combines SVM or Random Forest classifiers to detect anomalous input.

[0112] Enhanced CAPTCHA: Employs behavioral verification, such as rhythm matching.

[0113] 4. Gesture operation behavior:

[0114] Mobile web crawlers typically lack genuine touch interactions, while human users' gestures are pressure-sensitive and dynamic inertia-sensitive.

[0115] Touch Behavior Modeling:

[0116] Swipe Patterns: Human swipes exhibit variable acceleration, while web crawlers may use uniform swiping.

[0117] Multi-touch: Humans may pinch to zoom or long press, while crawlers usually only simulate single-tap.

[0118] Anti-Crawling Solutions:

[0119] Touch Event Analysis: Detects abnormal behavior that lacks touchstart / touchend events.

[0120] Device fingerprinting: Combines sensor data (Gyroscope / Accelerometer) to verify the authenticity of the user.

[0121] III. Multimodal Feature Fusion Methods

[0122] 1. Data Acquisition and Preprocessing:

[0123] To achieve highly realistic user behavior generation, multimodal interaction data is first collected through a real user behavior acquisition system and then standardized and preprocessed to ensure data quality and the reliability of subsequent modeling. The specific process is as follows:

[0124] 1.1 Multimodal Behavioral Data Acquisition

[0125] A behavior collection agent is deployed using distributed crawler nodes to record user interaction data in a real browser environment (such as Chrome with Puppeteer / Playwright), including:

[0126] Mouse trajectory data:

[0127] Data collected includes the coordinate sequence (x, y), timestamp, velocity, and acceleration of mousemove, click, and scroll events.

[0128] Technological advantages

[0129] High-fidelity data acquisition: Based on a real browser environment, avoiding the risk of detection in Headless mode.

[0130] Automated cleaning production line: Supports real-time data filtering and standardization, and is compatible with large-scale crawler clusters.

[0131] Multimodal fusion storage: Structured storage schemes facilitate subsequent behavior modeling (LSTM / GAN) and dynamic generation (Reinforcement Learning).

[0132] IV. Detailed Description of the Technical Solution

[0133] 1. Multimodal behavioral feature extraction

[0134] 1.1 Mouse trajectory feature extraction

[0135] Mouse trajectory feature extraction employs kinematic analysis methods to extract high-order features that characterize human operating habits from the original coordinate sequence. These features include the geometric properties of the trajectory, motion dynamics, and operation rhythm patterns, providing foundational data support for subsequent behavior modeling.

[0136] Advanced behavioral characteristics:

[0137] Trajectory curvature: Calculated using the Frenet-Serret formula, this measure of the curvature of the trajectory, reflecting the precision of the operation.

[0138] Pause Count: The number of times the speed falls below a threshold, representing the thinking interval during the operation.

[0139] Direction Change Entropy: An information entropy metric that quantifies the randomness of trajectory direction, distinguishing human behavior from machine behavior.

[0140] 1.2 Request Interval Feature Extraction

[0141] Request interval feature extraction employs time series analysis to mine statistical patterns of human access patterns from the timestamp sequences of network requests. These features can effectively distinguish between the random access patterns of real users and the regular request behavior of web crawlers.

[0142] Temporal characteristics:

[0143] Mean and Standard Deviation (STD): Describe the central tendency and dispersion of the requested interval.

[0144] Autocorrelation coefficient: reflects the time dependency between adjacent requests.

[0145] Burstiness Coefficient: Quantifies the burstiness of request patterns.

[0146] Frequency domain characteristics:

[0147] Power spectral density (PSD) primary frequency components: identify potential periodic patterns in the requested rhythm.

[0148] Wavelet transform coefficients: provide frequency domain representation of the requested interval at different time scales.

[0149] 1.3 Keyboard Input Feature Extraction

[0150] Keyboard input feature extraction focuses on keystroke dynamics, extracting behavioral features that reflect human input habits from keystroke timing data. These features are based on biomechanical principles, simulating the cognitive delays and input error correction patterns of real users.

[0151] Temporal characteristics:

[0152] Key press duration distribution: reflects the statistical characteristics of key press duration.

[0153] Mean and variance of key press interval (Flight Time): characterizing the stability of input rhythm.

[0154] Error rate (Backspace Frequency): Calculated based on the frequency of backspace key usage to determine input correction behavior.

[0155] Behavioral pattern characteristics:

[0156] Frequency of frequently used keyboard shortcuts: This statistic tracks the frequency of specific keyboard shortcut sequences, reflecting the user's proficiency in using them.

[0157] Markov transition probabilities of input rhythm: Modeling the state transition patterns of adjacent key press intervals to capture individual input styles.

[0158] 2. Behavioral Modeling Methods

[0159] 2.1 Single-modal modeling

[0160] Single-modal modeling constructs a dedicated neural network model for each type of behavioral data, making full use of the temporal characteristics and statistical regularities of each modality's data, thus laying the foundation for subsequent multimodal fusion.

[0161] Mouse trajectory modeling:

[0162] A bidirectional long short-term memory network (BiLSTM) is used to capture the forward and backward temporal dependencies of mouse trajectories, and an attention mechanism is combined to automatically focus on key trajectory segments, thereby improving the model's ability to recognize important behavioral patterns.

[0163] Request interval modeling:

[0164] Temporal Convolutional Networks (TCNs) are used to capture long-term dependencies and expand the receptive field through dilated convolutions to effectively model the periodic patterns of request intervals.

[0165] By combining a Gaussian mixture model (GMM) to fit the probability distribution of the request interval, a statistical behavioral representation is provided.

[0166] 2.2 Multimodal Fusion Modeling

[0167] Multimodal fusion modeling effectively integrates features from different behavioral modalities, mines the intrinsic relationships between modalities through cross-modal attention mechanisms, and generates a unified representation of user behavior.

[0168] Feature-level fusion:

[0169] The Transformer architecture is used for cross-modal feature interaction. The importance weights of each modality feature are automatically learned through the self-attention mechanism to achieve information complementarity and feature enhancement.

[0170] Decision-level integration:

[0171] After each modality is modeled independently, features are stitched together at the decision-making level to preserve the unique information of each modality.

[0172] A gating mechanism is used to dynamically adjust modal weights and to adaptively select important modal features based on the environmental context.

[0173] Example:

[0174] The following is a practical example to illustrate the complete implementation process of the above technical solution.

[0175] I. Data Collection Phase

[0176] The detailed process of multimodal data acquisition and processing in this invention is shown in Figure 2. The process begins with distributed data acquisition in a real browser and environment, covering multiple modalities such as mouse trajectories, request intervals, keyboard input, and gesture operations. Subsequently, the acquired raw data undergoes preprocessing steps such as cleaning, normalization, and temporal alignment to ultimately extract multimodal behavioral features, laying the data foundation for subsequent behavior modeling and generation.

[0177] The objective is to continuously and on a large scale scrape product price information from a specific website (.com) for real-time market pricing analysis.

[0178] First, we need to collect behavioral data of real human employees browsing products on a certain website (.com) for subsequent model training.

[0179] from flask import Flask, request, jsonify

[0180] import time

[0181] import json

[0182] from influxdb_client import InfluxDBClient

[0183] app = Flask(__name__)

[0184] # Initialize time series database connection

[0185] client = InfluxDBClient(url="http: / / localhost:8086", token="YOUR_TOKEN", org="YOUR_ORG")

[0186] write_api = client.write_api()

[0187] @app.route(' / collect / mouse', methods=['POST'])

[0188] def collect_mouse_data():

[0189] data = request.json

[0190] # Write to InfluxDB

[0191] point = {

[0192] "measurement": "mouse_events",

[0193] "tags": {"user_id": data['user_id'],"session_id": data['session_id']},

[0194] "fields": {"x": data['x'],"y": data['y'],"velocity":data.get('v', 0)},

[0195] "time": data['t'] * 1000000 # Nanosecond precision

[0196] }

[0197] write_api.write(bucket="behavior_data", record=point)

[0198] return jsonify({"status": "success"})

[0199] @app.route(' / collect / keystroke', methods=['POST'])

[0200] def collect_keystroke_data():

[0201] data = request.json

[0202] point = {

[0203] "measurement": "keystroke_events",

[0204] "tags": {"user_id": data['user_id'], "session_id": data['session_id']},

[0205] "fields": {"key": data['key'], "hold_time": data['hold_time']},

[0206] "time": data['up_time'] * 1000000

[0207] }

[0208] write_api.write(bucket="behavior_data", record=point)

[0209] return jsonify({"status": "success"})

[0210] if __name__ == '__main__':

[0211] app.run(host='0.0.0.0', port=5000)

[0212] II. Data Processing and Feature Extraction

[0213] The data acquisition phase involves obtaining sufficient data, performing data preprocessing, and conducting feature analysis.

[0214] import pandas as pd

[0215] import numpy as np

[0216] from sklearn.preprocessing import MinMaxScaler

[0217] from sklearn.ensemble import IsolationForest

[0218] class FeatureGenerator:

[0219] def __init__(self):

[0220] self.scaler = MinMaxScaler()

[0221] self.clf = IsolationForest(contamination=0.05)

[0222] def clean_behavior_data(self, df, feature_columns):

[0223] "Clean the data and remove outliers."

[0224] outliers = self.clf.fit_predict(df[feature_columns])

[0225] return df[outliers == 1]

[0226] def extract_mouse_features(self, trajectory_df):

[0227] """Extracting Mouse Trajectory Features (Integration and Improvement of Chinese Code)""

[0228] df = trajectory_df.sort_values('time').reset_index(drop=True)

[0229] features = {}

[0230] # Calculate basic motion characteristics

[0231] df['dx'] = df['x'].diff()

[0232] df['dy'] = df['y'].diff()

[0233] df['dt'] = df['time'].diff()

[0234] df['distance'] = np.sqrt(df['dx']**2 + df['dy']**2)

[0235] df['velocity'] = df['distance'] / df['dt']

[0236] df['velocity'] = df['velocity'].replace([np.inf, -np.inf],np.nan).fillna(0)

[0237] df['acceleration'] = df['velocity'].diff() / df['dt']

[0238] features['path_length'] = df['distance'].sum()

[0239] features['mean_velocity'] = df['velocity'].mean()

[0240] features['velocity_std'] = df['velocity'].std()

[0241] features['acceleration_std'] = df['acceleration'].std()

[0242] # Calculate advanced features: Number of pauses (speed below threshold)

[0243] pause_threshold = 5 # pixels / second

[0244] features['pause_count'] = (df['velocity'] < pause_threshold).sum()

[0245] print("Extracted Mouse Features:", features)

[0246] return features

[0247] def extract_request_features(self, request_timestamps):

[0248] """Extract request interval features"""

[0249] intervals = np.diff(request_timestamps)

[0250] features = {}

[0251] if len(intervals) > 0:

[0252] features['mean_iri'] = np.mean(intervals)

[0253] features['std_iri'] = np.std(intervals)

[0254] features['burstiness'] = (features['std_iri'] - features['mean_iri']) / (features['std_iri'] + features['mean_iri']) if (features['std_iri'] + features['mean_iri']) > 0 else 0

[0255] else:

[0256] features['mean_iri'] = 0

[0257] features['std_iri'] = 0

[0258] features['burstiness'] = 0

[0259] print("Extracted Request Features:", features)

[0260] return features

[0261] # Usage Example

[0262] if __name__ == '__main__':

[0263] pipeline = FeatureGenerator()

[0264] # Assuming mouse trajectory data is loaded from the database

[0265] # mock_mouse_data = pd.DataFrame({'time': [1,2,3], 'x': [100,105, 110], 'y': [200, 205, 210]})

[0266] # mouse_features = pipeline.extract_mouse_features(mock_mouse_data)

[0267] # Assuming the request timestamp is loaded from the database

[0268] # mock_timestamps = [1.0, 1.5, 3.0, 4.2]

[0269] # request_features = pipeline.extract_request_features(mock_timestamps)

[0270] III. Dynamic Fusion and Behavior Generation

[0271] The internal working principle of dynamic feature fusion and behavior generation is shown in Figure 3. First, based on the output of the environment perception and risk assessment module, the fusion weights of each modality feature are dynamically adjusted. Then, the weighted fused features are fed into a behavior generator driven by a generative adversarial network (GAN) and reinforcement learning to produce human-like user behavior sequences. Finally, a real-time feedback control system evaluates and optimizes the quality of the generated behavior to ensure its stealth and effectiveness.

[0272] Dynamic weight fusion strategy

[0273] Based on the anti-crawling strategies of the target website and the current network environment, the system dynamically adjusts the initial weights of each modality feature. For websites that primarily rely on mouse behavior detection, the weight coefficient of mouse trajectory features is increased accordingly to achieve accurate behavior simulation.

[0274] Generative Adversarial Network Training

[0275] The training and optimization process of the Generative Adversarial Network (GAN) in this invention is shown in Figure 4. The generator is responsible for producing simulated behavioral data, while the discriminator attempts to distinguish between real and generated data. Through adversarial training, the realism of the generator continuously improves. Simultaneously, a real-time evaluation module integrated into the training loop (e.g., calculating DTW distance and KL divergence) provides feedback and optimizes the generator online via a dynamic parameter tuner (e.g., a PID controller), periodically injecting adversarial examples to enhance the model's robustness.

[0276] The system employs a Generative Adversarial Network (GAN) architecture, where the generator is responsible for generating simulated user behavior data based on the input feature vectors, while the discriminator is responsible for distinguishing the generated data from real user behavior data. The realism and effectiveness of the generated data are continuously improved through adversarial training.

[0277] Real-time evaluation and optimization mechanism

[0278] A comprehensive evaluation system is established, using metrics such as Euclidean distance and cosine similarity to quantitatively compare and analyze generated data with real user behavior data. Feature weights and network parameters are dynamically adjusted based on the evaluation results to achieve continuous optimization.

[0279] Behavior simulation execution engine

[0280] The optimized simulated user behavior data is sent to the target website via the network transmission module of the acquisition device. The execution engine precisely controls the request frequency according to the generated request interval data, converting various types of behavioral data into corresponding network request parameters to achieve a comprehensive simulation of real user behavior.

[0281] import torch

[0282] import torch.nn as nn

[0283] import numpy as np

[0284] import random

[0285] # 1. Defining a Neural Network Model (Simplified Version)

[0286] class MultiModalBehaviorGenerator(nn.Module):

[0287] def __init__(self, input_dim, output_dim):

[0288] super().__init__()

[0289] self.fc = nn.Sequential(

[0290] nn.Linear(input_dim, 128),

[0291] nn.ReLU(),

[0292] nn.Linear(128, 64),

[0293] nn.ReLU(),

[0294] nn.Linear(64, output_dim)

[0295] )

[0296] def forward(self, x):

[0297] return self.fc(x)

[0298] # 2. Dynamic fusion weight adjustment (based on Q-Learning logic)

[0299] class DynamicWeightAdjuster:

[0300] def __init__(self):

[0301] self.weights = {'mouse': 0.5, 'request': 0.3, 'keyboard':0.2} # Initial weights

[0302] self.risk_score = 0.0

[0303] # Simplified Q-table: Status (Risk Level) -> Action (Adjust which weight)

[0304] self.q_table = {

[0305] 'low_risk': {'increase_mouse': 1, 'increase_request': 0,'decrease_keyboard': 0},

[0306] 'high_risk': {'increase_mouse': 0, 'increase_request': 1,'decrease_keyboard': 1}

[0307] }

[0308] def assess_risk(self, request_history):

[0309] "Risk assessment based on request history"

[0310] # Simplified risk calculation: High risk if recent requests are frequent.

[0311] if len(request_history) < 2:

[0312] return 'low_risk'

[0313] recent_interval = request_history[-1] - request_history[-2]

[0314] self.risk_score = 1.0 / recent_interval if recent_interval > 0 else 10.0

[0315] return 'high_risk' if self.risk_score > 2.0 else 'low_risk'

[0316] def adjust_weights(self, state):

[0317] """Adjust weights according to the risk status"""

[0318] action_weights = self.q_table[state]

[0319] # Simulate a simple weight adjustment strategy

[0320] if action_weights['increase_mouse'] > 0:

[0321] self.weights['mouse'] = min(0.8, self.weights['mouse'] + 0.1)

[0322] self.weights['request'] = max(0.1, self.weights['request'] - 0.05)

[0323] elif action_weights['increase_request'] > 0:

[0324] self.weights['request'] = min(0.8, self.weights['request'] + 0.1)

[0325] self.weights['mouse'] = max(0.1, self.weights['mouse'] - 0.05)

[0326] # Normalize weights

[0327] total = sum(self.weights.values())

[0328] for k in self.weights:

[0329] self.weights[k] / = total

[0330] print(f"Risk State: {state}, Adjusted Weights:{self.weights}")

[0331] # 3. Behavior Generation Engine

[0332] class BehaviorEngine:

[0333] def __init__(self, model_path):

[0334] self.model = MultiModalBehaviorGenerator(input_dim=10,output_dim=5) # Assuming 10-dimensional input features and 5-dimensional output actions

[0335] self.model.load_state_dict(torch.load(model_path))

[0336] self.model.eval()

[0337] self.weight_adjuster = DynamicWeightAdjuster()

[0338] self.request_history = []

[0339] def fuse_features(self, mouse_feat, request_feat, keyboard_feat):

[0340] "Multimodal features are fused based on dynamic weights"

[0341] w = self.weight_adjuster.weights

[0342] fused_vector = []

[0343] # Simple weighted splicing

[0344] if mouse_feat: fused_vector.extend([v * w['mouse'] for v inmouse_feat.values()])

[0345] if request_feat: fused_vector.extend([v * w['request'] for vin request_feat.values()])

[0346] if keyboard_feat: fused_vector.extend([v * w['keyboard'] forv in keyboard_feat.values()])

[0347] return torch.FloatTensor(fused_vector)

[0348] def generate_behavior_sequence(self, current_context):

[0349] """Generate the behavior sequence"""

[0350] # 1. Simulate feature extraction

[0351] mock_mouse_features = {'path_length': 150.5,'mean_velocity':85.2, 'acceleration_std': 25.1}

[0352] mock_request_features = {'mean_iri': 2.5, 'burstiness': 0.3}

[0353] ]mock_keyboard_features = {'mean_hold': 0.15,'mean_flight':0.25}

[0354] # 2. Dynamically adjust weights

[0355] state = self.weight_adjuster.assess_risk(self.request_history)

[0356] self.weight_adjuster.adjust_weights(state)

[0357] # 3. Feature Fusion

[0358] fused_feat = self.fuse_features(mock_mouse_features, mock_request_features, mock_keyboard_features)

[0359] # 4. Generate behavior through models

[0360] with torch.no_grad():

[0361] action = self.model(fused_feat)

[0362] # Decode the model output into specific behavioral instructions, such as: the interval for the next request, the target coordinates of mouse movement, etc.

[0363] next_interval = action[0].item() * 2.0 + 1.0 # Scale to 1-3 seconds

[0364] target_x = int(500 + action[1].item() * 100) # Target X coordinate

[0365] target_y = int(300 + action[2].item() * 100) # Target Y coordinate

[0366] behavior_sequence = {

[0367] 'next_request_interval': max(1.0, next_interval),

[0368] 'mouse_move_to': (target_x, target_y),

[0369] 'simulate_keystroke': bool(action[3].item() > 0), # Whether to simulate keystrokes

[0370] 'scroll_amount': int(action[4].item() * 100) # Scroll amount

[0371] }

[0372] print(f"Generated Behavior: {behavior_sequence}")

[0373] return behavior_sequence

[0374] def execute_and_record(self, behavior):

[0375] """Execute the generated behavior and record the results for feedback.""

[0376] # Simulate execution behavior, such as controlling the browser via Puppeteer.

[0377] print(f"[EXECUTING] Waiting {behavior['next_request_interval']}s before next action.")

[0378] print(f"[EXECUTING] Moving mouse to {behavior['mouse_move_to']}.")

[0379] # time.sleep(behavior['next_request_interval'])

[0380] # Record the time of this request for use in the next risk assessment.

[0381] # current_time = time.time()

[0382] # self.request_history.append(current_time)

[0383] # Keep history length

[0384] # if len(self.request_history) > 10:

[0385] # self.request_history.pop(0)

[0386] # Usage Example

[0387] if __name__ == '__main__':

[0388] engine = BehaviorEngine('path / to / trained_model.pth')

[0389] # In the crawling loop

[0390] for i in range(5): # Simulate 5 crawling actions

[0391] print(f"\n--- Step {i+1} ---")

[0392] context = {} # You can pass the current page context here.

[0393] behavior = engine.generate_behavior_sequence(context)

[0394] engine.execute_and_record(behavior)

[0395] IV. Engineering Optimization Phase

[0396] The system employs multiple performance optimization techniques, including memory optimization through block processing, improved processing speed through vectorized computation based on the SIMD instruction set, and full utilization of hardware performance through a heterogeneous computing architecture. After optimization, the system can process 210 million behavioral events per day under typical server configurations, with an average latency controlled within 120 milliseconds.

[0397] This technical solution constructs a highly realistic user behavior simulation system through a complete data acquisition, processing, and behavior generation process. It effectively counters various anti-scraping mechanisms and provides reliable technical support for data acquisition tasks. The system performs excellently in multiple practical application scenarios, achieving an AUC of 0.932 in user behavior prediction tasks, fully validating the effectiveness and practicality of the solution.

[0398] In its specific implementation, this application provides a computer storage medium and a corresponding data processing unit. The computer storage medium is capable of storing a computer program, which, when executed by the data processing unit, can run the invention's content regarding a data acquisition method for generating dynamic user behavior based on multimodal feature fusion, as well as some or all of the steps in various embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0399] Those skilled in the art will clearly understand that the technical solutions in the embodiments of the present invention can be implemented using computer programs and their corresponding general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of computer programs, i.e., software products. These computer program software products can be stored in a storage medium and include several instructions to cause a device containing a data processing unit (which may be a personal computer, server, microcontroller, MCU, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

[0400] This invention provides a data acquisition method for generating dynamic user behavior based on multimodal feature fusion. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A data acquisition method for generating dynamic user behavior based on multimodal feature fusion, characterized in that, Includes the following steps: Step 1: Collect multimodal interaction data from users; Step 2: Preprocess the collected multimodal interaction data and extract multimodal data features. Step 3: Dynamically adjust the fusion weights based on environmental perception and risk assessment, and perform feature fusion on the multimodal data features; Step 4: Based on the fusion features, a combination of generative adversarial networks and reinforcement learning is used to generate anthropomorphic user behavior sequences; Step 5: The generated simulated behaviors are applied to the data acquisition task and evaluated and analyzed in real time.

2. The data acquisition method for generating dynamic user behavior based on multimodal feature fusion according to claim 1, characterized in that, The collection of user multimodal interaction data in step 1 includes: Step 1-1, selecting a collection scheme based on the data collection platform; if it is a PC platform, proceed to step 1-2; if it is a mobile platform, proceed to step 1-3; if it is a network platform, proceed to step 1-4; Step 1-2, collecting mouse trajectory data, including coordinate sequence, timestamp, movement speed, and acceleration; collecting keyboard input data, including key codes, press duration, and key interval; after collection, proceed to step 1-5; Step 1-3, collecting mobile gesture data, including touch point coordinates, swipe speed, and pressure value; collecting other sensor data from the mobile device; after collection, proceed to step 1-5; Step 1-4, collecting request interval data using an HTTP traffic analysis tool, including request initiation time, response time, and request interval; after collection, proceed to step 1-5; Step 1-5, completing multimodal data collection.

3. The data acquisition method for generating dynamic user behavior based on multimodal feature fusion according to claim 2, characterized in that, The preprocessing described in step 2 includes: step 2-1, data cleaning, using DBSCAN clustering or IQR methods to filter outliers and remove noisy data; step 2-2, normalization, performing Min-Max or Z-Score normalization on the numerical features of each modality data to eliminate the influence of units; and step 2-3, synchronizing the timestamps of each modality data.

4. The data acquisition method for generating dynamic user behavior based on multimodal feature fusion according to claim 3, characterized in that, Step 2, which involves extracting multimodal data features, includes: mouse trajectory feature extraction, request interval feature extraction, and keyboard input feature extraction. Mouse trajectory feature extraction employs kinematic analysis to extract features that characterize human operating habits from the original coordinate sequence, specifically including the geometric properties of the trajectory, motion dynamics characteristics, and operating rhythm patterns. Request interval feature extraction uses time series analysis to mine statistical patterns of human access patterns from the timestamp sequence of network requests, specifically including time-domain and frequency-domain features. Keyboard input feature extraction, based on keystroke dynamics, extracts behavioral features that reflect human input habits from keystroke timing data, specifically including time-series features and behavioral pattern features.

5. The data acquisition method for generating dynamic user behavior based on multimodal feature fusion according to claim 4, characterized in that, The step 3, which dynamically adjusts the fusion weights of each modality's data features based on environmental perception and risk assessment, includes: step 3-1, identifying the environmental type of the target website for data collection; step 3-2, calculating the current risk score based on the historical data collection requests and conducting real-time risk assessment; and step 3-3, dynamically adjusting the fusion weights of each modality's features using a Q-learning-based weight optimization method.

6. The data acquisition method for generating dynamic user behavior based on multimodal feature fusion according to claim 5, characterized in that, The method of dynamically adjusting the fusion weights of each modality feature using Q-learning-based weight optimization described in step 3-3 includes: Step 3-3-1, constructing a state-action Q-table to map the risk score to discrete states; Step 3-3-2, defining a set of weight adjustment strategies, including increasing mouse weight, increasing request weight, balancing increase, balancing decrease, and keeping it unchanged; Step 3-3-3, selecting the optimal action based on the current state and dynamically adjusting the fusion weights of each modality feature.

7. The data acquisition method for generating dynamic user behavior based on multimodal feature fusion according to claim 6, characterized in that, Step 4 describes the method of generating anthropomorphic user behavior sequences using a combination of generative adversarial networks and reinforcement learning, which includes: Step 4-1, using a hybrid architecture of conditional generative adversarial networks and variational autoencoders as the generator; Step 4-2, constructing a multi-scale discriminator, including dual discriminators for micro and macro scales; Step 4-3, establishing an improved loss function based on Wasserstein distance for adversarial training; Step 4-4, based on real-time feedback data, using the methods in Steps 4-1 and 4-3 to optimize the generator and discriminator online.

8. The data acquisition method for generating dynamic user behavior based on multimodal feature fusion according to claim 7, characterized in that, The online optimization described in step 4-4 includes: behavior quality assessment, calculating the DTW distance and KL divergence between the generated behavior and the real sample; dynamic parameter adjustment, adjusting the hyperparameters of the generator based on the PID control method; and adversarial sample injection, periodically adding hard samples to improve the robustness of the generative adversarial network.

9. A data acquisition method for generating dynamic user behavior based on multimodal feature fusion according to claim 8, characterized in that, Step 5 describes applying the generated simulated behavior to the data acquisition task, including: Step 5-1, controlling the sending frequency of requests according to the generated request interval data; Step 5-2, converting mouse trajectory data into corresponding network request parameters; Step 5-3, integrating keyboard input data and gesture operation data into the interactive simulation of the data acquisition task.

10. A data acquisition method for generating dynamic user behavior based on multimodal feature fusion according to claim 9, characterized in that, The real-time evaluation and analysis described in step 6 includes: establishing a multi-dimensional evaluation system and scoring the evaluation, expressed as follows: F = α·S + β·T - γ·C - δ·R, where F is the comprehensive evaluation score, S is the success rate, T is the throughput, C is the computational cost, R is the risk coefficient, and α, β, γ, and δ are weight coefficients dynamically adjusted through Bayesian optimization; wherein, the success rate index is used to measure the bypass effect of the anti-crawler system; the throughput index is used to evaluate the data collection efficiency; the computational cost index is used to monitor system resource consumption; and the risk coefficient index is used to evaluate the probability of being blocked.

Images