Mobile bank user behavior recognition method and system based on multi-modal data

Abstract

The application relates to the technical field of mobile bank security protection, in particular to a mobile bank user behavior recognition method and system based on multi-modal data; the method comprises the following steps: three types of data, namely interface interaction events, device running states and user operation rhythms, are collected to construct multi-modal behavior feature units; the interaction events are subjected to semantic mapping to generate business semantic sequences, the core operation paths are extracted through compression processing, and path features are calculated; semantic stage rhythm sequences are constructed in combination with operation time to generate rhythm fingerprints containing overall stability and stage differences; the path features and the rhythm fingerprints are fused into multi-modal feature vectors, which are input into a recognition model after standardization and weighting, and a dynamic updating mechanism is introduced to continuously optimize the model, so that the mobile bank transaction security and the abnormal behavior recognition capability are effectively improved without involving user identity privacy; and the application effectively improves the accuracy and security of mobile bank behavior recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of mobile banking security protection, and specifically relates to a mobile banking user behavior identification method and system based on multimodal data. Background Art

[0002] With the rapid development of mobile communication technology and the widespread popularization of intelligent terminal devices, mobile banking services have become a core component of the financial service system, providing users with convenient and efficient financial services such as account management, payment and transfer, and wealth management and investment; while mobile banking improves user experience and business efficiency, its open network environment and business attributes also make it a key area for security protection; to effectively ensure user fund security and stable system operation, security identification technology based on user behavior analysis has gradually become an important direction of the mobile banking risk control system; by analyzing user operation habits, device usage characteristics and interaction modes, the abnormal operation identification capability can be improved without interfering with the normal user experience.

[0003] The Chinese invention patent application with publication number CN121300634A discloses a mobile banking multimodal interaction method and system based on a language user interface, and the method includes: acquiring input information of multiple modalities input by a user, fusing the input information of the multiple modalities to obtain a unified natural language instruction; inputting the unified natural language instruction into a large language model for intent classification and parameter extraction to obtain the operation intent corresponding to the unified natural language instruction and the corresponding business parameters; executing the operation intent according to the unified natural language instruction and the business parameters.

[0004] With the continuous advancement of data analysis technology, integrating multi-source information to comprehensively characterize user operation features has become an important trend in industry development; meanwhile, against the background of increasingly improved data security and privacy protection regulations, how to achieve accurate and reliable behavior identification on the premise of fully protecting users' personal information has become a key topic in mobile banking security technology research. Summary of the Invention

[0005] The purpose of the present invention is to address the problems existing in the background art, and to propose a mobile banking user behavior identification method and system based on multimodal data.

[0006] The technical solution of the present invention: A mobile banking user behavior identification method based on multimodal data includes the following specific implementation steps: S1. Construct a multimodal behavior data collection structure that does not involve user identity information and personal privacy in the mobile banking client, synchronously collect and uniformly structurally express three types of data: interface interaction events, device operating status, and user operation rhythm, so as to construct a multimodal behavior feature unit. S2. Semantic analysis of mobile banking users' interface interaction behavior is performed. By establishing a business semantic mapping space, the original interaction events are converted into semantic units with business meanings and user interaction semantic sequences are constructed in chronological order. Semantic compression is performed to extract the actual business operation path structure. Then, the user operation process is analyzed by statistically analyzing the transfer relationship between semantic units. Multimodal behavior path feature vectors are constructed by combining device status features and operation rhythm features. S3. By constructing a semantic stage rhythm time series, analyzing the overall rhythm stability, and establishing a semantic stage rhythm structure spectrum, a multi-dimensional behavioral rhythm fingerprint is generated to characterize the user operation rhythm features from the time dimension, combining business semantics with operation time. S4. The obtained multimodal behavior path features and behavior rhythm fingerprints are fused together. A unified multimodal behavior feature representation is constructed through feature vector connection. After standardization and weight allocation, the representation is input into the multimodal behavior recognition model to identify user operation behavior. At the same time, the model's adaptability is continuously optimized through a dynamic feature update mechanism to complete the recognition of mobile banking user behavior.

[0007] Preferably, step S1 specifically includes: By establishing an interface interaction listening mechanism within the mobile banking client, the user's page access, control triggering, and operation confirmation behaviors in the application are recorded in real time. Each operation is abstracted into an interactive event unit containing page identifier, operation type, control structure position, and timestamp, and a sequence of interface interaction trajectories is constructed in chronological order. While collecting interface interaction events, the system performance interface synchronously records the mobile terminal's operating status information, including processor utilization, memory usage ratio, network response latency, and interface rendering frame rate. The device status fluctuation index is constructed by calculating the degree of device status change between adjacent events to characterize the device's operating stability during operation. The time interval between adjacent operations is calculated by using the timestamps of the interface interaction events, and a user operation rhythm time series is constructed. The average rhythm and the degree of rhythm fluctuation are further calculated to form a rhythm stability index that can reflect the rhythm characteristics of user operations. After obtaining the interface interaction trajectory, device operating status, and operation rhythm data, the three types of behavioral information are integrated according to a unified time series to construct a multimodal behavioral feature unit and form a complete multimodal behavioral data set, so that operation path information, device environment characteristics, and operation rhythm characteristics can be expressed in the same data structure.

[0008] Preferably, step S2, which involves semantic parsing of the user interface interaction behavior of mobile banking users, includes: Establish a business semantic space based on the internal functional module structure of the mobile banking application, and construct a mapping relationship between the interface structure and business semantics; The collected page identifiers, control positions, and operation types are combined and parsed. The original interface interaction events are converted into corresponding business semantic units through a semantic mapping function, so that each operation is given a clear business meaning. This transforms the interface-level interaction behavior into an operation expression structure with business semantics.

[0009] Preferably, step S2, which involves constructing a user interaction semantic sequence and performing semantic compression processing, includes: After obtaining the semantic units corresponding to each interactive event, a user interaction semantic sequence is constructed in chronological order to describe the user's complete operation process. To address the numerous repetitive clicks, swipes, and refreshes encountered in actual operations, a semantic compression mechanism is introduced. This mechanism merges consecutively repeated semantic units using a semantic compression function, extracting the core operation path that reflects the actual business process structure.

[0010] Preferably, in step S2, the step of constructing the multimodal behavior path feature vector includes: Based on the compressed semantic sequence, statistical analysis is performed on the transition relationship between adjacent semantic units, and a semantic transition structure is established by calculating the transition frequency and transition probability between semantic units. Calculate path stability metrics to reflect whether user operation paths conform to normal business process patterns and identify abnormal path structures or unreasonable business jump behaviors. Based on the semantic path structure, path length, semantic transfer complexity, and path stability index are extracted as path features. These features are then fused with the equipment state fluctuation index and operating rhythm stability to construct a unified multimodal behavior path feature vector.

[0011] Preferably, in step S3, the step of constructing the semantic stage rhythm time series includes: The obtained compressed semantic sequence is combined with the operation time interval to form a semantic stage rhythm sequence. Each rhythm unit contains both business semantic information and operation interval, ensuring that the operation rhythm of different business stages is captured in a structured way. By combining business semantics to construct semantic phase rhythm units, and binding business operations with time intervals through semantic rhythm mapping functions, the system can distinguish the different meanings of the same time interval in different business operations, forming a complete semantic phase rhythm sequence.

[0012] Preferably, step S3, which involves analyzing overall rhythm stability and establishing a semantic stage rhythm structure spectrum, includes: Based on the semantic stage rhythm sequence, the stability of the user's overall operation rhythm is analyzed, the average operation interval and the standard deviation of fluctuation are calculated, and a rhythm stability index is generated to characterize the consistency and naturalness of the user's operation. For each semantic stage, the average operating rhythm and rhythm dispersion are calculated to construct a semantic stage rhythm structure spectrum, which reflects the rhythm characteristics of users in different business links and captures the rhythm differences between stages, enabling the system to identify abnormal jumps or unnatural operating patterns.

[0013] Preferably, in step S3, the step of generating a multidimensional behavioral rhythm fingerprint includes: Define a semantic rhythm consistency index based on the rhythm structure spectrum; By combining the overall rhythm stability with the semantic stage rhythm structure spectrum, a multidimensional behavioral rhythm feature vector containing stability index, semantic consistency, average interval and volatility is generated, forming a quantifiable and interpretable rhythm fingerprint.

[0014] Preferably, step S4 specifically includes: By fusing behavioral path features with behavioral rhythm fingerprints, a unified multimodal behavioral feature representation is constructed through feature vector connection. Semantic mapping is performed on different operation stages in conjunction with the functional structure of mobile banking business, so that the operation path structure, business semantic information and operation rhythm features form a correspondence, and a three-dimensional behavioral feature structure with business meaning is constructed. The fused multimodal behavioral feature vectors are standardized to eliminate the dimensional differences between different features and bring all feature dimensions into a uniform numerical range. Based on the operational logic of mobile banking and the results of statistical analysis, weights are assigned to each feature to highlight the importance of key behavioral paths and rhythmic features. Abnormal extreme values ​​are locally normalized to obtain standardized and weighted multimodal feature vectors. After feature standardization and weighting, the weighted multimodal feature vectors are used as input to construct a behavior recognition model. The complex relationship between path structure features and operation rhythm features is analyzed by nonlinear mapping method. The fused features are mapped to specific user behavior categories to identify normal user operations, abnormal operations and potential automated operations. At the same time, the interpretability of the recognition results is enhanced by combining feature importance analysis mechanism. After the behavior recognition model outputs results, the recognition results are bound and stored with the corresponding multimodal behavior features for subsequent statistical analysis and model optimization. At the same time, a dynamic feature update mechanism is introduced to weightedly fuse and update historical behavior features with the latest behavior data, enabling the model to continuously learn the trend of user behavior changes and trigger security prompts or risk control strategies when abnormal behavior is detected.

[0015] The technical solution of this invention: A mobile banking user behavior recognition system based on multimodal data, comprising: Memory; processor; A computer program stored in the memory and capable of running on the processor; The processor executes a computer program to implement the aforementioned method for recognizing mobile banking user behavior based on multimodal data.

[0016] Compared with the prior art, the above-mentioned technical solution of the present invention has the following beneficial technical effects: This invention designs a method and system for mobile banking user behavior recognition based on multimodal data. By constructing a multimodal data acquisition structure that does not involve user identity information, it integrates three types of data: interface interaction, device operating status, and operation rhythm. This allows for a comprehensive characterization of user operation behavior features without infringing on sensitive privacy. Based on big data analysis and mining techniques, it performs semantic mapping and compression processing on the original interaction events, extracting core business paths and generating path features. This enables behavior analysis to penetrate into the business logic level, effectively identifying abnormal path structures or unreasonable jump behaviors, significantly improving the accuracy of abnormal operation recognition. Furthermore, by constructing semantic stage rhythm sequences and multidimensional behavioral rhythm fingerprints, it integrates business semantics... Deeply integrated with operational rhythm, this invention quantifies user operating habits from a time perspective, accurately distinguishing between natural fluctuations in real user operations and the mechanical behavior of automated scripts. This enhances the system's defense capabilities against risks such as simulated attacks and machine trading, while also providing interpretability to the identification results. By fusing path features and rhythmic fingerprints into a multimodal feature vector, and then inputting it into the identification model after standardization and weighting, high-precision classification of user behavior is achieved. The introduction of a dynamic feature update mechanism enables the model to continuously adapt to the natural evolution of operating habits, ensuring the timeliness and robustness of risk control strategies. Relying on big data analysis and mining technology, this invention effectively improves the transaction security and abnormal behavior identification capabilities of mobile banking. Attached Figure Description

[0017] Figure 1 This is a flowchart of a mobile banking user behavior recognition method based on multimodal data proposed in this invention. Detailed Implementation

[0018] Example 1, as Figure 1 As shown, the present invention proposes a mobile banking user behavior recognition method based on multimodal data, which includes the following specific implementation steps: S1. Construct a multimodal behavioral data collection structure in the mobile banking client that does not involve user identity information or personal privacy. By synchronously collecting and uniformly structuring interface interaction events, device operating status, and user operation rhythms, behavioral information from different sources can be correlated on the same time series, thereby obtaining a complete multimodal behavioral data set. This provides a stable, reliable, and computable data foundation for subsequent user behavior semantic analysis and behavior recognition. The specific implementation process is as follows: S11. By establishing an interface interaction monitoring mechanism within the mobile banking client, user actions such as page access, control triggering, and operation confirmation are recorded in real time. Each operation is abstracted into a unified structured interaction event unit. Information such as page identifier, operation type, control structure position, and timestamp is introduced to form a sequence of interface interaction trajectories, thereby structurally depicting the user's complete operation path. Specifically: An interface interaction monitoring mechanism is established within the mobile banking client. This mechanism captures user actions within the application interface in real time via system event callback interfaces, abstracting each action into a unified interaction event unit, and representing interface interactions as structured interaction events. ; Based on this, a sequence of user interface interaction trajectories is constructed in chronological order: ; in, This represents the i-th UI interaction event unit; This parameter indicates the identifier of the function page that the user is currently accessing. It comes from the page module numbering system within the mobile banking application. This indicates the type of action the user triggers on this page, such as clicking the confirmation button, swiping the page, or entering information. This parameter comes from the application control event log. This parameter represents the structural position index of the triggered control on the current page. It is automatically generated by the interface control tree structure and is used to distinguish the triggering behavior of different controls. The system timestamp indicating the occurrence of this interaction event is automatically recorded by the client's system clock; This represents the sequence of user interface interactions during a single business operation; n represents the total number of interaction events during this operation. S12. While collecting interface interaction events, the system performance interface synchronously records the mobile terminal's operating status information, including processor utilization, memory usage ratio, network response latency, and interface rendering performance. A device state fluctuation index is constructed by calculating the degree of device state change between adjacent events to characterize the device's operational stability during operation. Specifically: To identify abnormal device environments or automated script behaviors, it is necessary to simultaneously record the changes in the mobile terminal device's operating state during operation while collecting interface interaction data. This is done by calling the system performance interface provided by the mobile operating system to record the device's operating state vector when each interaction event is triggered. ; To characterize the changing patterns of equipment operating status during continuous operation, the data is normalized, and an equipment status fluctuation index is introduced: ; in, This represents the device operating state vector corresponding to the i-th interaction event; This indicates CPU utilization, obtained from the mobile operating system's performance monitoring interface. This indicates the current memory usage percentage, provided by the system resource management module. This parameter indicates the network request response latency and is obtained by sending a lightweight request from the client to the server. This indicates the current frame rate change in the interface rendering, reflecting the device's graphics processing load. The equipment condition fluctuation index is used to reflect the degree of performance change of the equipment during operation. , , and These represent the changes in CPU, memory, network latency, and UI rendering frame rate between two consecutive interactive events; It should be noted that the device status fluctuation index can characterize the natural fluctuation characteristics of device performance during real user operations. For example, page loading, data requests and other behaviors during user operations will cause slight performance changes, while automated scripts or simulated environments often show abnormally stable or abnormally sudden device status. S13. Calculate the time interval between adjacent operations using the timestamps recorded by the interface interaction events, construct a user operation rhythm time series, and further calculate the rhythm average and the degree of rhythm fluctuation to form a rhythm stability index that reflects the rhythmic characteristics of user operations. This index can characterize the natural time fluctuations existing in real user operations, specifically: After obtaining the timestamps of the interaction events, the user's operation rhythm is characterized by calculating the time difference between adjacent interaction events: ; The operational rhythm sequence can be obtained by sorting all time intervals: ; Further calculation of rhythm statistical characteristics, including but not limited to: Mean rhythm duration: ; Rhythm fluctuation degree: ; And define the operational rhythm stability: ; in, This represents the time interval between the i-th operation and the (i+1)-th operation; Indicates the average time of the operating rhythm; Indicates the standard deviation of the operating rhythm; Indicates the stability of the operating rhythm; It should be noted that the stability of the operating rhythm A value close to 1 indicates a highly stable rhythm, while real user operations usually have a certain degree of randomness, so the rhythm stability is usually in the medium range. S14. After obtaining the interface interaction trajectory, device operating status, and operation rhythm data, the three types of behavioral information are integrated according to a unified time series to construct a multimodal behavioral feature unit, and further form a complete multimodal behavioral data set, so that operation path information, device environment characteristics, and operation rhythm characteristics can be expressed in association within the same data structure, specifically: After collecting data on interface interaction trajectories, device operating status, and operational rhythm, it is necessary to unify and structure these three types of data to enable them to be correlated on the same timeline. To this end, a multimodal behavioral feature unit is constructed. ; This ultimately forms a complete set of multimodal behavioral data: ; in, Let represent the i-th multimodal behavioral feature unit; M represents the complete set of multimodal behavioral data formed during a user's mobile banking transaction.

[0019] S2. Based on the multimodal behavioral feature units obtained in step S1, semantic analysis is performed on the interface interaction behavior of mobile banking users. By establishing a business semantic mapping space, the original interaction events are converted into semantic units with business meaning, and a user interaction semantic sequence is constructed according to the time sequence. The actual business operation path structure is extracted through semantic compression processing, and the user operation flow pattern is analyzed by statistically analyzing the transition relationship between semantic units. Combined with the device status features and operation rhythm features in step S1, a multimodal behavioral path feature vector is constructed, thus forming a behavioral feature expression that includes both business logic and operation rhythm and device status information. The specific implementation process is as follows: S21. Establish a business semantic space based on the internal functional module structure of the mobile banking application, and construct a mapping relationship between the interface structure and business semantics. Combine and parse the page identifiers, control positions, and operation types collected in step S1. Convert the original interface interaction events into corresponding business semantic units through a semantic mapping function, so that each user operation can be given a clear business meaning. This transforms the interface-level interaction behavior into an operation expression structure with business semantics, specifically: Obtain the interaction events in step S1 Establish a semantic mapping relationship between the functional structure and interface elements of mobile banking, and construct a business semantic space based on the internal functional modules of the mobile banking application: ; Based on this, construct a semantic mapping function: ; For example, clicking the "Next" button on the transfer page will be mapped to "transfer information submission semantics", while clicking the balance area on the account page will be mapped to "account information viewing semantics". Where C represents the semantic space of mobile banking services; This represents the j-th business semantic category, such as account query semantic, transaction record browsing semantic, transfer input semantic, transfer confirmation semantic, or wealth management browsing semantic; This represents the business semantic unit corresponding to the i-th interaction event; This represents a semantic mapping function, which converts UI operations into corresponding business semantics based on the page structure and control type. It should be noted that semantic mapping functions This function is used to convert raw interface interaction events generated by mobile banking clients into operational semantic units with business meaning. It comprehensively analyzes the page identifier corresponding to the interaction event, the type of triggered operation, and the structural position of the control in the interface. Through the pre-established correspondence between page structure and business function, it performs semantic recognition on the user's click, swipe, or input behavior, thereby determining the actual meaning of the operation in the banking business process. Through this mapping mechanism, the raw interaction logs can be transformed into a stable business semantic sequence, enabling subsequent behavior path analysis to understand the user's operation characteristics from the business logic level, while avoiding the collection of any user's personal privacy information. S22. After obtaining the semantic units corresponding to each interaction event, a user interaction semantic sequence is constructed in chronological order to describe the user's complete operation process. A semantic compression mechanism is introduced to address the large number of repetitive clicks, swipes, or refreshes in actual operations, merging consecutively repeated semantic units to extract the core operation path that reflects the actual business process structure. Specifically: After obtaining the semantic units of each interaction event, construct the user interaction semantic sequence in chronological order: ; Since users may perform multiple repetitive actions during page browsing, such as continuous page scrolling, repeated clicks, or interface refreshes, these repetitive actions can cause semantic sequence redundancy. Therefore, a semantic compression function is introduced: ; The compressed semantic sequence is represented as: ; in, Represents the semantic sequence of a user during a single business operation; This represents the compressed semantic element; This represents a semantic compression function that merges consecutive semantic units that are identical. This represents the compressed semantic sequence; m represents the length of the compressed sequence. It should be noted that semantic compression functions This function is used to structurally merge consecutive repetitive semantic units in user interaction semantic sequences. Its core function is to reduce redundant semantic information generated by behaviors such as interface browsing, repeated clicks, or page refreshes without changing the user's actual business operation logic. By identifying the similarity relationship between adjacent semantic units in the time sequence, when multiple consecutive semantic units belong to the same business semantic category, they are integrated into a unified semantic node, and their duration and trigger frequency are recorded. This forms a more compact semantic path structure with business meaning, enabling subsequent behavior path analysis to more accurately reflect the user's actual operation process, while avoiding interference from interface details on behavior recognition results. It should be further explained that the above processing can simplify a large number of repetitive operations into a key business path structure. For example: Original sequence: Account page → Account page → Account page → Transfer page → Input information; Compressed sequence: Account page → Transfer page → Input information; S23. Based on the compressed semantic sequence, statistical analysis is performed on the transition relationships between adjacent semantic units. A semantic transition structure is established by calculating the transition frequency and probability between semantic units. Furthermore, a path stability index is calculated to reflect whether the user's operation path conforms to normal business process rules, thereby identifying abnormal path structures or unreasonable business jump behaviors. Specifically: To further analyze the user behavior path structure, statistically analyze the transition relationships between semantic units, and calculate the semantic transition frequency. ; Based on this, the semantic transition probability is calculated: ; Further define the path stability metric: ; in, Represents semantic units Transfer to semantic units The number of times it appears; Indicates performing semantic operations Later shift to semantics The probability of; Indicates all semantic The sum of the number of transfers at the starting point; Indicates semantic path stability; S24. Extract path features such as path length, semantic transfer complexity, and path stability based on the semantic path structure, and fuse these features with the device state fluctuation index and operation rhythm stability obtained in step S1 to construct a unified multimodal behavior path feature vector. This allows the behavior features to simultaneously reflect the user's business operation logic, device operating environment, and operation rhythm characteristics, providing more comprehensive and stable behavior description information for subsequent user behavior recognition models. Specifically: After obtaining the semantic path structure, it is fused with the device state features and operation rhythm features collected in step S1 to construct a unified behavioral path feature representation, and the semantic path length is calculated: ; The complexity of semantic transfer computation: ; Finally, construct the multimodal behavior path feature vector: ; in, Indicates the semantic path length; B represents semantic transfer entropy, used to measure the complexity of the operation path; B represents the behavioral path feature vector.

[0020] S3. By constructing a semantic phase rhythm time series, analyzing the overall rhythm stability, establishing a semantic phase rhythm structure spectrum, and generating a multidimensional behavioral rhythm fingerprint, the user operation rhythm characteristics are characterized from a time dimension. By combining business semantics with operation time, the distinction between natural operations and abnormal behaviors is achieved, providing interpretable and stable rhythmic features for multimodal behavior recognition. The specific implementation process is as follows: S31. Combine the compressed semantic sequence obtained in step S2 with the operation time interval to form a semantic stage rhythm sequence. Each rhythm unit contains both business semantic information and operation interval, ensuring that the operation rhythm of different business stages is captured in a structured manner, providing basic data for rhythm analysis. Specifically: Based on compressed semantic sequences Construct semantic phase rhythm sequences using timestamp sequences: ; Then, semantic phase rhythm units are constructed by combining business semantics: ; Form a complete semantic stage rhythm sequence R: ; in, This represents the time interval between the i-th semantic operation and the (i+1)-th semantic operation; T represents the basic rhythm sequence, which describes the user's operation speed at each stage of the entire business operation process. This represents the i-th semantic stage rhythm unit, which contains both business semantic information and... It also includes the operation interval. ; This represents a semantic rhythm mapping function that binds business operations to time intervals, enabling the system to distinguish the different meanings of the same time interval in different business operations. It should be noted that the semantic rhythm mapping function This function is used to associate the business semantics of each user operation with its corresponding operation time interval, and to transform interface interaction events into semantic phase rhythm units. Specifically, it analyzes the semantic phase in which the operation is located and the duration of the operation, embeds time information into the business semantic structure, and generates a unified representation that reflects the operation rhythm and business process relationship. This function can distinguish the natural rhythm differences of different business links, and provide basic data for subsequent rhythm stability analysis and behavior recognition. S32. Based on the semantic stage rhythm sequence, analyze the stability of the user's overall operation rhythm, calculate the average operation interval and standard deviation of fluctuation, and generate a rhythm stability index to characterize the consistency and naturalness of user operations, and distinguish normal behavior from abnormal behavior from the perspective of overall rhythm. Specifically: After obtaining the semantic phase rhythm sequence, it is necessary to measure the overall rhythm stability of user operations. Real users usually show a relatively stable operation rhythm in multiple operations, while the rhythm of automated operations or abnormal behaviors often appears too fixed or fluctuates abnormally. Calculate the average operation interval: ; Calculate the standard deviation of rhythm fluctuations: ; Introducing the overall rhythm stability index: ; S33. Calculate the average operational rhythm and rhythm dispersion for each semantic stage, construct a semantic stage rhythm structure spectrum to reflect the rhythmic characteristics of users in different business processes, capture rhythmic differences between stages, and enable the system to identify abnormal jumps or unnatural operation patterns. Specifically: To further characterize the differences in user operational rhythms at different business stages, it is necessary to construct a semantic stage rhythm structure spectrum, and to analyze the semantics of each business stage. Statistical operation time interval: ; ; Constructing a semantic rhythmic structure spectrum: ; in, Semantic phase The average operating rhythm; Semantic phase Number of operations; This indicates that it belongs to the semantic stage. The set of operations; Semantic phase The rhythm dispersion; S34. Combining the overall rhythm stability with the semantic stage rhythm structure spectrum generates a multidimensional behavioral rhythm fingerprint, including stability index, semantic consistency, average interval, and volatility, forming a quantifiable and interpretable rhythm feature vector. This provides core input for multimodal behavior recognition and anomaly detection, specifically: Based on the rhythm structure spectrum, a semantic rhythm consistency index is defined: ; Finally, a multi-dimensional behavioral rhythm feature vector (rhythmic fingerprint) is generated: .

[0021] S4. Based on the behavioral rhythm fingerprint obtained in step S3 and the behavioral path features obtained in step S2, a unified multimodal behavioral feature representation is constructed. Through the fusion analysis of path structure information, operation rhythm information, and device status features, a highly discriminative behavioral feature vector is formed. The feature representation capability is improved through standardization and weight allocation. Subsequently, a multimodal behavior recognition model is used to distinguish user operation behavior, identifying normal operation, abnormal operation, or potential automated operation. At the same time, the model's adaptability is continuously optimized through a dynamic feature update mechanism, thereby achieving mobile banking user behavior recognition without relying on user privacy data. The specific implementation process is as follows: S41. The behavioral path features obtained in step S2 and the behavioral rhythm fingerprint obtained in step S3 are fused together. A unified multimodal behavioral feature representation is constructed through feature vector concatenation. Semantic mapping is then performed on different operational stages in conjunction with the mobile banking business functional structure, establishing a correspondence between the operational path structure, business semantic information, and operational rhythm features. This results in the construction of a three-dimensional behavioral feature structure with business meaning, specifically: Obtain the behavior path feature vector generated in step S2 With the behavioral rhythm fingerprint generated in step S3 They are then fused to construct a unified multimodal feature vector: ; Where F represents the fused multimodal behavior vector; || represents the feature vector concatenation operation, which unifies the representation of path features and rhythm features; S42. Standardize the fused multimodal behavioral feature vectors to eliminate dimensional differences between features, ensuring each feature dimension falls within a uniform numerical range. Simultaneously, assign weights to each feature based on mobile banking operational logic and statistical analysis results, highlighting the importance of key behavioral paths and rhythmic features. Local normalization is also performed on outliers to enhance the system's sensitivity to abnormal behaviors and automated operations, improving the stability and robustness of the behavior recognition model. Specifically: To eliminate the impact of differences in feature units on recognition performance, a standardization method is used to process the fused feature vector: ; Further, based on business experience and statistical analysis, weights are assigned to different features. Obtain the standardized and weighted multimodal feature vectors: ; in, This represents the standardized feature vector; and These represent the mean and standard deviation of the fused feature vectors, respectively. This represents the standardized and weighted multimodal feature vector; It should be noted that weight Allocation can be made through historical normal user operation statistics, business logic importance, and machine learning feature importance analysis; S43. After completing feature standardization and weighting, a behavior recognition model is constructed using the weighted multimodal feature vector as input. A nonlinear mapping method is used to analyze the complex relationship between path structure features and operational rhythm features, mapping the fused features to specific user behavior categories. This identifies normal user operations, abnormal operations, and potential automated operations. Simultaneously, a feature importance analysis mechanism is combined to enhance the interpretability of the recognition results, enabling the behavior recognition results to correspond to specific operational paths and rhythmic features. Specifically: Standardized and weighted multimodal feature vectors Based on this, a nonlinear, multidimensional behavior recognition model is constructed; lightweight deep neural networks (such as multilayer perceptrons), random forests, or gradient boosting trees can be used to balance recognition accuracy and real-time performance. The goal of model training is to map the input feature vectors to the behavior category Y; S44. After the behavior recognition model outputs results, the system binds and stores the recognition results with the corresponding multimodal behavior features for subsequent statistical analysis and model optimization. Simultaneously, a dynamic feature update mechanism is introduced to weightedly fuse and update historical behavior features with the latest behavior data. This allows the model to continuously learn user behavior change trends and trigger security alerts or risk control strategies when abnormal behavior is detected, thereby ensuring the system's recognition accuracy and adaptability during long-term operation. Specifically: After model recognition is completed, the system generates the behavior discrimination result Y for each operation and combines it with the multimodal feature vector. Binding storage is used for subsequent statistical analysis and model updates; To enhance the system's adaptability to changes in user behavior, a dynamic feature update mechanism is introduced: ; Security policies or risk warnings can be triggered based on the identification results, such as reminding users to confirm the operation or perform secondary verification, while retaining historical data for model optimization. in, This represents the updated multimodal feature vector; Represents real-time multimodal feature vectors; This represents the latest acquired multimodal features; This represents the weights of historical features, ensuring that the model remembers users' long-term behavioral habits. .

[0022] Example 2: This invention proposes a mobile banking user behavior recognition system based on multimodal data, which is used to execute a mobile banking user behavior recognition method based on multimodal data proposed in Example 1, comprising: Memory; processor; A computer program stored in the memory and capable of running on the processor; The processor executes a computer program to implement a mobile banking user behavior recognition method based on multimodal data, as described in Embodiment 1 above.

[0023] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited thereto. Various changes can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of the present invention.

Claims

1. A method for recognizing mobile banking user behavior based on multimodal data, characterized in that, The specific implementation steps include the following: S1. By constructing a multimodal behavioral data collection structure in the mobile banking client that does not involve user identity information and personal privacy, the three types of data—interface interaction events, device operating status, and user operation rhythm—are collected synchronously and expressed in a unified structure to construct a multimodal behavioral feature unit. S2. Semantic analysis of mobile banking users' interface interaction behavior is performed. By establishing a business semantic mapping space, the original interface interaction events are converted into semantic units with business meanings and user interaction semantic sequences are constructed in chronological order. Semantic compression processing is performed to extract the actual business operation path structure. Then, the user operation process rules are analyzed by statistically analyzing the transfer relationship between semantic units. Multimodal behavior path feature vectors are constructed by combining device status features and operation rhythm features. S3. By constructing a semantic stage rhythm time series, analyzing the overall rhythm stability, and establishing a semantic stage rhythm structure spectrum, a multi-dimensional behavioral rhythm fingerprint is generated to characterize the user operation rhythm features from the time dimension, combining business semantics with operation time. S4. The obtained multimodal behavior path feature vectors are fused with multidimensional behavior rhythm fingerprints. A unified multimodal behavior feature representation is constructed through feature vector connection. After standardization and weight allocation, the representation is input into the multimodal behavior recognition model to judge user operation behavior. At the same time, the model's adaptability is continuously optimized through a dynamic feature update mechanism to complete the recognition of mobile banking user behavior.

2. The mobile banking user behavior recognition method based on multimodal data according to claim 1, characterized in that, Step S1 specifically includes: By establishing an interface interaction listening mechanism within the mobile banking client, the user's page access, control triggering, and operation confirmation behaviors in the application are recorded in real time. Each operation is abstracted into an interactive event unit containing page identifier, operation type, control structure position, and timestamp, and a sequence of interface interaction trajectories is constructed in chronological order. While collecting interface interaction events, the system performance interface synchronously records the mobile terminal's operating status information, including processor utilization, memory usage ratio, network response latency, and interface rendering frame rate. The device status fluctuation index is constructed by calculating the degree of device status change between adjacent events to characterize the device's operating stability during operation. The time interval between adjacent operations is calculated by using the timestamps of the interface interaction events, and a user operation rhythm time series is constructed. The average rhythm and the degree of rhythm fluctuation are further calculated to form a rhythm stability index that can reflect the rhythm characteristics of user operations. After obtaining the interface interaction trajectory, device operating status, and operation rhythm data, the three types of behavioral information are integrated according to a unified time series to construct a multimodal behavioral feature unit and form a complete multimodal behavioral data set, so that operation path information, device environment characteristics, and operation rhythm characteristics can be expressed in the same data structure.

3. The mobile banking user behavior recognition method based on multimodal data according to claim 2, characterized in that, Step S2, which involves semantic parsing of the user interface interaction behavior of mobile banking users, includes: Establish a business semantic mapping space based on the internal functional module structure of the mobile banking application, and construct the mapping relationship between the interface structure and business semantics; The collected page identifiers, control positions, and operation types are combined and parsed. The original interface interaction events are converted into corresponding business semantic units through a semantic mapping function, so that each operation is given a clear business meaning. This transforms the interface-level interaction behavior into an operation expression structure with business semantics.

4. The mobile banking user behavior recognition method based on multimodal data according to claim 3, characterized in that, Step S2, which involves constructing a user interaction semantic sequence and performing semantic compression, includes the following steps: After obtaining the semantic units corresponding to each interactive event, a user interaction semantic sequence is constructed in chronological order to describe the user's complete operation process. To address the numerous repetitive clicks, swipes, and refreshes encountered in actual operations, a semantic compression mechanism is introduced. This mechanism merges consecutively repeated semantic units using a semantic compression function, extracting the core operation path that reflects the actual business process structure.

5. The mobile banking user behavior recognition method based on multimodal data according to claim 4, characterized in that, Step S2, the steps for constructing the multimodal behavior path feature vector include: Based on the compressed semantic sequence, statistical analysis is performed on the transition relationship between adjacent semantic units, and a semantic transition structure is established by calculating the transition frequency and transition probability between semantic units. Calculate path stability metrics to reflect whether user operation paths conform to normal business process patterns and identify abnormal path structures or unreasonable business jump behaviors. Based on the semantic path structure, path length, semantic transfer complexity, and path stability index are extracted as path features. These features are then fused with the equipment state fluctuation index and operating rhythm stability to construct a unified multimodal behavior path feature vector.

6. The mobile banking user behavior recognition method based on multimodal data according to claim 5, characterized in that, Step S3, the steps for constructing the semantic stage rhythm time series, include: Based on compressed semantic sequences and timestamp sequences, a basic rhythm sequence is constructed, which describes the user's operation speed at each stage of the entire business operation process. Semantic phase rhythm units are constructed by combining business semantics. Each rhythm unit contains both business semantic information and operation intervals. Business operations are bound to time intervals through a semantic rhythm mapping function, enabling the system to distinguish the different meanings of the same time interval in different business operations. Arrange semantic phase rhythm units to form a complete semantic phase rhythm time sequence, ensuring that the operational rhythm of different business phases is captured in a structured manner.

7. The mobile banking user behavior recognition method based on multimodal data according to claim 6, characterized in that, Step S3, which involves analyzing overall rhythm stability and establishing a semantic stage rhythm structure spectrum, includes the following steps: Based on the semantic stage rhythm time series, the stability of the overall user operation rhythm is analyzed, the average operation interval and the standard deviation of fluctuation are calculated, and a rhythm stability index is generated to characterize the consistency and naturalness of user operations. For each semantic stage, the average operating rhythm and rhythm dispersion are calculated to construct a semantic stage rhythm structure spectrum, which reflects the rhythm characteristics of users in different business links and captures the rhythm differences between stages, enabling the system to identify abnormal jumps or unnatural operating patterns.

8. The mobile banking user behavior recognition method based on multimodal data according to claim 7, characterized in that, Step S3, the steps for generating a multidimensional behavioral rhythm fingerprint, include: Define a semantic rhythm consistency index based on the rhythm structure spectrum; By combining the overall rhythm stability with the semantic stage rhythm structure spectrum, a multidimensional behavioral rhythm feature vector containing stability index, semantic consistency, average interval and volatility is generated, forming a quantifiable and interpretable multidimensional behavioral rhythm fingerprint.

9. A method for recognizing mobile banking user behavior based on multimodal data according to claim 8, characterized in that, Step S4 specifically includes: By fusing behavioral path features with behavioral rhythm fingerprints, a unified multimodal behavioral feature representation is constructed through feature vector connection. Semantic mapping is performed on different operation stages in conjunction with the functional structure of mobile banking business, so that the operation path structure, business semantic information and operation rhythm features form a correspondence, and a three-dimensional behavioral feature structure with business meaning is constructed. The fused multimodal behavioral feature vectors are standardized to eliminate the dimensional differences between different features and bring all feature dimensions into a uniform numerical range. Based on the operational logic of mobile banking and the results of statistical analysis, weights are assigned to each feature to highlight the importance of key behavioral paths and rhythmic features. Abnormal extreme values ​​are locally normalized to obtain standardized and weighted multimodal feature vectors. After feature standardization and weighting, a multimodal behavior recognition model is constructed using the weighted multimodal feature vector as input. The complex relationship between path structure features and operation rhythm features is analyzed through nonlinear mapping methods. The fused features are mapped to specific user behavior categories to identify normal user operations, abnormal operations, and potential automated operations. At the same time, the interpretability of the recognition results is enhanced by combining feature importance analysis mechanism. After the multimodal behavior recognition model outputs results, the recognition results are bound and stored with the corresponding multimodal behavior features for subsequent statistical analysis and model optimization. At the same time, a dynamic feature update mechanism is introduced to weightedly fuse and update historical behavior features with the latest behavior data, enabling the model to continuously learn the trend of user behavior changes and trigger security prompts or risk control strategies when abnormal behavior is detected.

10. A mobile banking user behavior recognition system based on multimodal data, characterized in that, include: Memory; processor; A computer program stored in the memory and capable of running on the processor; The processor executes a computer program to implement the mobile banking user behavior recognition method based on multimodal data as described in any one of claims 1 to 9.

Images