Method and system for human-computer interaction behavior analysis for financial scenarios

Abstract

The application discloses a human-computer interaction behavior analysis method and system for a financial scene, and the method comprises the following steps: collecting multi-source heterogeneous data through an intelligent building perception network, constructing a multi-modal space-time tensor through space-time alignment, dynamically calibrating the service state of a functional area in combination with a BIM model and a graph attention network, embedding context features, generating a comprehensive interaction behavior vector by adopting cross-attention Transformer to fuse multi-dimensional features, realizing behavior classification, intention reasoning and path prediction through a hierarchical model, detecting abnormal behaviors by using a variational autoencoder and grading early warning, and solving a service resource dynamic configuration strategy based on a mixed integer programming model, and meanwhile, an edge end privacy protection mechanism is matched. The system comprises the following steps: multi-source perception data collection, space-time alignment and tensor construction, dynamic semantic segmentation and context embedding, multi-modal feature fusion, hierarchical behavior analysis, and abnormal behavior detection and early warning. The application can realize full-link behavior analysis of a financial network, real-time risk control and operation resource optimization.

Description

Technical Field

[0001] This invention relates to the field of intelligent building sensing and financial branch operation optimization technology, specifically a human-computer interaction behavior analysis method and system for financial scenarios. Background Technology

[0002] As the digital transformation of finance deepens, online business has covered most standardized transactions, but physical bank branches remain the core channel for handling complex transactions, serving high-value customers, and showcasing the brand. Currently, the number of physical bank branches nationwide remains above 200,000, and more than 60% of complex transactions such as loans and wealth management still need to be completed offline. However, the current operation of these branches generally faces the following problems:

[0003] First, there is an imbalance in operational efficiency, with excessive average customer wait times during peak hours and high resource idle rates during off-peak hours. Second, risk management is lagging behind, with existing security systems primarily relying on post-incident tracing and lacking real-time anomaly warning capabilities. Third, customer experience is inadequate, with an inability to accurately identify customer needs and poor service targeting.

[0004] Among them, there are still significant shortcomings in the existing intelligent transformation technology for branches:

[0005] The ability to integrate multi-source data is weak. The sampling frequency and spatial reference system of data from different sensing devices are not uniform, resulting in data silos. It is difficult to build a complete customer behavior profile, and the accuracy of behavior recognition is low. The behavior analysis dimension is single, relying only on individual visual or device interaction features, ignoring the influence of spatial context and group environment, and the reasoning of business intent is inaccurate. The resource allocation is static, with fixed schedules based on historical data, which cannot respond to fluctuations in passenger flow in real time.

[0006] To address the aforementioned shortcomings, this application proposes a comprehensive solution for analyzing and optimizing human-computer interaction behavior at financial outlets. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention discloses a human-computer interaction behavior analysis method and system for financial scenarios, in order to solve the problems mentioned in the background.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a human-computer interaction behavior analysis method for financial scenarios, comprising the following steps:

[0009] S1. Collect multi-source heterogeneous sensing data through an intelligent building sensing network deployed in financial physical outlets; the intelligent building sensing network includes UWB indoor positioning base stations, high-definition cameras, self-service terminal interaction interfaces, environmental sensors, and counter business system interfaces; the multi-source heterogeneous sensing data includes environmental status data, customer visual data, customer 3D positioning data, device interaction data, and business attribute data, wherein the business attribute data includes business type, processing time, device business status, and transaction amount;

[0010] S2. Spatiotemporal alignment processing is performed on multi-source heterogeneous sensing data with different sampling frequencies and different spatial reference systems to construct multimodal spatiotemporal tensor data;

[0011] S3. Based on the intelligent building BIM spatial information model, the initial functional areas are divided, and the financial physical network is modeled as a directed passenger flow graph by combining real-time passenger flow data, where the functional areas are the graph nodes. The passenger flow transfer relationship between regions is a directed edge The graph attention network (GAT) is used to update the features of the directed passenger flow graph, dynamically label the service status of each functional area during the current business period, and embed the service status of the area as a contextual feature into the subsequent behavior analysis model; the service status labels include idle, normal, busy, fault, and maintenance.

[0012] S4. Extract individual micro-behavioral features, human-computer interaction features, interpersonal interaction features, and group context features of customers; after layer normalization of each feature, they are concatenated into a feature matrix; a multimodal Transformer fusion network based on cross-attention mechanism is used to map heterogeneous features to a unified representation space to generate a comprehensive interactive behavior vector CIBV.

[0013] S5. Based on the time-series sequence of comprehensive interactive behavior vectors, a hierarchical behavior analysis model is constructed to realize real-time behavior classification, business intent reasoning and behavior path prediction in sequence.

[0014] S6. Input the comprehensive interactive behavior vector into the pre-trained variational autoencoder model to calculate the reconstruction error. When the reconstruction error is greater than the preset threshold obtained based on the statistics of normal samples, it is judged as abnormal interactive behavior, and a graded early warning information is sent to the management decision system, triggering the linkage response of the intelligent building system.

[0015] S7. Input the behavioral analysis results into the mixed integer programming model, and solve for the dynamic allocation strategy of service resources with the goal of achieving the optimal synergy between average customer waiting time and operational resource utilization.

[0016] Preferably, in step S2, the spatiotemporal alignment process includes:

[0017] The financial physical network of branches is divided into a three-dimensional spatial grid of 1m×1m×3m, with a total of N grids; a unified timestamp is provided using BeiDou time synchronization. A three-dimensional spatial coordinate system with the network entrance as the origin. Based on this, all perceived data are mapped to a three-dimensional spatiotemporal tensor:

[0018] ,

[0019] Where: T is the aligned time series length, Ω is the spatial network of the financial scenario.

[0020] A lattice set, where D is the number of dimensions of the perceptual modality; each tensor element This represents the perception value of the d-th perception modality at time t within the n-th spatial grid, thus constructing multimodal spatiotemporal cube data for financial scenarios.

[0021] Preferably, the construction and feature update of the directed passenger flow transfer map in step S3 specifically includes:

[0022] Node features Includes the current passenger flow density of region i, the operating status of equipment within the region, and the current number of people in the queue; edge features The historical passenger flow transfer probability from region i to region j is calculated, with the statistical period being the same time period of the most recent 7 working days and the update frequency being once every 15 minutes. The attention weight of each node is calculated using a graph attention network (GAT), and the node features are aggregated and updated to dynamically label the service status of each functional area.

[0023] Preferably, the feature extraction and multimodal fusion in step S4 specifically includes:

[0024] Extracted individual micro-behavioral characteristics Including walking speed, dwell time, head orientation, and hand gestures; human-computer interaction features This includes click frequency, error rate, input duration, and page navigation path; interpersonal interaction characteristics. This includes interaction distance, interaction duration, and voice intensity; group contextual features. This includes regional passenger density, queue length, and group movement direction;

[0025] Layer normalization was performed on the four types of heterogeneous features according to their feature dimensions, resulting in:

[0026] ,

[0027] The normalized features are concatenated into a feature matrix:

[0028] ,

[0029] A multimodal Transformer fusion network based on a cross-attention mechanism is adopted. Individual micro-behavioral features are used as the query matrix Q, and human-computer interaction features, interpersonal interaction features, and group context features are concatenated as the key matrix K and value matrix V. Cross-attention is calculated and residual connections are performed to generate a comprehensive interaction behavior vector CIBV, which satisfies:

[0030] ,

[0031] The cross-attention calculation function is as follows:

[0032] ,

[0033] The feature dimension of the query matrix Q is specified; the multimodal Transformer fusion network contains 3 encoder layers, each with 8 attention heads.

[0034] Preferably, the construction of the hierarchical behavior analysis model in step S5 specifically includes:

[0035] 1) Real-time behavior classification layer: A three-layer temporal convolutional network is used to extract the temporal dependency features of the comprehensive interactive behavior vector sequence to identify the current behavior stage of the customer. The behavior stages include entering the branch, waiting in line, operating equipment, consulting business, and leaving the branch.

[0036] 2) Business Intent Inference Layer: Construct a Bayesian network containing business intent nodes, behavioral feature nodes, historical behavior nodes, and reservation information nodes, based on the comprehensive interactive behavior vector sequence from time 1 to time t. Customer historical behavior characteristics and customer appointment information characteristics Calculate the posterior probability of each business intent:

[0037] ,

[0038] Where b is a business intent category variable, including deposit, withdrawal, transfer, loan consultation, financial consultation and loss reporting; prior probabilities are obtained based on the statistical data of branch business over the past 3 months;

[0039] 3) Behavioral path prediction layer: The attention-enhanced sequence-to-sequence model is adopted. The historical comprehensive interaction behavior vector sequence is used as input. The attention mechanism is applied to the functional area transfer sequence to predict the probability of the customer's functional area transfer and the terminal devices that may be reached in the next 30 minutes.

[0040] Preferably, the abnormal interaction behavior detection and risk warning in step S6 specifically includes:

[0041] The variational autoencoder model was pre-trained using a historical dataset containing only normal interaction behavior samples. The pre-training epochs were 100, and the batch size was 32.

[0042] Calculate the reconstruction error of the input comprehensive interaction behavior vector:

[0043] ,

[0044] Where Enc is the encoder of the variational autoencoder, and Dec is the decoder of the variational autoencoder; preset threshold Take the 95th percentile of the reconstruction error for normal samples; when the reconstruction error When the error occurs, it is judged as abnormal interaction behavior; the warning is divided into three levels according to the size of the reconstruction error: the first level warning triggers on-site patrol by the security personnel of the site, the second level warning triggers temporary locking of self-service equipment, and the third level warning triggers linkage alarm of the public security system.

[0045] Preferably, in step S7, solving the dynamic configuration strategy for service resources specifically includes constructing an optimization objective function for a mixed-integer programming model:

[0046] ,

[0047] Where W represents the average customer wait time, and U represents the operational resource utilization rate; weighting coefficients and The analytic hierarchy process (AHP) was used to determine that the following conditions were met. The dynamic configuration strategies for service resources obtained from the solution include: dynamic counter scheduling strategy, self-service equipment load balancing strategy, personalized service triggering strategy, and environmental temperature and humidity adjustment strategy.

[0048] Preferably, the method further includes a data privacy protection step:

[0049] Sensitive data collected is locally anonymized at the edge computing nodes of the sensing devices; the original face image data is deleted within 100ms after facial features are extracted; customer trajectory data is anonymized using the k-anonymity algorithm to remove all identifying information that can directly identify an individual, such as the customer's ID number, mobile phone number, bank card number, and facial feature vector, so that the processed behavioral analysis data cannot be associated with a specific natural person.

[0050] This invention also provides a human-computer interaction behavior analysis system for financial scenarios, comprising:

[0051] The multi-source sensing data acquisition module is used to collect multi-source heterogeneous sensing data through the intelligent building sensing network deployed in financial physical outlets;

[0052] The spatiotemporal alignment and tensor construction module is used to perform spatiotemporal alignment processing on multi-source heterogeneous sensing data and construct multimodal spatiotemporal tensor data.

[0053] The dynamic semantic segmentation and context embedding module is used to perform dynamic semantic segmentation on financial physical outlets, divide functional areas and dynamically label service status, and embed the area status as context features into the behavior analysis model.

[0054] The multimodal feature fusion module is used to extract multi-dimensional interactive behavior features of customers. It uses a multimodal fusion network to map heterogeneous features to a unified representation space and generate a comprehensive interactive behavior vector.

[0055] The hierarchical behavior analysis module is used to build hierarchical behavior analysis models to achieve real-time customer behavior classification, business intent reasoning, and behavior path prediction.

[0056] The abnormal behavior detection and early warning module is used to detect abnormal interactive behaviors and trigger tiered early warnings and intelligent building system linkage responses.

[0057] The service resource optimization and configuration module is used to solve the dynamic configuration strategy of service resources based on the results of behavior analysis.

[0058] The edge privacy protection module is used to perform localized de-identification processing on the collected sensitive data.

[0059] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0060] 1. This invention constructs a unified representation framework for multimodal spatiotemporal tensors, which realizes spatiotemporal alignment of multi-source heterogeneous data through a unified spatiotemporal benchmark. It adopts a cross-attention Transformer with individual micro-behavior as the query to deeply fuse four types of heterogeneous features: individual, human-machine, interpersonal, and group, and generate a comprehensive interactive behavior vector. This solves the problem of insufficient behavior representation ability caused by data silos in existing technologies and significantly improves the accuracy of full-link behavior analysis.

[0061] 2. This invention proposes a spatial context embedding method that combines BIM and GAT. Based on the intelligent building BIM model, a dynamic directed passenger flow transfer map is constructed. The features of the neighborhood area are aggregated through the graph attention network and the service status is labeled in real time. The spatial context features are embedded into the behavior analysis model, which solves the defect of isolated modeling of individual behaviors in the existing technology and greatly improves the accuracy of business intent reasoning and behavior path prediction.

[0062] 3. This invention forms a closed-loop system covering perception, analysis, early warning, and optimization. It integrates unsupervised variational autoencoder anomaly detection and mixed integer programming resource optimization modules, which significantly reduces the false alarm rate of anomaly detection, effectively shortens the average customer waiting time, and improves the utilization rate of operational resources. It is equipped with a full-process privacy protection mechanism at the edge, achieving optimal synergy between security, efficiency, and data compliance in financial outlets. Attached Figure Description

[0063] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0064] In the attached diagram:

[0065] Figure 1 This is a flowchart of a human-computer interaction behavior analysis method for financial scenarios according to the present invention;

[0066] Figure 2 This is a simplified diagram of the module framework of a human-computer interaction behavior analysis system for financial scenarios according to the present invention. Detailed Implementation

[0067] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0068] This embodiment provides a method for analyzing human-computer interaction behavior in financial scenarios. The overall process is as follows: Figure 1 As shown, the specific steps include:

[0069] S1. Multi-source heterogeneous sensing data acquisition: Multi-source heterogeneous sensing data is acquired through the intelligent building sensing network deployed in financial physical outlets.

[0070] (1) Deployment architecture and parameter basis of the perception network:

[0071] UWB indoor positioning base stations: Deployed on the ceiling of the branch, spaced 6-8m apart, with a positioning accuracy of ±10cm and a sampling frequency of 10Hz. Selection criteria: The normal walking speed of customers in financial branches is 0.5-1.5m / s. 10Hz sampling ensures that a positioning point is obtained every 0.1m, which is sufficient to capture continuous walking trajectories and turning actions. If the sampling frequency is lower than 5Hz, key behavioral characteristics such as rapid turns and sudden stops will be lost; if it is higher than 20Hz, redundant data will be generated, increasing the load on edge computing nodes by more than 3 times.

[0072] High-definition AI cameras: Deployed diagonally across various functional areas, with a resolution of 1920×1080 and a frame rate of 25fps. Selection criteria: 25fps aligns with the persistence of vision effect in the human eye, ensuring continuous capture of rapid movements such as gestures and head movements, while also meeting the minimum frame rate requirements for video surveillance in the "Security Requirements for Bank Premises" (GA38-2021); 1080P resolution allows for clear identification of key human points and gestures within a 3-meter distance, while resolutions higher than 2K would increase video transmission bandwidth requirements by four times.

[0073] Environmental sensors: Temperature and humidity sensor with a sampling frequency of 1Hz, and human infrared sensor with a sampling frequency of 5Hz. Selection criteria: Temperature and humidity change slowly, and 1Hz sampling is sufficient to reflect the environmental conditions; human infrared sensor is used to detect people entering and exiting, and 5Hz sampling can promptly capture personnel movement and avoid missed detections.

[0074] Interface integration: All self-service terminals, queuing machines, and counter business systems have reserved RESTful API interfaces, with data push latency ≤500ms, meeting the requirements for real-time analysis.

[0075] Data types collected include: environmental status data (temperature, humidity, light intensity); customer visual data including coordinates of 17 key points on the human body, head orientation angle, and gesture types; customer 3D positioning data including x / y / z coordinates, movement speed, and movement direction; device interaction data including click coordinates, input duration, page jump time, and number of operation errors; and business attribute data including business type, processing time, device business status, transaction amount, and customer appointment information.

[0076] S2. Spatiotemporal alignment processing and multimodal spatiotemporal tensor construction: Spatiotemporal alignment processing is performed on multi-source heterogeneous sensing data with different sampling frequencies and different spatial reference frames to construct multimodal spatiotemporal tensor data. The specific process is as follows:

[0077] (1) Three-dimensional spatial grid division: The entire financial physical network is divided into a 1m×1m×3m three-dimensional spatial grid, with a total of N grids. Each grid corresponds to a unique spatial index n, where n=1,2,...,N. A right-handed three-dimensional spatial coordinate system (x,y,z) is established with the center of the ground at the network entrance as the origin. The x-axis points in the depth direction of the network interior, the y-axis points in the width direction of the network, and the z-axis points in the direction perpendicular to the ground upwards. Selection criteria:

[0078] (2) Horizontal resolution 1m: The spacing between self-service equipment in financial outlets is generally 1.2-1.5m, and the queuing channel width is 1m. A 1m grid can accurately distinguish the equipment and queuing position of the customer. If a 0.5m grid is used, the tensor dimension will increase by 4 times, and the amount of computation will increase exponentially. If a 2m grid is used, the behavioral differences between adjacent areas will be lost, and it will be impossible to identify the customer's switching between two adjacent self-service equipment.

[0079] (3) Vertical height 3m: Covers the range of activity of customers standing (1.7m) and sitting (1.2m), while excluding interference data from the ceiling (above 3m) and the ground (below 0m).

[0080] (4) Unified time reference: The BeiDou satellite timing system is used to provide a unified timestamp for all sensing devices with a time accuracy of 1ms. The timestamps of all sensing data are aligned to a 1s granularity to form a unified time series. , where T is the length of the aligned time series.

[0081] Selection criteria: A 1-second granularity can cover the sampling interval of all sensing devices (1-10Hz) while ensuring the real-time performance of behavior analysis; a granularity of less than 1 second will result in a data storage volume increase of more than 10 times, while a granularity of more than 1 second will result in the loss of instantaneous action information such as customer clicks and gestures.

[0082] (5) Construction of multimodal spatiotemporal tensors: Map all sensing data to a unified spatiotemporal reference to generate a three-dimensional spatiotemporal tensor. Ω represents the spatial grid set, and D=12, which includes 3D positioning, 4D vision, 3D environment, and 2D device interaction. Each tensor element... This represents the perception value of the d-th perception modality at time t within the n-th spatial grid, thus constructing multimodal spatiotemporal cube data for financial scenarios.

[0083] Selection criteria: The 12-dimensional perceptual modality covers all the core data required for financial branch behavior analysis. After feature importance assessment, adding other modalities (such as sound frequency) will not improve the model accuracy by more than 0.8%, but will instead increase the tensor dimension and computational cost.

[0084] S3. Dynamic Area Modeling and Contextual Feature Embedding: Based on the intelligent building BIM spatial information model, initial functional areas are divided. Combined with real-time passenger flow data, the financial physical network is modeled as a directed passenger flow transfer graph. A graph attention network (GAT) is used to dynamically calibrate the service status of each functional area. The specific process is as follows:

[0085] (1) Initial functional area division: Import the BIM model of the financial branch, extract the spatial boundary information of walls, partitions, equipment, etc., and automatically divide the initial functional areas such as entrance area, waiting area, self-service area, cash counter area, non-cash counter area, financial consultation room, and restrooms. Each area corresponds to a section. .

[0086] (2) Construction of directed passenger flow transfer map: Count the number of passenger flow transfers between regions during the same time period in the last 7 working days, and calculate the historical passenger flow transfer probability from region i to region j:

[0087] ,

[0088] in Let M be the number of passenger transfers from region i to region j, and M be the total number of functional regions. Directed edges are constructed using passenger transfer probabilities as edge features. This forms a directional passenger flow transfer map. The edge features are updated every 15 minutes.

[0089] Selection criteria:

[0090] The statistical period is 7 working days: the customer flow of financial outlets has a significant weekly cycle (the customer flow pattern is similar from Monday to Friday), and 7 working days can cover the complete weekly customer flow distribution; if 3 working days are used, the sample size is insufficient and the shift probability deviation exceeds 15%; if 1 month is used, outdated business models (such as temporary promotional activities) will be included.

[0091] Update frequency of 15 minutes: Customer flow at the outlet will fluctuate significantly within 15 minutes (such as a group of customers leaving after completing their business). A 15-minute update can ensure the real-time status of the area. If the update frequency is 5 minutes, GAT calculation will be too frequent, and the system CPU usage will increase by 40%. If it is 30 minutes, the area status will lag behind the actual changes in customer flow, resulting in untimely allocation of service resources.

[0092] (3) Node feature aggregation and state labeling: initial features of each node ,in Let be the current passenger flow density (people / m²) in area i. This represents the operating status of equipment within the area, where 0 = fault, 1 = normal, and 2 = maintenance. Let be the current number of people queuing in region i. A two-layer graph attention network (GAT) is used to calculate the attention weights of each node, and the features of neighboring nodes are aggregated and updated.

[0093] ,

[0094] in Let be the set of neighboring nodes of node i. For attention weights, Let σ be the learnable weight matrix of the l-th layer, and σ be the ReLU activation function.

[0095] Selection criteria: A 2-layer GAT can aggregate the features of second-order neighborhoods, which is sufficient to capture the impact of passenger flow in adjacent areas (such as an increase in passenger flow in the waiting area leading to a shift in passenger flow to the self-service area); a 3-layer or higher GAT will have an oversmoothing problem, resulting in a decrease in the distinguishability of node features in different areas by more than 20%.

[0096] Dynamically label the regional service status based on the updated node characteristics:

[0097] when people / and The time is marked as "idle";

[0098] When 0.1≤ρ_i≤0.5; ㎡ and The time stamp is marked as "normal";

[0099] when people / , The time is marked as "busy";

[0100] when The time stamp is marked as "fault";

[0101] when The time stamp is marked as "maintenance".

[0102] Selection criteria: The passenger flow density thresholds of 0.1 people / m² and 0.5 people / m² are aligned with the "Fire Safety Management of Densely Populated Places" (GA654-2020). 0.1 people / m²... 2 The following represents a sparsely populated state, while a density of 0.5 people / m² or higher indicates a crowded state, both of which meet the safety operation standards of the financial industry. The regional service status label is encoded as a 5-dimensional one-hot vector and embedded as a contextual feature into the subsequent behavior analysis model.

[0103] S4. Multimodal Feature Extraction and Fusion

[0104] Multi-dimensional customer interaction behavior features are extracted, and a multimodal Transformer fusion network based on a cross-attention mechanism is used to generate a comprehensive interaction behavior vector (CIBV). The specific process is as follows:

[0105] (1) Multi-dimensional feature extraction:

[0106] Individual micro-behavioral characteristics : Extract walking speed (m / s), dwell time (s), head orientation angle (°), and gesture categories including 0=no gesture, 1=waving, 2=pointing, 3=grabbing from UWB positioning data and visual data to form a 4-dimensional feature vector;

[0107] Human-computer interaction features : Extract click frequency (times / minute), operation error rate (%), single input duration (s), and page jump path length from the self-service terminal interface data to form a 4-dimensional feature vector;

[0108] Interpersonal interaction characteristics Extract interpersonal interaction distance (m), interaction duration (s), and voice intensity (dB) from visual data and voice sensor data to form a 3D feature vector;

[0109] Group context features Extract customer flow density (people / ㎡), queue length (people), and group movement direction angle (°) of the customer's area from the multimodal spatiotemporal tensor to form a 3D feature vector.

[0110] Selection criteria: Through feature selection experiments (using random forest feature importance assessment), the above 14 features contribute more than 95% to the differentiation of customer behavior in financial outlets; adding other features (such as customer height and clothing color) improves the model accuracy by less than 1%, but instead increases the feature dimension and computational load; removing any core feature will decrease the model accuracy by 3%-7%.

[0111] (2) Feature normalization and concatenation: The four types of heterogeneous features are subjected to layer normalization according to the feature dimension:

[0112] ,

[0113] in and Let be the mean and standard deviation of the k-th feature. and Let ε be a learnable scaling and translation parameter, ε = 1e-8 (to prevent local minima caused by division by zero, conforming to general deep learning conventions). The normalized features are then concatenated row-wise into a feature matrix.

[0114] .

[0115] (3) Cross-attention fusion: A multimodal Transformer fusion network with 3 encoder layers and 8 attention heads per layer is used for feature fusion. Individual micro-behavioral features are integrated. As the query matrix Q, Concatenate the key matrix K and value matrix V, and calculate the cross attention:

[0116] ,

[0117] in (Feature dimensions of the query matrix Q). A residual concatenation is performed between the cross-attention output and the original query matrix to generate a 128-dimensional comprehensive interaction behavior vector CIBV: CIBV = CrossAttn(Q,K,V) + ;

[0118] Selection criteria: Transformer configuration, 3-layer encoder can capture feature associations at different levels, including the bottom layer capturing local features and the high layer capturing global associations, and 8 attention heads can compute the dependencies between different features in parallel; Experimental verification shows that the 3-layer 8-head configuration has a 3.2% higher accuracy than the 2-layer 4-head configuration, and only a 0.5% higher accuracy than the 4-layer 16-head configuration, but the inference time is doubled, so it is the optimal configuration.

[0119] CIBV 128-dimensional vector: Ablation experiments have verified that a 128-dimensional vector can fully represent the information of four types of heterogeneous features; 64-dimensional vectors will lead to information loss and decreased accuracy; 256-dimensional vectors will increase model parameters, increase inference time, and make it difficult to improve accuracy.

[0120] S5, Hierarchical Behavior Analysis

[0121] A hierarchical behavior analysis model is constructed based on the time-series sequence of comprehensive interaction behavior vectors, which sequentially realizes real-time behavior classification, business intent reasoning, and behavior path prediction, specifically as follows:

[0122] (1) Real-time behavior classification layer: A three-layer temporal convolutional network (TCN) is used to extract the temporal dependency features of the CIBV temporal sequence. The kernel sizes are 3, 3, and 5, and the number of output channels per layer is 64. The ReLU activation function and Dropout are used, with a dropout rate of 0.2 to prevent overfitting. Finally, the probability distribution of five behavioral stages is output through a fully connected layer and a softmax function to identify the current behavioral stage of the customer: entering the branch, waiting in line, operating equipment, consulting services, and leaving the branch.

[0123] Selection criteria:

[0124] TCN layer count and convolution kernel: A 3-layer TCN combined with convolution kernel sizes of 3, 3, and 5 can capture temporal dependencies of 1-10 seconds, covering the time scale of a single behavior of a customer at a financial branch; if a 2-layer TCN is used, it cannot capture long-term temporal behavioral features, and the accuracy drops by 5.3%.

[0125] Dropout rate 0.2: Experiments have shown that a dropout rate of 0.2 can effectively prevent overfitting without affecting the model's convergence speed; a dropout rate higher than 0.3 will lead to underfitting of the model, while a dropout rate lower than 0.1 will not have a regularization effect.

[0126] (2) Business Intent Inference Layer: Construct a Bayesian network containing four types of nodes: business intent nodes, with six states including deposit, withdrawal, transfer, loan consultation, financial consultation, and loss reporting; behavioral feature nodes (corresponding to CIBV vectors); historical behavior nodes (corresponding to the customer's business records for the past 3 months); and appointment information nodes (corresponding to the customer's appointment business type). Based on the CIBV sequence from time 1 to time t. Customer historical behavior characteristics and customer appointment information characteristics Calculate the posterior probability of each business intent:

[0127] ,

[0128] Where prior probability This is based on statistical analysis of business data from the branch offices over the past three months. The business intent with the highest posterior probability is taken as the inference result.

[0129] Selection criteria: 3 months of business data can cover the business distribution of different seasons and holidays, while avoiding the impact of outdated data; 1 month of data has insufficient sample size and the prior probability deviation exceeds 12%; 6 months of data will include offline business models, which will lead to a decrease in inference accuracy.

[0130] (3) Behavior path prediction layer: An attention-enhanced sequence-to-sequence (Seq2Seq) model is adopted, with both the encoder and decoder being 2-layer LSTM networks and the hidden layer dimension being 128. The historical CIBV sequence is used as the encoder input, and an attention mechanism is applied to the functional area transfer sequence to predict the probability distribution of the customer's access to each functional area and the possible terminal device IDs within the next 30 minutes.

[0131] Selection criteria: LSTM configuration; a 2-layer LSTM can capture long-term temporal dependencies, and a 128-dimensional hidden layer can fully represent behavioral sequence information; a 3-layer LSTM will lead to the vanishing gradient problem, increasing training difficulty. Prediction duration: 30 minutes. The average customer dwell time in financial branches is 18 minutes, and 30 minutes can cover more than 90% of customers' complete business processes; prediction accuracy drops below 55% after 60 minutes, making it impractical; prediction within 15 minutes cannot cover the complete business process and cannot be configured with resources in advance.

[0132] S6. Abnormal Interaction Behavior Detection and Tiered Early Warning

[0133] The comprehensive interaction behavior vector is input into a pre-trained variational autoencoder model to calculate the reconstruction error, thereby achieving abnormal behavior detection and hierarchical early warning. The specific process is as follows:

[0134] (1) Model pre-training: Collect normal interaction behavior samples from network points over the past 6 months to construct a training dataset (sample size ≥ 100,000). Train a variational autoencoder (VAE) model. The encoder consists of 3 fully connected layers with dimensions of 128→64→32, and the decoder consists of 3 fully connected layers with dimensions of 32→64→128. The activation function is ReLU. The pre-training rounds are 100, the batch size is 32, the optimizer is Adam (learning rate = 0.001), and the loss function is the sum of reconstruction error and KL divergence.

[0135] Selection criteria: VAE structure; a symmetrical structure with 3 fully connected layers is the standard configuration for autoencoders, effectively learning the low-dimensional feature distribution of normal behavior; halving the dimension layer by layer can progressively compress features and extract core information. Pre-training parameters: The model loss converges after 100 epochs (loss decrease <0.1% / epoch), further training will lead to overfitting; batch size of 32 is the optimal batch size for deep learning, balancing the accuracy of gradient estimation and training speed; learning rate of 0.001 is the default optimal value for the Adam optimizer, providing the fastest convergence speed in this scenario.

[0136] (2) Anomaly detection: For the input CIBV vector, calculate its reconstruction error:

[0137] ,

[0138] Where Enc is the encoder and Dec is the decoder. Preset threshold. Take the 95th percentile of the reconstruction error of all normal samples in the training set. When the behavior is deemed abnormal, it is considered an abnormal interaction. Selection criteria: In statistics, the 95% confidence interval is a commonly used standard for anomaly detection, ensuring that 95% of normal behavior will not be misjudged as abnormal, while detecting over 90% of abnormal behavior. Using the 90th percentile would increase the false positive rate by 3 times; using the 99th percentile would increase the false negative rate by 2.5 times. Therefore, 95% is the optimal balance between false positive and false negative rates.

[0139] (3) Tiered early warning and coordinated response: The early warning is divided into three levels according to the magnitude of the reconstruction error:

[0140] Level 1 warning <ε≤1.5 : Sends an alert to the branch management system, triggering on-site patrols by branch security personnel;

[0141] Level II warning (1.5) <ε≤ 2 ): Temporarily lock the self-service device currently being used by the customer, and notify the branch lobby manager to handle the situation;

[0142] Level III warning ( This immediately triggers a coordinated alarm within the public security system, simultaneously locking all self-service devices at the branch and activating the emergency response plan. Selection criteria: Verified through abnormal sample testing, the reconstruction error is within 1-1.5. Most of the anomalies were due to customer misoperation or temporary pauses (high false alarm rate); 1.5-2 The anomalies in between were mostly suspicious operations, including entering the wrong password multiple times and lingering for extended periods; 2 Most of the above anomalies are malicious behaviors, including brute-force attacks on devices or damage to self-service terminals. Therefore, setting these two thresholds enables tiered early warning and precise handling.

[0143] S7, Dynamic Configuration of Service Resources

[0144] The behavioral analysis results are input into a mixed integer programming model. With the objective of achieving the optimal synergy between average customer waiting time and operational resource utilization, a dynamic service resource allocation strategy is solved. The specific process is as follows:

[0145] (1) Construction of the objective function: Construct the objective function of the mixed integer programming model:

[0146] ,

[0147] Where W is the average customer waiting time (minutes), and U is the operational resource utilization rate (0≤U≤1); the weighting coefficients α and β are determined by the analytic hierarchy process (AHP) and satisfy α+β=1.

[0148] In this embodiment, during peak business hours (9:00-11:30, 14:00-16:30), α=0.7 and β=0.3 are set to prioritize reducing customer waiting time; during off-peak business hours, α=0.3 and β=0.7 are set to prioritize improving resource utilization.

[0149] Selection criteria: Based on an operational survey of 10 bank branches, customer waiting time during peak hours was the primary reason for complaints (accounting for 72%), thus giving it higher weight; resource idle rate during off-peak hours was as high as 40%, thus prioritizing improving resource utilization; the weighting coefficients were determined by the AHP expert scoring method, and the consistency ratio CR=0.08<0.1, which meets the consistency requirements of AHP.

[0150] (2) Constraint settings: including counter number constraints (number of open counters ≤ total number of counters), self-service equipment load constraints (number of people served by a single device ≤ 3), employee continuous working hours ≤ 4 hours (in accordance with the provisions of the Labor Law on the right to rest of workers and the banking operation norms), employee rest time constraints, etc.

[0151] (3) Strategy Solution and Output: The branch and bound method is used to solve the mixed integer programming model (solution time ≤ 10s, meeting real-time requirements), and the following four types of dynamic allocation strategies for service resources are output:

[0152] Dynamic counter scheduling strategy: Adjust the number of open cash counters and non-cash counters in real time;

[0153] Self-service equipment load balancing strategy: Guide customers to self-service equipment with lower load to conduct business;

[0154] Personalized service trigger strategy: Push exclusive financial advisor appointment services to high-value clients;

[0155] Environmental temperature and humidity control strategy: The air conditioning temperature and fan speed are automatically adjusted according to the area’s passenger flow density (the temperature is reduced by 0.5℃ for every 0.1 people / ㎡ increase in passenger flow density).

[0156] Data privacy protection steps

[0157] This method also includes a full-process data privacy protection mechanism, as detailed below:

[0158] (1) Edge-end localized desensitization: The sensitive data collected is processed locally at the edge computing node of the sensing device. Only the processed feature data is transmitted to the back-end server, and the original sensitive data does not leave the edge node.

[0159] (2) Timed deletion of face data: After face feature extraction is completed in the edge computing module of the high-definition camera, the original face image data is automatically deleted within 100ms, and only the 128-dimensional face feature vector is retained for identity association. Selection criteria: The face feature extraction speed of the edge computing module is generally 30-50ms. 100ms can ensure that the original image is deleted immediately after feature extraction is completed, while leaving enough processing time for the system. If the time is less than 50ms, the image may be deleted before feature extraction is completed, affecting identity recognition. If it is longer than 1s, it will increase the risk of leakage of original face data.

[0160] (3) Trajectory data k-anonymization: The customer trajectory data is anonymized using the k-anonymization algorithm (where k=10), removing all identifying information that can directly identify an individual, such as the customer's ID number, mobile phone number, bank card number, and facial feature vector. Temporary identifiers used only for internal system identity association are encrypted using the SHA-256 irreversible hash algorithm. The encrypted identifiers cannot be reversed to restore the original identity information, ensuring that the processed behavioral analysis data cannot be associated with a specific natural person alone or in combination with other public information.

[0161] Selection criteria: The Personal Information Protection Law requires that anonymized data cannot be restored or identified as a specific natural person; k=10 ensures that each trajectory data is indistinguishable from at least 9 other trajectory data, with a re-identification risk of less than 0.1%; when k=5, the anonymization level is insufficient, and the re-identification risk exceeds 1%; when k=20, the usability of the data will decrease significantly and it cannot be used for behavioral analysis.

[0162] This embodiment provides a human-computer interaction behavior analysis system for financial scenarios, the system architecture of which is as follows: Figure 2 As shown, it includes the following eight modules:

[0163] Multi-source sensing data acquisition module: It interfaces with UWB positioning base stations, high-definition cameras, environmental sensors, self-service terminals and counter business systems, and is responsible for collecting multi-source heterogeneous sensing data and transmitting the data to the spatiotemporal alignment and tensor construction module.

[0164] Spatiotemporal alignment and tensor construction module: Executes the spatiotemporal alignment process described in step S2 to map the sensing data with different sampling frequencies and different spatial reference systems to a unified spatiotemporal reference, and constructs multimodal spatiotemporal tensor data.

[0165] Dynamic semantic segmentation and context embedding module: Based on the BIM model, the initial functional areas are divided, a directed passenger flow transfer map is constructed, the service status labels of each area are dynamically labeled using GAT, and the area status is embedded as a context feature into the subsequent model.

[0166] Multimodal feature fusion module: Extracts four types of features from customers: individual micro-behavior, human-computer interaction, interpersonal interaction and group context. Generates comprehensive interaction behavior vector (CIBV) through a multimodal Transformer network based on cross-attention.

[0167] Layered Behavior Analysis Module: Constructs a layered model that includes an immediate behavior classification layer, a business intent reasoning layer, and a behavior path prediction layer to achieve full-link analysis of customer behavior.

[0168] Abnormal Behavior Detection and Early Warning Module: Utilizes a pre-trained variational autoencoder to detect abnormal interactive behaviors, triggers tiered early warnings based on the magnitude of reconstruction errors, and links with intelligent building systems and public security systems.

[0169] Service resource optimization and allocation module: Construct a mixed integer programming model to solve the dynamic allocation strategy of service resources that achieves the optimal synergy between average customer waiting time and operational resource utilization.

[0170] Edge privacy protection module: Deployed on edge computing nodes at the sensing device, responsible for local desensitization of sensitive data, time-limited deletion of original face images, and k-anonymization of trajectory data.

[0171] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for analyzing human-computer interaction behavior in financial scenarios, characterized in that, Includes the following steps: S1. Collect multi-source heterogeneous sensing data through a smart building sensing network deployed in financial physical outlets; S2. Spatiotemporal alignment processing is performed on multi-source heterogeneous sensing data with different sampling frequencies and different spatial reference systems to construct multimodal spatiotemporal tensor data; S3. Based on the BIM spatial information model of intelligent buildings, the initial functional areas are divided, and the financial physical network is modeled as a directed passenger flow transfer map by combining real-time passenger flow data; the graph attention network GAT is used to update the features of the directed passenger flow transfer map, dynamically label the service status of each functional area in the current business period, and embed the regional service status as a context feature into the subsequent behavior analysis model. S4. Extract customer features, perform layer normalization on each feature and then concatenate them into a feature matrix. Use a multimodal Transformer fusion network based on cross-attention mechanism to map heterogeneous features to a unified representation space and generate a comprehensive interactive behavior vector CIBV. S5. Based on the time-series sequence of comprehensive interactive behavior vectors, a hierarchical behavior analysis model is constructed to realize real-time behavior classification, business intent reasoning and behavior path prediction in sequence. S6. Input the comprehensive interactive behavior vector into the pre-trained variational autoencoder model to calculate the reconstruction error. When the reconstruction error is greater than the preset threshold obtained based on the statistics of normal samples, it is judged as abnormal interactive behavior, and a graded early warning information is sent to the management decision system, triggering the linkage response of the intelligent building system. S7. Input the behavioral analysis results into the mixed integer programming model, and solve for the dynamic allocation strategy of service resources with the goal of achieving the optimal synergy between average customer waiting time and operational resource utilization.

2. The human-computer interaction behavior analysis method for financial scenarios according to claim 1, characterized in that: In step S2, the spatiotemporal alignment process includes: The financial physical network of branches is divided into a three-dimensional spatial grid of 1m×1m×3m, with a total of N grids; a unified timestamp is provided using BeiDou time synchronization. A three-dimensional spatial coordinate system with the network entrance as the origin. Based on this, all perceived data are mapped to a three-dimensional spatiotemporal tensor: ， Where: T is the aligned time series length, Ω is the spatial network of the financial scenario. A lattice set, where D is the number of dimensions of the perceptual modality; each tensor element This represents the perception value of the d-th perception modality at time t within the n-th spatial grid, thus constructing multimodal spatiotemporal cube data for financial scenarios.

3. The human-computer interaction behavior analysis method for financial scenarios according to claim 1, characterized in that: The construction and feature update of the directed passenger flow transfer map in step S3 specifically includes: Node features Includes the current passenger flow density of region i, the operating status of equipment within the region, and the current number of people in the queue; edge features The historical passenger flow transfer probability from region i to region j is given; the attention weight of each node is calculated using a graph attention network (GAT), the node features are aggregated and updated, and the service status labels of each functional area are dynamically labeled.

4. The human-computer interaction behavior analysis method for financial scenarios according to claim 1, characterized in that: The feature extraction and multimodal fusion described in step S4 specifically include: Extracted individual micro-behavioral characteristics Walking speed, dwell time, head orientation, and hand gestures; human-computer interaction features This includes click frequency, error rate, input duration, and page navigation path; interpersonal interaction characteristics. This includes interaction distance, interaction duration, and voice intensity; group contextual features. This includes regional passenger density, queue length, and group movement direction; Layer normalization was performed on the four types of heterogeneous features according to their feature dimensions, resulting in: ， The normalized features are concatenated into a feature matrix: ， A multimodal Transformer fusion network based on a cross-attention mechanism is adopted. Individual micro-behavioral features are used as the query matrix Q, and human-computer interaction features, interpersonal interaction features, and group context features are concatenated as the key matrix K and value matrix V. Cross-attention is calculated and residual connections are performed to generate a comprehensive interaction behavior vector CIBV, which satisfies: ， The cross-attention calculation function is as follows: ， To query the feature dimensions of matrix Q.

5. The human-computer interaction behavior analysis method for financial scenarios according to claim 1, characterized in that: The construction of the hierarchical behavior analysis model in step S5 specifically includes: 1) Real-time behavior classification layer: A three-layer temporal convolutional network is used to extract the temporal dependency features of the comprehensive interactive behavior vector sequence to identify the current behavior stage of the customer. The behavior stages include entering the branch, waiting in line, operating equipment, consulting business, and leaving the branch. 2) Business Intent Inference Layer: Construct a Bayesian network containing business intent nodes, behavioral feature nodes, historical behavior nodes, and reservation information nodes, based on the comprehensive interactive behavior vector sequence from time 1 to time t. Customer historical behavior characteristics and customer appointment information characteristics Calculate the posterior probability of each business intent: ， Where b is a business intent category variable, including deposit, withdrawal, transfer, loan consultation, financial consultation and loss reporting; 3) Behavioral path prediction layer: The attention-enhanced sequence-to-sequence model is adopted. The historical comprehensive interactive behavior vector sequence is used as input. The attention mechanism is applied to the functional area transfer sequence to predict the probability of the customer's subsequent functional area transfer and the terminal devices that may be reached.

6. The human-computer interaction behavior analysis method for financial scenarios according to claim 1, characterized in that: The abnormal interaction behavior detection and risk warning mentioned in step S6 specifically includes: The variational autoencoder model was pre-trained using a historical dataset containing only normal interaction behavior samples. The pre-training epochs were 100, and the batch size was 32. Calculate the reconstruction error of the input comprehensive interaction behavior vector: ， Where Enc is the encoder of the variational autoencoder, and Dec is the decoder of the variational autoencoder; preset threshold Take the 95th percentile of the reconstruction error for normal samples; when the reconstruction error When the error occurs, it is judged as abnormal interaction behavior; the warning is divided into three levels according to the size of the reconstruction error: the first level warning triggers on-site patrol by the security personnel of the site, the second level warning triggers temporary locking of self-service equipment, and the third level warning triggers linkage alarm of the public security system.

7. The human-computer interaction behavior analysis method for financial scenarios according to claim 1, characterized in that: In step S7, solving the dynamic configuration strategy for service resources specifically includes constructing an optimization objective function for a mixed-integer programming model: ， Where W represents the average customer wait time, and U represents the operational resource utilization rate; weighting coefficients and satisfy The dynamic configuration strategies for service resources obtained from the solution include: dynamic counter scheduling strategy, self-service equipment load balancing strategy, personalized service triggering strategy, and environmental temperature and humidity adjustment strategy.

8. The human-computer interaction behavior analysis method for financial scenarios according to claim 1, characterized in that: The method also includes data privacy protection steps: Sensitive data collected is locally anonymized at the edge computing nodes of the sensing devices; the original face image data is deleted within 100ms after facial features are extracted; customer trajectory data is anonymized using the k-anonymity algorithm to remove all identifying information, so that the processed behavioral analysis data cannot be associated with a specific natural person.

9. A human-computer interaction behavior analysis system for financial scenarios, applied to the human-computer interaction behavior analysis method for financial scenarios as described in claim 1, characterized in that, include: The multi-source sensing data acquisition module is used to collect multi-source heterogeneous sensing data through the intelligent building sensing network deployed in financial physical outlets; The spatiotemporal alignment and tensor construction module is used to perform spatiotemporal alignment processing on multi-source heterogeneous sensing data and construct multimodal spatiotemporal tensor data. The dynamic semantic segmentation and context embedding module is used to perform dynamic semantic segmentation on financial physical outlets, divide functional areas and dynamically label service status, and embed the area status as context features into the behavior analysis model. The multimodal feature fusion module is used to extract multi-dimensional interactive behavior features of customers. It uses a multimodal fusion network to map heterogeneous features to a unified representation space and generate a comprehensive interactive behavior vector. The hierarchical behavior analysis module is used to build hierarchical behavior analysis models to achieve real-time customer behavior classification, business intent reasoning, and behavior path prediction. The abnormal behavior detection and early warning module is used to detect abnormal interactive behaviors and trigger tiered early warnings and intelligent building system linkage responses. The service resource optimization and configuration module is used to solve the dynamic configuration strategy of service resources based on the results of behavior analysis. The edge privacy protection module is used to perform localized de-identification processing on the collected sensitive data.

Images