A bank branch employee behavior prediction method based on multi-stream network

Abstract

The present invention discloses a method for predicting employee behavior in bank branches based on a multi-stream network, which relates to the field of computer vision behavior analysis. By introducing three characteristic streams, namely, time stream, space stream and event stream, the system can comprehensively consider the temporal changes, spatial positions and specific situations of abnormal behaviors of employees in bank branches, thereby comprehensively capturing the dynamic behavior characteristics of employees. Compared with the traditional single-stream network, this solution can more accurately analyze the behavioral characteristics of employees in different scenarios, identify potential abnormal behaviors, and improve the accuracy of behavior recognition.

Description

Technical Field

[0001] The present invention relates to the field of computer vision behavior analysis, and in particular to a bank branch employee behavior prediction method based on a multi-stream network. Background Art

[0002] With the continuous development of banking business, the operational efficiency of bank branches and the management of employee behavior have become increasingly important. In order to improve service quality and ensure bank security, bank managers need to monitor and evaluate employee behavior in real time. Most existing employee behavior monitoring technologies rely on traditional monitoring equipment and manual review methods. By installing video surveillance equipment, banks can record employees' daily operations and capture employee behavior patterns, such as customer reception, counter operations, equipment use, etc. In these systems, the detection and analysis of employee behavior mainly relies on manual intervention. Manual analysts watch surveillance videos to identify whether employees follow the prescribed procedures and whether there are any improper behaviors. Manual review is inefficient and easily interfered by subjective factors.

[0003] In recent years, with the rapid development of artificial intelligence and computer vision technology, more and more banks have begun to try to use automated means to analyze employee behavior. The basic components of traditional video surveillance systems are cameras, video storage and playback systems. In some high-end application scenarios, banks have begun to introduce video analysis technology, especially algorithms based on image processing and deep learning. These algorithms automatically identify employees' basic behaviors through frame-to-frame analysis of video images and behavioral pattern recognition. Such systems can effectively improve the detection accuracy of abnormal behaviors and reduce reliance on manual analysis through real-time analysis of employee behavior.

[0004] However, although the existing employee behavior recognition technology based on computer vision and deep learning has achieved initial results in many scenarios, there are still many problems in its actual application in bank branches. Most of the existing technologies focus on static behavior or basic dynamic behavior analysis, such as routine teller operations, customer reception, equipment operation, etc., and do not fully consider the special environment and potential complex situations of bank branches. For example, employees' abnormal behavior is often more hidden and the boundary between it and regular behavior is blurred. Traditional image analysis methods have difficulty in effectively distinguishing the two. In addition, existing behavior classification systems usually rely on manual annotation and traditional supervised learning methods, which have weak processing capabilities on high-dimensional behavior data and cannot efficiently capture complex and nonlinear behavior patterns.

[0005] Although computer vision and deep learning technologies have made some progress in employee behavior monitoring, existing technical solutions still have several significant deficiencies. First, existing technologies mainly focus on the analysis of a single feature stream, usually focusing only on one aspect of behavioral characteristics, such as static or dynamic information in an image, or simple detection of abnormal behavior. This single-stream network approach fails to fully consider the diversity and complexity of behavior, and cannot fully capture the comprehensive behavioral characteristics of employees in various complex situations in bank branches. In the working environment of bank branches, the performance of employee behavior not only depends on static scenes or actions, but is also affected by situational changes and behavioral history. Simple single-stream networks are insufficient in processing such multi-dimensional and multi-time series behavioral information, and are prone to overlooking certain potential key behaviors. Summary of the invention

[0006] In view of the above-mentioned defects of the prior art, the present application provides a bank branch employee behavior prediction method based on a multi-stream network, aiming to solve the shortcomings of the prior art in bank branch employee behavior monitoring.

[0007] Among them, a bank branch employee behavior prediction method based on a multi-stream network includes the following steps:

[0008] S1. Collect employee-related videos through the network camera equipment, extract image frames based on the video captured by the camera equipment, calculate the optical flow information between adjacent frames, and analyze the image frames, and add corresponding status tags according to the employee behavior appearing in the image frames, wherein the status tags include regular behavior tags and abnormal behavior tags;

[0009] S2. construct a three-stream network including a time stream, a space stream and an event stream, and model the dynamic features, static features and abnormal behavior features in the video through the three-stream network, wherein the three-stream network extracts the dynamic features, static features and abnormal behavior features through the time stream, the space stream and the event stream respectively;

[0010] S3. Dynamically generate the weight of each feature stream through the attention mechanism, automatically adjust the weight according to the rarity of the behavior feature in the current frame, and fuse the three stream features according to the adaptive weights based on the generated weights to obtain the final comprehensive behavior feature. The feature streams include time stream, space stream and event stream.

[0011] S4. Based on the comprehensive behavioral characteristics, the model learns and adjusts through loss functions and training optimization methods to output the behavioral classification and scoring of bank branch employees.

[0012] Furthermore, the step S2 specifically includes the following sub-steps:

[0013] S201. Capturing static behavior features of employees through spatial stream CNN according to the extracted image frames;

[0014] S202. Based on the calculated optical flow information, the dynamic behavior characteristics of the employees are captured through the time flow CNN;

[0015] S203. According to the image frames containing abnormal behavior labels, the abnormal behavior characteristics of the employees are captured through the event stream EventNet.

[0016] Furthermore, the static behavior features are used to analyze the movement trend and frequency of employee behavior; the dynamic behavior features are used to identify the posture and position of employees; and the abnormal behavior features are used to identify potential risky behaviors.

[0017] Furthermore, the step S3 specifically includes the following sub-steps:

[0018] S301. Through the attention mechanism, the model analyzes the specific behavior content of the bank employee in the current frame image, and generates weights of different feature streams according to the rarity of regular or abnormal behavior features;

[0019] S302. By gradually weighting and aggregating the features of the time stream, space stream and event stream, the dynamic features, static features and abnormal behavior features are fused.

[0020] Furthermore, in step S301, the weights of different feature streams generated according to the rarity of the regular or abnormal behavior features are specifically expressed as:

[0021] ;

[0022] Among them, the Representing feature flow The adaptive weight of Representing feature flow The rarity parameter, Representing feature flow The scene importance parameter, Determined according to the scene content, the represents the rarity parameter of the feature flow set, Represents the scene importance parameter of the feature flow set, Represents the set of all feature flows, namely the set of time flow, space flow and event flow.

[0023] Furthermore, in step S302, the features of the time stream, space stream and event stream are gradually weighted and aggregated as follows:

[0024] ;

[0025] Among them, the Represents the final comprehensive behavioral characteristics, Represents the features of the extracted time stream, space stream and event stream, Representing feature flow The adaptive weight of .

[0026] Furthermore, the step S4 specifically includes the following sub-steps:

[0027] S401. Construct a classification model by combining convolutional neural network and multi-layer perceptron to classify comprehensive behavioral features;

[0028] S402. Optimizing the classification accuracy of the model using a dual loss function;

[0029] S403. Extracting comprehensive behavioral features through the classification model, generating employee behavior categories through the classification layer, the behavior categories include regular behavior and abnormal behavior;

[0030] S404. Based on the classification results, analyze the employee's performance in the behavior category through the defined scoring criteria and output a comprehensive scoring result.

[0031] Furthermore, in step S402, the dual loss function includes a cross entropy loss and a category rarity penalty, and the specific loss function is expressed as:

[0032] ;

[0033] Among them, the represents the dual loss function, Represents the total number of category labels. Represents the index of the category label. Indicates the actual behavior category label, which is set according to the work content of bank branch employees. Represents the probability of predicting the behavior category, which is used to measure the classification accuracy of the model. represents the parameter used to adjust the rareness penalty, Indicates the frequency of occurrence of the behavior category.

[0034] Furthermore, in step S404, the specific calculation process of the comprehensive evaluation result is as follows:

[0035] ;

[0036] Among them, the Indicates the comprehensive behavior score of the employee, that is, the comprehensive evaluation result. Represents the total number of category labels. Represents the index of the category label. represents the probability of predicting the behavior category, Indicates the scoring criteria for each behavior category. represents the frequency of occurrence of the behavior category. Represents the smoothing factor.

[0037] Furthermore, in step S1, calculating the optical flow information between adjacent frames is specifically expressed as:

[0038] ;

[0039] Among them, the Indicates the motion energy at the current location, used to analyze the intensity and frequency of employee movements, and helps distinguish between normal and abnormal behaviors. represents an image frame collected from a camera device, Respectively axis, axis and time axis, the and Respectively represent the image frames in axis, Gradients on the time and time axes, that is, small changes in the posture and position of employees in the branch environment.

[0040] The beneficial effects of the invention are:

[0041] By introducing three characteristic streams, namely time stream, space stream and event stream, the system can comprehensively consider the temporal changes of employee behaviors in bank branches, spatial locations and specific situations of abnormal behaviors, thereby comprehensively capturing the dynamic behavioral characteristics of employees. Compared with the traditional single-stream network, this solution can more accurately analyze the behavioral characteristics of employees in different scenarios, identify potential abnormal behaviors, and improve the accuracy of behavior recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A method structure diagram of a bank branch employee behavior prediction method based on a multi-stream network provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings, but the protection scope of the present invention is not limited to the following.

[0044] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention, that is, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0045] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative work are within the scope of protection of the present invention. It should be noted that relational terms such as the terms "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0046] Moreover, the terms "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a list of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or apparatus. In the absence of more restrictions, an element defined by the phrase "comprises a ..." does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0047] The features and performance of the present invention are further described in detail below in conjunction with the embodiments.

[0048] like Figure 1 As shown, a bank branch employee behavior prediction method based on a multi-stream network includes the following steps:

[0049] S1. Collect employee-related videos through the network camera equipment, extract image frames based on the video captured by the camera equipment, calculate the optical flow information between adjacent frames, and analyze the image frames, and add corresponding status tags according to the employee behavior appearing in the image frames, wherein the status tags include regular behavior tags and abnormal behavior tags;

[0050] S2. construct a three-stream network including a time stream, a space stream and an event stream, and model the dynamic features, static features and abnormal behavior features in the video through the three-stream network, wherein the three-stream network extracts the dynamic features, static features and abnormal behavior features through the time stream, the space stream and the event stream respectively;

[0051] S3. Dynamically generate the weight of each feature stream through the attention mechanism, automatically adjust the weight according to the rarity of the behavior feature in the current frame, and fuse the three stream features according to the adaptive weights based on the generated weights to obtain the final comprehensive behavior feature. The feature streams include time stream, space stream and event stream.

[0052] S4. Based on the comprehensive behavioral characteristics, the model learns and adjusts through loss functions and training optimization methods to output the behavioral classification and scoring of bank branch employees.

[0053] Specifically, prime camera equipment covers key areas, including counters, business processing areas, customer reception areas, etc., to ensure that all relevant employee behaviors can be captured; and optical flow data is used to assist in identifying the dynamic behavior characteristics of employees; the conventional behavior tags include customer service and file processing, and the abnormal behavior tags include entering restricted areas and staying still for a long time. The above tags are defined according to actual needs.

[0054] Furthermore, the step S2 specifically includes the following sub-steps:

[0055] S201. Capturing static behavior features of employees through spatial stream CNN according to the extracted image frames;

[0056] S202. Based on the calculated optical flow information, the dynamic behavior characteristics of the employees are captured through the time flow CNN;

[0057] S203. According to the image frames containing abnormal behavior labels, the abnormal behavior characteristics of the employees are captured through the event stream EventNet.

[0058] Specifically, the three-stream network is used to model the dynamic features, static features and abnormal behavior features in the video through the three parts of time stream, space stream and event stream, respectively, to capture the behavioral information of employees; wherein, the three-stream network uses the spatial stream CNN model to capture the static features of employees, such as posture, starting position of action, etc., based on the extracted image frames, and captures static behaviors such as customer service and counter work; and uses the time stream CNN model to capture the dynamic behavior characteristics of employees through optical flow data. The time stream can identify the details of actions such as walking and object delivery, and process motion behavior analysis; and based on image frames containing abnormal labels, the event stream EventNet model is used to capture abnormal behavior characteristics. The EventNet model is used to identify low-frequency or rare abnormal behaviors (such as unauthorized entry and abnormal stay) and strengthen the detection effect of specific risk behaviors in bank branches.

[0059] Furthermore, the spatial stream CNN captures the static behavior characteristics of employees as follows:

[0060] ;

[0061] Among them, the represents the spatial flow characteristics, Represents the total number of spatial stream CNN layers. Represents the index of the number of spatial stream CNN layers. represents the spatial features extracted by the convolutional neural network, specifically representing a certain pattern of static behavior in the image frame. Represents the weight of the spatial stream feature, which is used to strengthen the important posture features. Represents an image frame captured from a camera device.

[0062] Furthermore, the capture of the dynamic behavior characteristics of employees by time stream CNN is specifically expressed as follows:

[0063] ;

[0064] ;

[0065] Among them, the represents the dynamic features of the nth frame, represents the time stream feature weight of the nth frame, represents the fused time stream features, i.e., the dynamic behavior of employees. A sequence of video frames representing employee behavior, represents the temporal features extracted by the convolutional neural network. represents the total length of the frame sequence extracted by the time stream CNN, Indicates the index of the video frame in the sequence.

[0066] Furthermore, the event stream EventNet captures the abnormal behavior characteristics of employees as follows:

[0067] ;

[0068] Among them, the Represents event stream features, i.e., abnormal patterns in employee behavior, represents the number of layers in the event stream network used to extract abnormal behavior features. Represents the index of a specific layer in the event stream network. represents the weight of the event stream feature, Represents the abnormal behavior features extracted by the EventNet network. Represents image frames containing abnormal behavior labels.

[0069] Furthermore, the static behavior features are used to analyze the movement trend and frequency of employee behavior; the dynamic behavior features are used to identify the posture and position of employees; and the abnormal behavior features are used to identify potential risky behaviors.

[0070] Furthermore, the step S3 specifically includes the following sub-steps:

[0071] S301. Through the attention mechanism, the model analyzes the specific behavior content of the bank employee in the current frame image, and generates weights of different feature streams according to the rarity of regular or abnormal behavior features;

[0072] S302. By gradually weighting and aggregating the features of the time stream, space stream and event stream, the dynamic features, static features and abnormal behavior features are fused.

[0073] Specifically, the model dynamically calculates the weights of spatial flow, temporal flow and event flow through the attention mechanism, and automatically adjusts the weight distribution of each feature based on the rarity and correlation of the behavioral features in the current frame, thereby improving the model's response sensitivity under rare and abnormal behaviors; and based on the generated weights, the three-stream features are fused according to adaptive weights to obtain the final comprehensive behavioral feature representation. This comprehensive feature effectively integrates static, dynamic and abnormal information, provides a unified feature input for subsequent behavior classification, and improves classification accuracy.

[0074] Furthermore, in step S301, the weights of different feature streams generated according to the rarity of the regular or abnormal behavior features are specifically expressed as:

[0075] ;

[0076] Among them, the Representing feature flow The adaptive weight of Representing feature flow The rarity parameter, Representing feature flow The scene importance parameter, Determined according to the scene content, the represents the rarity parameter of the feature flow set, Represents the scene importance parameter of the feature flow set, Represents the set of all feature flows, namely the set of time flow, space flow and event flow.

[0077] Furthermore, in step S302, the features of the time stream, space stream and event stream are gradually weighted and aggregated as follows:

[0078] ;

[0079] Among them, the Represents the final comprehensive behavioral characteristics, Represents the features of the extracted time stream, space stream and event stream, Representing feature flow The adaptive weight of .

[0080] Furthermore, the step S4 specifically includes the following sub-steps:

[0081] S401. Construct a classification model by combining convolutional neural network and multi-layer perceptron to classify comprehensive behavioral features;

[0082] S402. Optimizing the classification accuracy of the model using a dual loss function;

[0083] S403. Extracting comprehensive behavioral features through the classification model, generating employee behavior categories through the classification layer, the behavior categories include regular behavior and abnormal behavior;

[0084] S404. Based on the classification results, analyze the employee's performance in the behavior category through the defined scoring criteria and output a comprehensive scoring result.

[0085] Specifically, when the model identifies routine behaviors (such as counter service and document processing), it classifies them as normal working behaviors based on the information fused from static and dynamic feature flows; for abnormal behaviors such as unauthorized entry into areas or customer interactions, the model classifies these behaviors as abnormalities, helping branch managers to identify potential problems in a timely manner. In addition, the scoring criteria may include work efficiency (such as the speed of document delivery), service quality (such as the fluency of customer interactions), and security (such as the frequency of entering restricted areas), which are calculated by the quality and quantity of behavioral features.

[0086] Furthermore, for the dual loss function, it is preferred that in bank branches, the tolerance for misclassification of routine behaviors is lower, so a higher penalty value is assigned to the classification error of this behavior type, so that the model can give priority to accurately classifying normal working behaviors; and then dynamically adjust the weights of different scoring indicators through the attention mechanism to further improve the accuracy of the model's scoring of key behaviors (such as customer reception).

[0087] Furthermore, for the behavior categories output by the classification model, the model directly labels routine work behaviors as "normal"; for potential abnormal behaviors, detailed abnormal category labels are generated, such as "unauthorized entry", "abnormal stay", etc.

[0088] Furthermore, in step S402, the dual loss function includes a cross entropy loss and a category rarity penalty, and the specific loss function is expressed as:

[0089] ;

[0090] Among them, the represents the dual loss function, Represents the total number of category labels. Represents the index of the category label. Indicates the actual behavior category label, which is set according to the work content of bank branch employees. Represents the probability of predicting the behavior category, which is used to measure the classification accuracy of the model. represents the parameter used to adjust the rareness penalty, Indicates the frequency of occurrence of the behavior category.

[0091] Furthermore, in step S404, the specific calculation process of the comprehensive evaluation result is as follows:

[0092] ;

[0093] Among them, the Indicates the comprehensive behavior score of the employee, that is, the comprehensive evaluation result. Represents the total number of category labels. Represents the index of the category label. represents the probability of predicting the behavior category, Indicates the scoring criteria for each behavior category. Indicates the frequency of occurrence of the behavior category. Represents the smoothing factor.

[0094] Furthermore, in step S1, calculating the optical flow information between adjacent frames is specifically expressed as:

[0095] ;

[0096] Among them, the Indicates the motion energy at the current location, used to analyze the intensity and frequency of employee movements, and helps distinguish between normal and abnormal behaviors. represents an image frame collected from a camera device, Respectively axis, axis and time axis, the and Respectively represent the image frames in axis, Gradients on the time and time axes, that is, small changes in the posture and position of employees in the branch environment.

[0097] Further, as a preferred implementation of the above embodiment, the three-stream network mentioned in step S2 and the corresponding extracted spatial stream features, temporal stream features and abnormal behavior features can perform correlation modeling on the above features, that is, it also includes step S204: behavioral correlation modeling of the features of the temporal stream, spatial stream and event stream, and establishing a correlation model between the features through a multi-level deep learning network. For example, a graph convolutional network GCN is selected to construct a spatiotemporal behavior graph between the temporal stream features, spatial stream features and event stream features, and the dependency between the behavior streams is learned through the network; for the spatiotemporal behavior graph, each graph node represents a behavior feature, and the edges between the nodes represent the dependency between the nodes. For the temporal stream features, the edges between the nodes represent the temporal dependency; for the spatial stream and event stream, the edges between the nodes represent the mutual influence of space or events; the correlation of the behavior features is calculated through the graph neural network to capture the mutual influence and long-term dependency between the feature streams. For example, some spatial streams and temporal streams (such as the duration of an action) may have a strong correlation, and abnormal behaviors (such as employees abnormally staying in a certain area) may also have a significant correlation with spatial features. Further, the steps S301-302 are changed to dynamically generate weight coefficients through the attention mechanism, and dynamically adjust the weights of the time stream, space stream and event stream according to the strength of the correlation of each behavior feature. The features of the time stream, space stream and event stream will be weighted according to the correlation and combined into a comprehensive feature representation. Specifically, for feature streams with strong correlation, the model automatically enhances its weight; for feature streams with weak correlation, its weight will be reduced. Exemplarily, when detecting abnormal behavior, there may be a strong correlation between the spatial stream and the time stream (such as employees staying in a certain area for too long may indicate abnormality). At this time, the model will automatically increase the weights of the spatial stream and the time stream to more accurately identify abnormal behavior. For relatively rare behaviors (such as some special abnormal behaviors), the model dynamically adjusts the weights to increase the weight of the event stream (abnormal behavior), thereby enhancing the sensitivity of abnormal behavior detection.

[0098] Furthermore, for the above embodiment, its attention mechanism uses a graph attention network (GAT) to dynamically adjust the influence of each node.

[0099] Furthermore, as a preferred implementation of the above embodiment, a bank branch employee behavior prediction method based on a multi-stream network is proposed, comprising:

[0100] The data acquisition module is used to collect employee-related videos through the network point camera equipment, extract image frames based on the videos collected by the camera equipment, calculate the optical flow information between adjacent frames, and analyze the image frames, and add corresponding status labels according to the employee behaviors appearing in the image frames, and the status labels include regular behavior labels and abnormal behavior labels;

[0101] A three-stream network construction module is used to construct a three-stream network including a time stream, a space stream and an event stream, and to model dynamic features, static features and abnormal behavior features in the video through the three-stream network. The three-stream network extracts features of the dynamic features, static features and abnormal behavior features through the time stream, the space stream and the event stream respectively;

[0102] A feature fusion module is used to dynamically generate the weight of each feature stream through an attention mechanism, automatically adjust the weight according to the rarity of the behavior feature in the current frame, and fuse the three-stream features according to the adaptive weights based on the generated weights to obtain the final comprehensive behavior feature. The feature streams include time stream, space stream and event stream;

[0103] The model analysis module is used to learn and adjust the model based on comprehensive behavioral characteristics through loss functions and training optimization methods, and output the behavioral classification and scoring of bank branch employees.

[0104] Furthermore, the three-stream network construction module specifically includes:

[0105] A static behavior feature capturing unit, used to capture the static behavior features of employees through spatial stream CNN according to the extracted image frames;

[0106] The dynamic behavior feature capture unit is used to capture the dynamic behavior features of employees through the time stream CNN based on the calculated optical flow information;

[0107] The abnormal behavior feature capture unit is used to capture the abnormal behavior features of employees through the event stream EventNet based on the image frames containing abnormal behavior labels.

[0108] Furthermore, the static behavior features are used to analyze the movement trend and frequency of employee behavior; the dynamic behavior features are used to identify the posture and position of employees; and the abnormal behavior features are used to identify potential risky behaviors.

[0109] Furthermore, the feature fusion module specifically includes:

[0110] The weight adaptive adjustment unit is used to analyze the specific behavior content of the bank employees in the current frame image through the attention mechanism and model, and generate weights of different feature streams according to the rarity of regular or abnormal behavior features;

[0111] The feature fusion unit is used to perform feature fusion of dynamic features, static features and abnormal behavior features by gradually weighting and aggregating the features of time stream, space stream and event stream.

[0112] Furthermore, in the weight adaptive adjustment unit, the weights of different feature streams generated according to the rarity of normal or abnormal behavior features are specifically expressed as:

[0113] ;

[0114] Among them, the Representing feature flow The adaptive weight of Representing feature flow The rarity parameter, Representing feature flow The scene importance parameter, Determined according to the scene content, the represents the rarity parameter of the feature flow set, Represents the scene importance parameter of the feature flow set, Represents the set of all feature flows, namely the set of time flow, space flow and event flow.

[0115] Furthermore, in the feature fusion unit, the features of the time stream, space stream and event stream are gradually weighted and aggregated as follows:

[0116] ;

[0117] Among them, the Represents the final comprehensive behavioral characteristics, Represents the features of the extracted time stream, space stream and event stream, Representing feature flow The adaptive weight of .

[0118] Furthermore, the model analysis module specifically includes:

[0119] A classification model building unit, used to build a classification model by combining a convolutional neural network and a multi-layer perceptron to classify comprehensive behavioral features;

[0120] A loss function definition unit, used to optimize the classification accuracy of the model using a dual loss function;

[0121] A classification model output unit, used to extract comprehensive behavior features through the classification model, and generate employee behavior categories through the classification layer, wherein the behavior categories include regular behavior and abnormal behavior;

[0122] The comprehensive scoring unit is used to analyze the performance of employees in the behavior categories based on the classification results and the defined scoring criteria, and output the comprehensive scoring results.

[0123] Furthermore, in the loss function definition unit, the dual loss function includes cross entropy loss and category rarity penalty, and the specific loss function is expressed as:

[0124] ;

[0125] Among them, the represents the dual loss function, Represents the total number of category labels. Represents the index of the category label. Indicates the actual behavior category label, which is set according to the work content of bank branch employees. Represents the probability of predicting the behavior category, which is used to measure the classification accuracy of the model. represents the parameter used to adjust the rareness penalty, Indicates the frequency of occurrence of the behavior category.

[0126] Furthermore, in the comprehensive scoring unit, the specific calculation process of the comprehensive scoring result is as follows:

[0127] ;

[0128] Among them, the Indicates the comprehensive behavior score of the employee, that is, the comprehensive evaluation result. Represents the total number of category labels. Represents the index of the category label. represents the probability of predicting the behavior category, Indicates the scoring criteria for each behavior category. represents the frequency of occurrence of the behavior category. Represents the smoothing factor.

[0129] Furthermore, in the data acquisition module, calculating the optical flow information between adjacent frames is specifically expressed as:

[0130] ;

[0131] Among them, the Indicates the motion energy at the current location, used to analyze the intensity and frequency of employee movements, and helps distinguish between normal and abnormal behaviors. represents an image frame collected from a camera device, Respectively axis, axis and time axis, the and Respectively represent the image frames in axis, Gradients on the time and time axes, that is, small changes in the posture and position of employees in the branch environment.

[0132] The above implementation plan comprehensively extracts and analyzes the static, dynamic and abnormal behavior characteristics of bank branch employees by constructing a multi-stream network model. The adaptive weight mechanism effectively integrates multi-stream characteristics, and the classification and scoring system outputs clear behavior analysis and improvement suggestions, which is suitable for daily behavior management and risk control of bank branches.

[0133] The above is only a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not deviate from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A bank branch employee behavior prediction method based on a multi-stream network, characterized in that: The following steps are involved: S1. Collect employee-related videos through the network camera equipment, extract image frames based on the video captured by the camera equipment, calculate the optical flow information between adjacent frames, and analyze the image frames, and add corresponding status tags according to the employee behavior appearing in the image frames, wherein the status tags include regular behavior tags and abnormal behavior tags; S2. construct a three-stream network including a time stream, a space stream and an event stream, and model the dynamic features, static features and abnormal behavior features in the video through the three-stream network, wherein the three-stream network extracts the dynamic features, static features and abnormal behavior features through the time stream, the space stream and the event stream respectively; S3. Dynamically generate the weight of each feature stream through the attention mechanism, automatically adjust the weight according to the rarity of the behavior feature in the current frame, and fuse the three stream features according to the adaptive weights based on the generated weights to obtain the final comprehensive behavior feature. The feature streams include time stream, space stream and event stream. S4. Based on the comprehensive behavioral characteristics, the model learns and adjusts through loss functions and training optimization methods, and outputs the behavioral classification and scoring of bank branch employees; Among them, the automatic adjustment of weights according to the rarity of the behavior features in the current frame is specifically expressed as: ； Among them, the Representing feature flow The adaptive weight of Representing feature flow The rarity parameter, Representing feature flow The scene importance parameter, Determined according to the scene content, the represents the rarity parameter of the feature flow set, Represents the scene importance parameter of the feature flow set, Represents the set of all feature flows, namely the set of time flow, space flow and event flow; The specific calculation process of the comprehensive scoring results is as follows: ； Among them, the Indicates the comprehensive behavior score of the employee, that is, the comprehensive evaluation result. Represents the total number of category labels. Represents the index of the category label. represents the probability of predicting the behavior category, Indicates the scoring criteria for each behavior category. Indicates the frequency of occurrence of the behavior category. represents the smoothing factor; The dual loss function includes cross entropy loss and category rarity penalty. The specific loss function is expressed as: ； Among them, the represents the dual loss function, Represents the total number of category labels. Represents the index of the category label. Indicates the actual behavior category label, which is set according to the work content of bank branch employees. Represents the probability of predicting the behavior category, which is used to measure the classification accuracy of the model. represents the parameter used to adjust the rareness penalty, Indicates the frequency of occurrence of the behavior category.

2. A bank branch employee behavior prediction method based on a multi-stream network as claimed in claim 1, characterized in that: The step S2 specifically includes the following sub-steps: S201. According to the extracted image frames, the static behavior characteristics of the employees are captured by using a spatial stream CNN convolutional neural network; S202. Based on the calculated optical flow information, the dynamic behavior characteristics of the employees are captured through the time flow CNN convolutional neural network; S203. According to the image frames containing abnormal behavior labels, the abnormal behavior characteristics of the employees are captured through the event stream EventNet asynchronous neural network.

3. A bank branch employee behavior prediction method based on a multi-stream network as claimed in claim 2, characterized in that: The static behavior features are used to analyze the movement trend and frequency of employee behavior; the dynamic behavior features are used to identify the posture and position of employees; and the abnormal behavior features are used to identify potential risky behaviors.

4. A bank branch employee behavior prediction method based on a multi-stream network as claimed in claim 1, characterized in that: The step S3 specifically includes the following sub-steps: S301. Through the attention mechanism, the model analyzes the specific behavior content of the bank employee in the current frame image, and generates weights of different feature streams according to the rarity of regular or abnormal behavior features; S302. By gradually weighting and aggregating the features of the time stream, space stream and event stream, the dynamic features, static features and abnormal behavior features are fused.

5. A bank branch employee behavior prediction method based on a multi-stream network as claimed in claim 4, characterized in that: In step S302, the features of the time stream, space stream and event stream are gradually weighted and aggregated as follows: ； Among them, the Represents the final comprehensive behavioral characteristics, Represents the features of the extracted time stream, space stream and event stream, Representing feature flow The adaptive weight of .

6. A bank branch employee behavior prediction method based on a multi-stream network as claimed in claim 1, characterized in that: The step S4 specifically includes the following sub-steps: S401. Construct a classification model by combining convolutional neural network and multi-layer perceptron to classify comprehensive behavioral features; S402. Optimizing the classification accuracy of the model using a dual loss function; S403. Extracting comprehensive behavioral features through the classification model, generating employee behavior categories through the classification layer, the behavior categories include regular behavior and abnormal behavior; S404. Based on the classification results, analyze the employee's performance in the behavior category through the defined scoring criteria and output a comprehensive scoring result.

7. A bank branch employee behavior prediction method based on a multi-stream network as claimed in claim 1, characterized in that: In step S1, calculating the optical flow information between adjacent frames is specifically expressed as: ； Among them, the Indicates the motion energy at the current location, used to analyze the intensity and frequency of employee movements, and helps distinguish between normal and abnormal behaviors. represents an image frame collected from a camera device, Respectively axis, axis and time axis, the and Respectively represent the image frames in axis, Gradients on the time and time axes, that is, small changes in the posture and position of employees in the branch environment.

Images