Store operator performance evaluation system based on deep learning

Abstract

The present application relates to the technical field of deep learning, in particular to a store operation personnel performance evaluation system based on deep learning. A video acquisition front end extracts a skeleton node sequence to construct a space-time graph, and a graph convolution network extracts spatial behavior features. A transaction flow receiving interface extracts a transaction occurrence time to construct a transaction event vector, and a time series transformer extracts transaction context features. A cross-attention alignment module takes the timestamp of the transaction event vector as a query key, searches in the time dimension of the spatial behavior features, calculates a cross-modal similarity matrix, and outputs a joint feature vector to a fully connected layer to generate a performance score. In back propagation, the space-time graph edge weights of the corresponding time steps are updated according to the transaction event gradient. The present application eliminates the interference of redundant action features without transaction association, and establishes the causal correspondence between physical operation behavior and transaction results.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and more specifically to a deep learning-based performance evaluation system for store operations personnel. Background Technology

[0002] Currently, performance evaluation for store operations staff typically combines video surveillance analysis with transaction data statistics. In practice, the conventional approach involves capturing store surveillance video via cameras, extracting skeletal key points of personnel using human pose estimation algorithms, and then inputting these key points from consecutive frames into a recurrent neural network to extract temporal action features. Simultaneously, the conventional approach obtains transaction data from the POS system, extracting transaction time and amount. In the feature fusion stage, the conventional approach usually performs a simple concatenation of the video feature vector and the transaction data vector along the time dimension, or aligns them temporally within a fixed time window before inputting them into a fully connected layer, outputting the final performance score. This approach treats video features and transaction features as independent parallel input channels, relying on the fully connected layer to learn the mapping relationship between them.

[0003] The aforementioned conventional approach suffers from a core technical problem in practical applications: it cannot establish a specific causal relationship between the physical operational behavior of staff and the results of digital transactions. Because the concatenation processing within a fixed time window lacks a targeted retrieval mechanism for the occurrence of discrete transaction events, the temporal action features extracted by the recurrent neural network contain a large number of redundant actions unrelated to transactions. When receiving simple concatenated vectors, the fully connected layer cannot distinguish which action features actually led to the corresponding transaction, causing the model to be affected by noise from non-business behavior features during training. This results in the final performance score deviating from the staff's true business contribution. Summary of the Invention

[0004] The purpose of this invention is to provide a deep learning-based performance evaluation system for store operations personnel, which can solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A deep learning-based performance evaluation system for store operations personnel includes a video capture front-end and a transaction log receiving interface. The video capture front-end extracts the skeleton node sequence of operations personnel and constructs a spatiotemporal graph, where the skeleton nodes are vertices and the physical distance and relative motion speed between nodes are edge weights. The spatiotemporal graph is input into a graph convolutional network to extract spatial behavioral features. The transaction log receiving interface extracts the transaction occurrence time, product category, and processing time to construct a transaction event vector. This transaction event vector is input into a temporal Transformer to extract transaction context features. The system also includes a cross-attention alignment module, which uses the timestamp of the transaction event vector as the query key to perform a search on the temporal dimension of the spatial behavioral features, calculates a cross-modal similarity matrix, and outputs a joint feature vector that fuses operational actions and transaction results. This joint feature vector is input into a fully connected layer to output a performance score.

[0007] Preferably, the video acquisition front end extracts the coordinates of key skeletal points of the operators using the OpenPose algorithm to form a skeleton node sequence, and uses forward kinematics interpolation to complete occluded key points; when constructing the spatiotemporal graph, skeleton nodes at the same time are connected by physical connection edges, and the same skeleton nodes at adjacent times are connected by temporal connection edges; when calculating the weight of the physical connection edges, the Euclidean distance between adjacent skeleton nodes is extracted and divided by a preset standard skeleton length ratio as the spatial weight; when calculating the weight of the temporal connection edges, the difference between the displacement vector magnitude of the same skeleton node at adjacent times and a preset velocity threshold is extracted as the temporal weight, and the spatial weight and temporal weight are added together to obtain the edge weight of the spatiotemporal graph.

[0008] Preferably, the graph convolutional network comprises three graph convolutional layers. The first graph convolutional layer extracts local limb features of adjacent skeleton nodes, the second graph convolutional layer extracts cross-limb trunk coordination features, and the third graph convolutional layer extracts global body pose features. Each graph convolutional layer is followed by a batch normalization layer and a modified linear unit activation function. The output of the third graph convolutional layer is processed by a flattening operation and a one-dimensional convolutional layer with a kernel size of 3, recombining the spatial dimension and channel dimension to obtain a spatial behavior feature matrix. The row dimension of the spatial behavior feature matrix corresponds to the time step, and the column dimension corresponds to the behavior feature channel.

[0009] Preferably, the transaction flow receiving interface converts the transaction occurrence time into a relative timestamp, the product category into a one-hot encoded vector, and the processing time into a normalized scalar. The relative timestamp, the one-hot encoded vector, and the normalized scalar are concatenated to obtain the transaction event vector. The temporal Transformer includes a position encoder and a multi-head self-attention layer. The position encoder uses sinusoidal position encoding to inject temporal position information into the transaction event vector. The multi-head self-attention layer sets up 8 attention heads and maps the transaction event vector into a query matrix, a key matrix, and a value matrix through linear mapping, outputting transaction context features.

[0010] Preferably, the cross-attention alignment module transforms the transaction context features through a first linear layer to generate a query matrix, and transforms the spatial behavior features through a second and third linear layer to generate a key matrix and a value matrix, respectively. The query matrix and the transposed key matrix are multiplied together and divided by the square root of the key matrix's dimension. The result is input into a softmax function to obtain a cross-modal similarity matrix. The cross-modal similarity matrix and the value matrix are multiplied together to obtain preliminary fused features. The preliminary fused features are added element-wise to the query matrix transformed through a fourth linear layer to achieve residual connection. The residual connection result is input into a layer normalization layer, and a joint feature vector is output.

[0011] Preferably, after calculating the loss function between the performance score output by the fully connected layer and the true performance label, the backpropagation process is performed. During the backpropagation process, the gradient matrix of the loss function with respect to the timestamp dimension of the transaction event vector is extracted. The gradient values ​​in the gradient matrix corresponding to time steps where no transaction occurred are set to zero, and the gradient values ​​corresponding to time steps where a transaction occurred are retained to form a truncated gradient matrix. After multiplying the truncated gradient matrix by a preset learning rate, the weights of the temporal connection edges and the weights of the physical connection edges corresponding to time steps where a transaction occurred in the spatiotemporal graph are updated.

[0012] Preferably, when performing forward kinematic interpolation for completion of occluded keypoints, the coordinates of the parent node and child node of the occluded keypoint are extracted, and the bone vector pointing from the parent node to the child node is calculated; the bone length ratio and deflection angle of the occluded keypoint relative to the parent node in the standard pose are obtained; starting from the parent node coordinates, the bone vector is rotated and scaled according to the bone length ratio and deflection angle to calculate the predicted coordinates of the occluded keypoint; the predicted coordinates are used to replace the original coordinates of the occluded keypoint in the skeleton node sequence and output to the spatiotemporal graph construction process.

[0013] Preferably, after the output of the third graph convolutional layer is flattened and input into a one-dimensional convolutional layer, the output feature map of the one-dimensional convolutional layer is input into a max pooling layer. The max pooling layer is set with a pooling window size of 2 and a stride of 2 to downsample the temporal dimension of the output feature map. The downsampled output feature map is then input into a one-dimensional dilated convolutional layer, which is set with a dilation rate of 2 and a kernel size of 3. Long-distance temporal dependency features are extracted from the downsampled output feature map, and these long-distance temporal dependency features are reorganized into a spatial behavior feature matrix.

[0014] Preferably, the multi-head self-attention layer introduces a time interval mask matrix, the dimension of which is consistent with the sequence length of the transaction event vector; the relative time difference between two adjacent vectors in the transaction event vector is calculated, and when the relative time difference is greater than a preset time interval threshold, the element at the corresponding row and column position in the time interval mask matrix is ​​set to negative infinity, and when the relative time difference is less than or equal to the preset time interval threshold, the element at the corresponding position is set to 0; the product of the time interval mask matrix with the query matrix and the key matrix is ​​added element by element and then input into the softmax function.

[0015] Preferably, when updating the edge weights of the spatiotemporal graph corresponding to the transaction occurrence time step, the truncated gradient matrix of the transaction occurrence time step is input into the first fully connected layer and the second fully connected layer; the first fully connected layer maps the truncated gradient matrix to the temporal connection edge weight update amount, and the second fully connected layer maps the truncated gradient matrix to the physical connection edge weight update amount; the temporal connection edge weight update amount is added to the initial temporal connection edge weights of the spatiotemporal graph, and the physical connection edge weight update amount is added to the initial physical connection edge weights of the spatiotemporal graph, and the result of the sum is used as the edge weights of the spatiotemporal graph in the next round of forward propagation.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0017] 1. This invention establishes a specific causal correspondence between operational actions and transaction results by constructing a spatiotemporal graph and transaction event vectors, and introducing a cross-attention alignment module. Using the timestamp of the transaction event vector as the query key, it retrieves and calculates the cross-modal similarity matrix along the temporal dimension of spatial behavioral features. This allows the model to directly focus on the specific actions that trigger transactions during the feature extraction stage, eliminating redundant action features with no transaction association within a fixed time window. Combined with gradient truncation during backpropagation, only the edge weights of the spatiotemporal graph corresponding to the time step where a transaction occurred are updated. This further strengthens the binding relationship between physical actions and transaction events at the time node level at the parameter update level, reducing the noise impact of non-business behaviors on the evaluation results.

[0018] 2. In the pre-feature extraction stage, to address key point occlusion in the surveillance video, forward kinematics interpolation based on the skeletal vectors of parent and child nodes is used to complete the spatiotemporal graph vertex coordinate data. In the spatial behavior feature extraction stage, features from local limbs to global body pose are extracted step-by-step through a three-layer graph convolutional layer, and then processed by max pooling downsampling and one-dimensional dilated convolution to extract long-distance temporal dependency features in the action sequence. In the transaction context feature extraction stage, a time interval mask matrix is ​​introduced to shield attention calculations between transaction events with excessively large time intervals, avoiding erroneous association of irrelevant transaction records within the time-series transformer. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the overall system workflow of the present invention.

[0020] Figure 2 This is a flowchart of the video acquisition front-end image processing and skeleton node extraction process of the present invention;

[0021] Figure 3 This is a flowchart of the spatiotemporal graph construction process of the present invention;

[0022] Figure 4 This is a flowchart of the spatial behavior feature extraction process of the present invention;

[0023] Figure 5 This is a flowchart illustrating the transaction event vector construction and transaction context feature extraction process of the present invention.

[0024] Figure 6 This is a flowchart of the cross-attention alignment fusion and performance scoring output of the present invention. Detailed Implementation

[0025] refer to Figure 1In one embodiment, a deep learning-based store operations performance evaluation system is deployed in the computing unit of a retail store. The system includes a video acquisition front-end and a transaction log receiving interface. The video acquisition front-end establishes a data communication link with the surveillance video acquisition equipment installed in the store to acquire real-time surveillance video streams within the store's operating area. The surveillance video streams are processed frame by frame to extract the skeleton node sequences of operations personnel from the video frames, and a spatiotemporal graph is constructed based on these skeleton node sequences. The spatiotemporal graph uses the skeleton nodes extracted from a single frame as vertices and the physical distance and relative motion speed between nodes as edge weights to complete the spatiotemporal topological representation of the operations personnel's action sequences. The constructed spatiotemporal graph is input into a graph convolutional network. The graph convolutional network performs convolution operations on the topological structure and node features of the spatiotemporal graph to extract the spatial behavioral features of the operations personnel. The transaction log receiving interface establishes data communication with the store's POS management system through a standardized application programming interface (API) to acquire all transaction log data within the evaluation period. From each transaction log data point, three core fields are extracted: transaction occurrence time, product category, and processing time. A transaction event vector is constructed based on these extracted fields. The completed transaction event vectors are input into a temporal Transformer. The temporal Transformer models the temporal dependencies of the transaction event vector sequence to extract transaction context features. The system also includes a cross-attention alignment module. This module receives transaction context features and spatial behavior features, uses the timestamp of the transaction event vector as the query key, performs feature retrieval along the temporal dimension of the spatial behavior features, calculates the cross-modal similarity matrix between the transaction context features and the spatial behavior features, and aligns and fuses the two types of features based on the cross-modal similarity matrix, outputting a joint feature vector that fuses the operational actions and transaction results. The output joint feature vector is input into a fully connected layer. The fully connected layer performs dimensionality mapping and linear transformation on the joint feature vector, outputting the performance score of the operations personnel within the corresponding evaluation period.

[0026] refer to Figure 2The evaluation cycle is pre-set according to the store's performance evaluation needs, and can be set to a daily, weekly, or monthly cycle. The video data collection time range within the evaluation cycle is completely consistent with the statistical time range of transaction flow data, ensuring the consistency of the time base for the two types of data. The video acquisition front end decodes the acquired monitoring video stream, converting the compressed video stream into frame-by-frame RGB image data. The converted image data is then normalized, mapping the image pixel values ​​from the integer range of 0-255 to the floating-point range of 0-1. Simultaneously, the image is scaled, scaling the width and height to a preset uniform size to ensure the consistency of the input data dimensions in the subsequent skeleton node extraction process. Furthermore, the video acquisition front end performs human detection and instance segmentation on the pre-processed image, identifying all human instances in the image. Combined with a preset store operation area mask, human instances located within the operation area are filtered out, while human instances of customers or other irrelevant personnel located outside the operation area are excluded, thus determining the human instances of the operation personnel to be extracted for skeleton nodes. Specifically, the operational area mask is a pre-labeled binary image. Pixels in the mask corresponding to the store's operational area have a value of 1, while pixels corresponding to non-operational areas have a value of 0. A bitwise AND operation is performed between the bounding rectangle of a human instance and the operational area mask. If the percentage of valid pixels in the result exceeds a preset threshold, the human instance is identified as an operational staff member and included in the subsequent skeleton node extraction range. When multiple operational staff members exist in a single frame, an independent skeleton node sequence is generated for each staff member, and subsequent processing is performed separately to ensure that each staff member's performance score is calculated independently without interference.

[0027] refer to Figure 3For the selected human body instances of operational personnel, the video acquisition front-end extracts human skeleton nodes from a single frame image. Each skeleton node contains the horizontal and vertical coordinates in a two-dimensional image plane, as well as the detection confidence score of that node. The skeleton nodes corresponding to consecutive video frames are arranged in the order of the video frame acquisition time to form a skeleton node sequence. The time step length of the skeleton node sequence is consistent with the number of video frames in the evaluation period, and each time step corresponds to the skeleton node set of one video frame. A spatiotemporal graph is constructed based on the skeleton node sequence. The spatiotemporal graph is an undirected weighted graph, and the vertex set of the graph contains all skeleton nodes of all time steps in the skeleton node sequence. The edges of the spatiotemporal graph are divided into two categories: the first category is physical connection edges, which are the connection edges between two skeleton nodes with a human physiological structure connection relationship within the same time step; the second category is temporal connection edges, which are the connection edges between the same skeleton nodes in two adjacent time steps. Physical connection edges and temporal connection edges together constitute the edge set of the spatiotemporal graph. The weight of each edge in the edge set is calculated based on the physical distance and relative motion speed between the corresponding nodes. Specifically, the weight of a physical connection edge is calculated based on the physical distance between two connected nodes within the same time step, while the weight of a temporal connection edge is calculated based on the relative motion velocity of the same node within adjacent time steps. The two types of weights together constitute the total weight of the corresponding edge in the spatiotemporal graph.

[0028] Table 1. Definitions and Initial Physical Connections of Human Skeleton Nodes

[0029]

[0030] Table 1 defines the physical meaning of vertices and the initial physical connection relationships during the construction of the spatiotemporal graph, providing a standardized topological basis for the generation of physical connection edges. This ensures that the basic topological structure of the spatiotemporal graph remains consistent under different operators and different acquisition environments, and provides a unified input format for feature extraction by subsequent graph convolutional networks.

[0031] refer to Figure 4The constructed spatiotemporal graph is transformed into corresponding node feature matrices and adjacency matrices, which are then input into the graph convolutional network. Each row of the node feature matrix corresponds to a skeleton node, and each column corresponds to the feature dimension of the node. The node features include the x-coordinate, y-coordinate, and confidence score of the node. The adjacency matrix is ​​a two-dimensional square matrix, with the dimension of the matrix matching the total number of vertices. The element value at the corresponding position in the adjacency matrix is ​​the weight of the edge between the corresponding two vertices in the spatiotemporal graph, and the element value at the corresponding position of a vertex without an edge is 0. The graph convolutional network performs multiple rounds of graph convolution operations on the input node feature matrix and adjacency matrix, aggregating the feature information of adjacent nodes to extract the spatial behavioral features of the operators' actions. The spatial behavioral features are output in matrix form, with the row dimension corresponding to the time step and the column dimension corresponding to the behavioral feature channel. The feature vector at each time step corresponds to the spatial feature representation of the operators' actions at that time.

[0032] refer to Figure 5 The transaction log receiving interface retrieves transaction log data from the POS management system and sorts it according to the chronological order of transaction occurrences to form a transaction log sequence. For each individual transaction log entry in the sequence, three core fields are extracted: transaction occurrence time, product category, and processing time. The transaction occurrence time is the system record time when the payment for that transaction is completed; the product category is the highest-level category identifier corresponding to all products included in that transaction; and the processing time is the time interval from the start of scanning to the completion of payment. Based on these three core fields, the data is structured and concatenated to generate a transaction event vector corresponding to each transaction log entry. All transaction event vectors are arranged in chronological order of transaction occurrences to form a transaction event vector sequence.

[0033] The transaction event vector sequence is input into a temporal Transformer. The temporal Transformer first injects positional information into the transaction event vector sequence, encoding the temporal positional information of the transaction events into the transaction event vectors. Then, a multi-head self-attention layer models the contextual dependencies of the injected positional information transaction event vector sequence, capturing the temporal relationships between different transaction events and outputting transaction context features. These transaction context features retain the temporal information of the transaction events along with business information such as category and duration, while also incorporating the contextual relationship features of adjacent transaction events.

[0034] refer to Figure 6The cross-attention alignment module receives transaction context features output by the temporal Transformer and spatial behavior features output by the graph convolutional network. Using the timestamp corresponding to the transaction event vector as the query key, the module performs a full search along the temporal dimension of the spatial behavior features, obtaining the time step corresponding to each transaction event timestamp and the spatial behavior features within a preset time window before and after that time step. The preset time window is adjusted according to the store's business type: 3 seconds for fast-moving consumer goods stores and 10 seconds for bulk commodity stores, ensuring coverage of the complete service action sequence before the transaction. Further, the cross-attention alignment module calculates the feature similarity between the transaction context features and the retrieved spatial behavior features, generating a cross-modal similarity matrix. The element values ​​in the cross-modal similarity matrix correspond to the degree of association between the transaction event and the corresponding time step action features. Based on the cross-modal similarity matrix, the spatial behavior features are weighted and summed. The weighted summed spatial behavior features are then fused with the transaction context features to generate a joint feature vector that integrates the fused operation actions and transaction results.

[0035] The joint feature vector is input into a fully connected layer, which contains at least one linear transformation layer. The input dimension of the linear transformation layer is the same as the dimension of the joint feature vector, and the output dimension is 1. The high-dimensional joint feature vector is mapped to a single-dimensional quantized value through linear transformation. This quantized value is the performance score of the operator in the corresponding evaluation period.

[0036] This embodiment achieves a unified topological representation of the spatial structure and temporal motion characteristics of operators' action sequences by constructing a spatiotemporal graph. It also achieves a structured and standardized representation of transaction flow data by constructing transaction event vectors. Furthermore, it achieves precise alignment and fusion of transaction context features and spatial behavioral features by using the timestamp of transaction events as the core through a cross-attention alignment module. This establishes a correspondence between operators' physical spatial operation actions and digital transaction results, avoiding interference from redundant action features without transaction association within a fixed time window on the performance evaluation process.

[0037] In a preferred embodiment, the video acquisition front-end extracts the coordinates of key skeletal points of the operators using the OpenPose algorithm to form a skeleton node sequence, and uses forward kinematics interpolation to complete occluded key points. When constructing the spatiotemporal graph, skeleton nodes at the same time are connected by physical edges, and the same skeleton nodes at adjacent time points are connected by temporal edges. When calculating the weight of physical edges, the Euclidean distance between adjacent skeleton nodes is extracted and divided by a preset standard skeleton length as the spatial weight; when calculating the weight of temporal edges, the difference between the magnitude of the displacement vector of the same skeleton node at adjacent time points and a preset velocity threshold is extracted as the temporal weight. The spatial weight and temporal weight are added together to obtain the edge weights of the spatiotemporal graph.

[0038] Specifically, the OpenPose algorithm extracts features from pre-trained convolutional neural networks on preprocessed single-frame images. First, the input image is downsampled and feature-encoded using the first 10 convolutional layers of the VGG-19 backbone network, generating a high-dimensional feature map. This high-dimensional feature map is then fed into two parallel convolutional branches. The first branch is a keypoint heatmap prediction branch, which generates heatmaps corresponding to human skeleton nodes through multi-layer convolutional operations. The value of each pixel in the heatmap corresponds to the probability that its location corresponds to a skeleton node. The second branch is a component affinity field prediction branch, which generates component affinity fields corresponding to human skeletal segments through multi-layer convolutional operations. These component affinity fields are two-dimensional vector fields representing the orientation and position information of the corresponding skeletal segments. Further, non-maximum suppression is applied to the heatmap output by the keypoint heatmap prediction branch, extracting local peak points as candidate keypoints for skeleton nodes. Each candidate keypoint includes its coordinates and a confidence score. For the extracted candidate keypoints, based on the component affinity field output by the component affinity field prediction branch, the integral value of the connection between two candidate keypoints is calculated. When the integral value is greater than a preset threshold, it is determined that the two candidate keypoints have a skeletal connection relationship. This completes the matching and assembly of human skeletal keypoints within a single frame image, obtaining the coordinates of the operator's skeletal keypoints and forming a skeleton node set for a single frame. The skeleton node sets of consecutive frames are arranged in chronological order to form a skeleton node sequence.

[0039] During skeleton node extraction, when the confidence score of a skeleton node is lower than a preset confidence threshold, the keypoint is determined to be an occluded keypoint, and forward kinematics interpolation is used to complete the occluded keypoint. Specifically, when performing forward kinematics interpolation on the occluded keypoint, the coordinates of the parent node and child node of the occluded keypoint are first extracted. The parent node is the node in the human skeleton hierarchy that has a direct connection to the occluded keypoint and is closer to the center of the human torso. The child node is the node in the human skeleton hierarchy that has a direct connection to the occluded keypoint and is farther away from the center of the human torso. Based on the extracted parent node coordinates and child node coordinates, the bone vector pointing from the parent node to the child node is calculated. Furthermore, the proportion of bone length and the deflection angle of the occluded keypoint relative to the parent node in the standard pose are obtained. The standard pose is a predefined neutral human pose in which all bone segments are in the standard extended state corresponding to the physiological structure. Starting from the parent node coordinates, the calculated bone vector is rotated and scaled according to the obtained bone length ratio and deflection angle to calculate the predicted coordinates of the occluded keypoints. The calculated predicted coordinates replace the original coordinates of the occluded keypoints in the skeleton node sequence and are output to the subsequent spatiotemporal graph construction process.

[0040]

[0041] in, The predicted two-dimensional coordinate vector of the occluded key point. The parent node's two-dimensional coordinate vector for the occluded keypoint. The two-dimensional coordinate vector of the child nodes of the occluded keypoint. Based on standard deflection angle Two-dimensional rotation matrix, This is a scaling factor based on the standard bone length ratio.

[0042] Two-dimensional rotation matrix The specific form is:

[0043]

[0044] in, The angle of deflection of the occluded keypoint relative to the line connecting the parent and child nodes in the standard pose.

[0045] When constructing the spatiotemporal graph, based on the physical connection relationships of the skeleton nodes defined in Table 1, physical connection edges are generated between skeleton nodes at the same time. The two endpoints of the physical connection edge are adjacent nodes with physical connection relationships as defined in Table 1. At the same time, temporal connection edges are generated between the same skeleton nodes at adjacent time points. The two endpoints of the temporal connection edge are skeleton nodes with the same number at the previous time point and the current time point.

[0046]

[0047] in, Let be the spatial weight of the physical connection edge between node i and node j at time t. Let be the Euclidean distance between the coordinates of node i and node j at time t. Let be the standard bone length between node i and node j. The standard bone length is the length of the corresponding bone segment under the standard pose.

[0048]

[0049] in, Let be the time weight of the temporal connection edge between node i at time t and time t-1. Let be the magnitude of the displacement vector of node i between two adjacent frames. The preset speed threshold is based on the video sampling frame rate and the speed range of normal human movements.

[0050]

[0051] in, This is the total adjacency matrix of the spatiotemporal graph, where the matrix elements are the weights of the corresponding edges; It is a spatial adjacency matrix, the dimension of which is the number of skeleton nodes in a single frame × the number of skeleton nodes in a single frame. Non-zero elements are the spatial weights of node pairs that are physically connected at the same time, and the rest of the elements are 0. This is a temporal adjacency matrix, with dimensions equal to the total number of time steps multiplied by the total number of time steps. Non-zero elements represent the time weights of the same node at adjacent times, while the remaining elements are 0.

[0052] Table 2. Relationship between Standard Skeletal Parameters and Hierarchical Structure of Human Skeletal Nodes

[0053]

[0054] Table 2 defines the hierarchical topology and standard kinematic parameters of the human skeleton. The standard bone length ratio is the ratio of the corresponding bone segment length to the total height of the human body, and the standard deflection angle is the deflection angle of the corresponding bone segment relative to its parent node bone segment in the standard pose. This table provides a standardized calculation basis for forward kinematic interpolation completion of occluded keypoints, ensuring that the completed keypoint coordinates conform to human kinematics. It also provides a benchmark for the standard bone length in the spatial weight calculation process.

[0055] Furthermore, when constructing the total adjacency matrix of the spatiotemporal graph, for each time step, the spatial adjacency matrix corresponding to that time step is superimposed with the temporal adjacency matrix to generate the spatiotemporal adjacency matrix corresponding to that time step. The spatiotemporal adjacency matrices of all time steps are combined in chronological order to form the complete total adjacency matrix of the spatiotemporal graph, which is then input into the subsequent graph convolutional network.

[0056] This embodiment achieves stable extraction and matching of key points on the skeletons of operational personnel using the OpenPose algorithm. By employing forward kinematic interpolation based on the skeletal vectors of parent and child nodes, it solves the problem of missing key point data caused by personnel occlusion and viewpoint limitations in store monitoring scenarios, ensuring the integrity of vertex coordinate data in the spatiotemporal graph. Through edge weight calculation combining physical distance and relative motion velocity, it achieves a quantitative representation of the spatial structure changes and temporal motion features of actions in the spatiotemporal graph, providing high-quality input data for subsequent spatial behavior feature extraction.

[0057] In another preferred embodiment, the graph convolutional network comprises three graph convolutional layers. The first graph convolutional layer extracts local limb features of adjacent skeleton nodes, the second graph convolutional layer extracts cross-limb trunk coordination features, and the third graph convolutional layer extracts global body pose features. Each graph convolutional layer is followed by a batch normalization layer and a modified linear unit activation function. The output of the third graph convolutional layer is flattened and processed by a one-dimensional convolutional layer with a kernel size of 3, recombining the spatial dimension and channel dimension to obtain a spatial behavior feature matrix. The row dimension of the spatial behavior feature matrix corresponds to the time step, and the column dimension corresponds to the behavior feature channel. The transaction log receiving interface converts the transaction occurrence time into a relative timestamp, the product category into a one-hot encoded vector, and the processing time into a normalized scalar. The relative timestamp, one-hot encoded vector, and normalized scalar are concatenated to obtain the transaction event vector. The temporal Transformer includes a position encoder and a multi-head self-attention layer. The position encoder uses sinusoidal position encoding to inject temporal position information into the transaction event vector. The multi-head self-attention layer sets up 8 attention heads and maps the transaction event vector into a query matrix, a key matrix, and a value matrix through linear mapping, outputting transaction context features.

[0058] Specifically, the input to the first graph convolutional layer is the node feature matrix and the total adjacency matrix of the spatiotemporal graph. The feature dimension of the node feature matrix is ​​3, corresponding to the x-coordinate, y-coordinate, and confidence score of each skeleton node. The convolution operation of the first graph convolutional layer focuses on feature aggregation between adjacent skeleton nodes with direct physical connections within a single frame, extracting local limb action features, such as arm bending and leg extension. The output of the first graph convolutional layer is fed into a batch normalization layer. The batch normalization layer normalizes the mean and variance of each channel of the output features, mapping the feature values ​​to the standard normal distribution range, alleviating the internal covariate shift problem during network training. The output of the batch normalization layer is fed into a modified linear unit activation function. The activation function performs a nonlinear transformation on the feature values, introducing nonlinear expressive power into the network. The expression of the activation function is: , where x is the input feature value.

[0059] The input to the second graph convolutional layer is the output feature of the first graph convolutional layer after processing by the activation function. The convolution operation of the second graph convolutional layer focuses on extracting cross-limb trunk coordination features, aggregating features from multiple limb nodes with indirect connections, such as coordinated movements between the upper limbs and trunk, and coordinated movements between the left and right limbs. The output of the second graph convolutional layer is also fed into a batch normalization layer and a modified linear unit activation function to complete the feature normalization and nonlinear transformation.

[0060] The input to the third graph convolutional layer is the output feature of the second graph convolutional layer after the activation function has been processed. The convolution operation of the third graph convolutional layer focuses on the extraction of global body posture features, aggregating the feature information of all skeleton nodes to capture the overall body posture changes and global action patterns of the operators. The output of the third graph convolutional layer is fed into a batch normalization layer and a modified linear unit activation function to complete the final graph convolutional feature extraction.

[0061]

[0062] in, For the first The input feature matrix of the layered graph convolutional layer, This is the output feature matrix of the l-th graph convolutional layer; To add the total adjacency matrix of self-loops, It is the identity matrix; for The degree matrix is ​​a diagonal matrix, and the elements on the diagonal are... ; Let be the trainable weight matrix of the l-th graph convolutional layer; To modify the activation function of the linear unit.

[0063] The feature map output by the third-layer convolutional layer is a three-dimensional feature tensor. The three dimensions of the tensor are the time step dimension, the number of nodes dimension, and the feature channel dimension. The three-dimensional feature tensor output by the third-layer convolutional layer is flattened, concatenating the node number dimension and the feature channel dimension corresponding to each time step to transform it into a two-dimensional feature matrix. The row dimension of the two-dimensional feature matrix is ​​the time step, and the column dimension is the product of the number of nodes and the feature channels. The flattened two-dimensional feature matrix is ​​input into a one-dimensional convolutional layer. The kernel size of the one-dimensional convolutional layer is set to 3, the stride to 1, and the padding to 1 to ensure that the time step dimension of the feature matrix remains unchanged after convolution. The one-dimensional convolutional layer performs a convolution operation on the flattened feature matrix, recombining the node dimension and the feature channel dimension to eliminate the influence of the node number dimension, generating a behavior feature vector of a unified dimension corresponding to each time step. The behavior feature vectors of all time steps are combined to form a spatial behavior feature matrix, where the row dimension corresponds to the time step, and the column dimension corresponds to the behavior feature channels.

[0064] Furthermore, after the output of the third graph convolutional layer is flattened and input into a one-dimensional convolutional layer, the output feature map of the one-dimensional convolutional layer is input into a max pooling layer. The max pooling layer is set with a pooling window size of 2 and a stride of 2 to downsample the temporal dimension of the output feature map. The downsampled output feature map is then input into a one-dimensional dilated convolutional layer, which is set with a dilation rate of 2 and a kernel size of 3. Long-distance temporal dependency features are extracted from the downsampled output feature map, and these long-distance temporal dependency features are reorganized into a spatial behavior feature matrix.

[0065] Specifically, the max pooling layer downsamples the temporal dimension of the feature matrix output from the one-dimensional convolutional layer, taking the maximum value within each pooling window of size 2 as the output. This reduces the time step length of the feature matrix to half its original value, lowering the temporal resolution of the features while retaining the most salient action features within each time window, thus reducing the computational cost of subsequent operations. The downsampled feature matrix is ​​then input into a one-dimensional dilated convolutional layer. This layer has a kernel size of 3, a dilation rate of 2, and an effective receptive field of 5. This expands the temporal receptive field of feature extraction without increasing the number of kernel parameters, capturing long-distance temporal dependencies in action sequences, such as coherent action sequences across multiple time steps prior to a transaction. The output of the one-dimensional dilated convolutional layer is then re-divided to generate the final spatial behavior feature matrix, which is then input into the subsequent cross-attention alignment module.

[0066] Table 3 Core parameters and input / output dimensions of each layer of the spatial behavior feature extraction network

[0067]

[0068] Table 3 clarifies the input and output dimensions and core computational parameters of each layer of the spatial behavior feature extraction network, fully covering the entire process of computational dimension transformation from three-layer graph convolutional layers to one-dimensional dilated convolutional layers. This provides a standardized basis for network deployment, reproduction, and parameter adjustment, ensuring the consistency of dimension transformation and the traceability of computational logic during feature extraction.

[0069] The transaction log receiving interface converts the extracted transaction occurrence times into relative timestamps. Specifically, taking the start time of the evaluation period as the zero point, the occurrence time of each transaction is converted into the number of seconds from the zero point, serving as the relative timestamp for that transaction event. The time base of the relative timestamp is consistent with the acquisition time base of the video frame, ensuring precise alignment between the timestamps of the transaction event and the video frame. The extracted product categories are converted into one-hot encoded vectors. The preset product category set contains N predefined primary product categories, each corresponding to an N-dimensional one-hot encoded vector. Only the element corresponding to the category in the vector has a value of 1, while the other elements have a value of 0. When a single transaction contains multiple product categories, the one-hot encoded vector corresponding to the product category with the highest transaction amount percentage is taken as the product category representation for that transaction event. Alternatively, the one-hot encoded vectors of all product categories can be weighted and summed, with the weight being the transaction amount percentage of the corresponding product, to generate the product category representation for that transaction event. The extracted processing time is converted into a normalized scalar. Specifically, the original processing time in seconds is divided by a preset maximum processing time, mapping the processing time to the range of 0-1 to obtain a normalized scalar. The preset maximum processing time is pre-set based on the processing time statistics of the store's daily transactions. The converted relative timestamp, one-hot encoded vector, and normalized scalar are concatenated to obtain the transaction event vector corresponding to a single transaction event. All transaction event vectors are arranged in chronological order according to their relative timestamps to form a transaction event vector sequence.

[0070] The temporal Transformer includes a position encoder and a long self-attention layer. The position encoder uses sinusoidal position encoding, which injects the transaction event vector into the temporal position information. The calculation formula for sinusoidal position encoding is as follows:

[0071]

[0072]

[0073] in, This is the position index of the transaction event vector in the sequence, starting from 0 and increasing sequentially according to the order in which the transactions occurred; This is the index for the feature dimension, with values ​​ranging from 0 to... ; The total dimension of the transaction event vector; For even-numbered dimensions in positional encoding, For odd-numbered dimensions in location encoding.

[0074] Specifically, the location encoder adds the calculated location encoding vector element-wise to the corresponding transaction event vector, injecting temporal location information into the transaction event vector. This enables the temporal Transformer to capture the temporal order information of the transaction event sequence. The transaction event vector sequence after injecting location information is input into a multi-head self-attention layer. The multi-head self-attention layer sets up eight attention heads and maps the transaction event vector into a query matrix, key matrix, and value matrix through linear mapping. It captures the contextual dependencies between different transaction events through self-attention operations and outputs transaction context features.

[0075] Furthermore, a time interval mask matrix is ​​introduced into the long-side self-attention layer. The dimension of the time interval mask matrix is ​​consistent with the sequence length of the transaction event vector. The relative time difference between two adjacent vectors in the transaction event vector is calculated. When the relative time difference is greater than a preset time interval threshold, the element at the corresponding row and column position in the time interval mask matrix is ​​set to negative infinity. When the relative time difference is less than or equal to the preset time interval threshold, the element at the corresponding position is set to 0. The product of the time interval mask matrix and the query matrix and the key matrix is ​​added element by element and then input into the softmax function.

[0076]

[0077]

[0078]

[0079] in, The number of attention heads is set to 8 here; , , These are the linear mapping weight matrices for the query, key, and value corresponding to the m-th attention head, respectively. The linear mapping weight matrix for the multi-head attention output; Let be the dimension of the key matrix; This is a time interval mask matrix; This is a concatenation operation for multiple attention head outputs.

[0080]

[0081] in, Let be the value of the element in the p-th row and q-th column of the time interval mask matrix; This represents the relative timestamp corresponding to the p-th transaction event vector; This is the relative timestamp corresponding to the q-th transaction event vector; The preset time interval threshold is based on the statistical data of daily transactions in the store.

[0082] Specifically, the time interval mask matrix serves to shield attention calculations between trading events with excessively long time intervals. When the time interval between two trading events exceeds a preset threshold, the mask value at the corresponding position is negative infinity. After processing by the softmax function, the attention weight at that position approaches 0, thus avoiding erroneous associations between unrelated trading events with excessively long time intervals. This ensures that the long-only self-attention layer only captures the contextual dependencies between trading events with similar time intervals. The output of the long-only self-attention layer undergoes linear transformation and layer normalization to generate the final trading context features, which are then input into the subsequent cross-attention alignment module.

[0083] This embodiment utilizes a hierarchical design of a three-layer graph convolutional network to extract multi-scale spatial behavioral features, ranging from local limb features and cross-limb torso collaboration features to global body posture features. By combining max pooling downsampling with one-dimensional dilated convolution, it effectively captures long-distance temporal dependencies in action sequences while reducing computational cost. A standardized transaction event vector construction process enables structured representation of transaction flow data. A temporal Transformer with time interval masks effectively shields against interference from irrelevant transaction events with excessively long time intervals, achieving accurate modeling of transaction event context dependencies and enhancing the representation capability of transaction context features.

[0084] In another preferred embodiment, the cross-attention alignment module transforms the transaction context features through a first linear layer to generate a query matrix, and transforms the spatial behavior features through a second and third linear layer to generate a key matrix and a value matrix, respectively. The query matrix and the transposed key matrix are multiplied together and divided by the square root of the key matrix's dimension. The result is input into a softmax function to obtain a cross-modal similarity matrix. The cross-modal similarity matrix and the value matrix are multiplied together to obtain preliminary fused features. The preliminary fused features are added element-wise to the query matrix transformed through a fourth linear layer to achieve residual connections. The residual connection result is input into a normalization layer, and a joint feature vector is output. After calculating the loss function between the performance score output by the fully connected layer and the true performance label, the backpropagation process is performed. During the backpropagation process, the gradient matrix of the loss function with respect to the timestamp dimension of the transaction event vector is extracted. The gradient values ​​in the gradient matrix corresponding to time steps where no transaction occurred are set to zero, and the gradient values ​​corresponding to time steps where a transaction occurred are retained to form a truncated gradient matrix. After multiplying the truncated gradient matrix by a preset learning rate, the weights of the temporal and physical connections in the spatiotemporal graph corresponding to time steps where a transaction occurred are updated.

[0085] Specifically, the cross-attention alignment module receives transaction context features output by the temporal Transformer. These features are a two-dimensional matrix, with rows representing the number of transaction events and columns representing the feature channel dimensions. The transaction context features are input to the first linear layer, which performs a linear transformation, mapping the feature channel dimensions to a preset key-value dimension to generate a query matrix. Each row of the query matrix corresponds to a query vector for a transaction event, with the timestamp of that transaction event as the core query key. Simultaneously, the cross-attention alignment module receives the spatial behavior feature matrix output by the graph convolutional network. The row dimension of the spatial behavior feature matrix represents the number of time steps, and the column dimension represents the feature channel dimension. This spatial behavior feature matrix is ​​then input to the second and third linear layers. The second linear layer performs a linear transformation on the spatial behavior feature matrix to generate a key matrix that matches the dimensions of the query matrix. Each row of the key matrix corresponds to a key vector at a time step; the third linear layer performs a linear transformation on the spatial behavior feature matrix to generate a value matrix. Each row of the value matrix corresponds to a value vector for a time step.

[0086] Furthermore, the query matrix and the transposed key matrix are multiplied to obtain an initial similarity matrix. This initial similarity matrix is ​​then divided by the square root of the key matrix's dimension to scale the similarity values, preventing the vanishing gradient problem of the softmax function due to excessively large similarity values ​​at high dimensions. The scaled similarity matrix is ​​then input into the softmax function, which normalizes the similarity values ​​in each row, ensuring the sum of the elements in each row is 1, resulting in a cross-modal similarity matrix. Each row in the cross-modal similarity matrix corresponds to a transaction event, and each column corresponds to a time step. The element values ​​represent the association weight between the transaction event and the action features at the corresponding time step; a higher weight indicates a stronger association between the action at that time step and the transaction event.

[0087] The cross-modal similarity matrix and the value matrix are multiplied together. The row vectors of the value matrix are then weighted and summed according to their association weights to obtain the fused action features for each transaction event. The fused action features of all transaction events are combined to form the preliminary fused features. These preliminary fused features are then added element-wise to the query matrix transformed by the fourth linear layer to achieve residual connections. Residual connections alleviate the gradient vanishing problem during deep network training and preserve the core information of the transaction context features. The results of the residual connections are input to the normalization layer. The normalization layer normalizes the mean and variance of each feature sample and outputs the final joint feature vector. This joint feature vector integrates the contextual information of the transaction event with the spatial behavioral features of the corresponding action, establishing a correspondence between the operation action and the transaction result.

[0088]

[0089] in, This is a cross-modal similarity matrix; The query matrix is ​​generated by transforming the transaction context features through the first linear layer. The key matrix is ​​generated by transforming spatial behavior features through a second linear layer. The feature dimension of the key matrix; This is a normalized exponential function operation performed row by row.

[0090]

[0091] in, This is the joint feature vector of the output; The value matrix generated by transforming spatial behavior features through the third linear layer; This is the weight matrix for the fourth linear layer, used to query the linear transformation of the matrix to match the dimension; This is a layer normalization operation.

[0092] The joint feature vector is input into a fully connected layer. The fully connected layer performs global average pooling on the joint feature vector, aggregating the joint feature vectors from multiple transaction events into a single-dimensional feature vector. This single-dimensional feature vector is then mapped to a single-dimensional performance score through a linear transformation layer. The loss function between the performance score output by the fully connected layer and the true performance label is calculated, using the mean squared error function.

[0093]

[0094] in, The loss function value used for model training; The performance score output by the fully connected layer; The actual performance label is a manually labeled quantitative value of the operations staff's performance. This refers to the number of samples within a training batch. This is for the calculation of mean square error.

[0095] After the loss function is calculated, backpropagation is performed. The gradient values ​​of the loss function with respect to each trainable parameter of the model are calculated based on the chain rule, and the trainable parameters of the model are updated based on these gradient values. During backpropagation, the gradient matrix of the loss function with respect to the timestamp dimension of the transaction event vector is extracted. The row dimension of the gradient matrix represents the time step, and the column dimension represents the feature dimension. Each element is the gradient value of the loss function with respect to the feature at the corresponding time step. The extracted gradient matrix is ​​then truncated, setting the gradient values ​​for time steps without transactions to zero and retaining only the gradient values ​​for time steps with transactions, forming a truncated gradient matrix.

[0096]

[0097] in, The cutoff gradient value at time t; This represents the original gradient value of the loss function with respect to the timestamp dimension of the transaction event vector at time t; This is the set of time steps corresponding to the time when all transactions occur, including the time step corresponding to the time when the transaction occurs, as well as the time steps within the preset time window before and after that time step.

[0098] Specifically, the preset time window is pre-set based on the average duration of transaction processing, ensuring that the gradient values ​​of the relevant action time steps before and after a transaction are retained, while the gradient values ​​of time steps without transactions are all set to zero and do not participate in the subsequent parameter update process.

[0099] Furthermore, when updating the edge weights in the spatiotemporal graph corresponding to the time step in which the transaction occurred, the truncated gradient matrix of the time step in which the transaction occurred is input into the first fully connected layer and the second fully connected layer; the first fully connected layer maps the truncated gradient matrix to the update amount of the temporal connection edge weights, and the second fully connected layer maps the truncated gradient matrix to the update amount of the physical connection edge weights; the update amount of the temporal connection edge weights is added to the initial temporal connection edge weights of the spatiotemporal graph, and the update amount of the physical connection edge weights is added to the initial physical connection edge weights of the spatiotemporal graph, and the result of the sum is used as the edge weights of the spatiotemporal graph in the next round of forward propagation.

[0100]

[0101] in, This is the updated total adjacency matrix of the spatiotemporal graph; The total adjacency matrix of the spatiotemporal graph before the update; The preset learning rate; This is the weight matrix of the first fully connected layer, used to map the truncated gradient matrix to the temporal connection edge weight update amount; This is the weight matrix of the second fully connected layer, used to map the truncated gradient matrix to the physical connection edge weight update amount; This is to truncate the gradient matrix.

[0102] Specifically, both the first and second fully connected layers are single-layer linear transformation layers. Their input dimension matches the feature dimension of the truncated gradient matrix, and their output dimensions match the dimensions of the temporal and spatial adjacency matrices, respectively. This allows the truncated gradient matrix to be mapped to the corresponding edge weight update values. In this way, only the spatiotemporal graph edge weights of time steps related to transaction events are updated, while the edge weights of time steps unrelated to transactions remain unchanged, thus strengthening the binding relationship between actions and transaction events at the model parameter update level.

[0103] Table 4 Gradient Truncation and Edge Weight Update Rule Configuration Table

[0104]

[0105] Table 4 defines the execution rules for gradient truncation and the boundary conditions for edge weight updates during backpropagation. It clarifies the gradient processing methods, edge weight update ranges, and corresponding learning rate configurations for different types of time steps, ensuring that the spatiotemporal graph edge weights of time steps related only to transaction events are effectively updated, while the parameters of time steps without transaction association remain unchanged. This continuously strengthens the temporal binding relationship between operational actions and transaction events during model training, reducing the interference of non-business behavior features on model training.

[0106] Furthermore, during the inference phase of the model, there is no need to perform backpropagation and gradient truncation; only the forward propagation process is performed. The monitoring video stream and transaction flow data to be evaluated are input into the system, and the system sequentially performs skeleton node extraction, spatiotemporal graph construction, spatial behavior feature extraction, transaction event vector construction, transaction context feature extraction, cross-attention alignment and fusion, and fully connected layer mapping, and outputs the performance score of the operations personnel.

[0107] This embodiment achieves precise alignment and deep fusion of transaction context features and spatial behavioral features through linear transformation, similarity calculation, weighted fusion, and residual connection design in the cross-attention alignment module. This allows the model to focus on the specific action features that trigger transactions. By performing gradient truncation during backpropagation, the influence of gradients from time steps unrelated to transactions on parameter updates is shielded. Only the spatiotemporal graph edge weights corresponding to the transaction occurrence time step are updated. This further strengthens the causal correspondence between operational personnel actions and transaction results at the model parameter update level, reducing the interference of non-business behavioral features on performance evaluation results.

Claims

1. A deep learning-based performance evaluation system for store operations personnel, characterized in that: This includes the video capture front-end and the transaction log receiving interface; The video acquisition front end extracts the skeleton node sequence of the operators and constructs a spatiotemporal graph, wherein the spatiotemporal graph has skeleton nodes as vertices and physical distance and relative motion speed between nodes as edge weights. The spatiotemporal graph input graph convolutional network extracts spatial behavioral features; The transaction flow receiving interface extracts the transaction occurrence time, product category and processing time to construct a transaction event vector, and the transaction event vector is input into a time series Transformer to extract transaction context features; The system also includes a cross-attention alignment module, which uses the timestamp of the transaction event vector as the query key to perform retrieval on the time dimension of the spatial behavior features, calculates the cross-modal similarity matrix, and outputs a joint feature vector that fuses the operation action and the transaction result. The joint feature vector is input into the fully connected layer to output a performance score.

2. The deep learning-based performance evaluation system for store operations personnel according to claim 1, characterized in that, The video acquisition front end uses the OpenPose algorithm to extract the coordinates of the skeletal key points of the operators to form a skeleton node sequence, and uses forward kinematics interpolation to complete the key points that are occluded. When constructing the spatiotemporal graph, skeleton nodes at the same time are connected by physical connection edges, and the same skeleton nodes at adjacent times are connected by temporal connection edges. When calculating the weight of physical connection edges, the Euclidean distance between adjacent skeleton nodes is extracted and divided by the preset standard skeleton length ratio as the spatial weight. When calculating the weight of the temporal connection edge, the difference between the displacement vector magnitude of the same skeleton node at adjacent time points and the preset velocity threshold is extracted as the temporal weight. The spatial weight and the temporal weight are added together to obtain the edge weight of the spatiotemporal graph.

3. The deep learning-based performance evaluation system for store operations personnel according to claim 1, characterized in that, The graph convolutional network contains three graph convolutional layers. The first graph convolutional layer extracts local limb features of adjacent skeleton nodes, the second graph convolutional layer extracts cross-limb trunk coordination features, and the third graph convolutional layer extracts global body posture features. Each graph convolutional layer is followed by a batch normalization layer and a modified linear unit activation function. The output of the third graph convolutional layer is flattened and processed by a one-dimensional convolutional layer with a kernel size of 3. The spatial dimension and channel dimension are reorganized to obtain a spatial behavior feature matrix. The row dimension of the spatial behavior feature matrix corresponds to the time step, and the column dimension corresponds to the behavior feature channel.

4. The deep learning-based performance evaluation system for store operations personnel according to claim 1, characterized in that, The transaction flow receiving interface converts the transaction occurrence time into a relative timestamp, the product category into a one-hot encoded vector, and the processing time into a normalized scalar. The relative timestamp, the one-hot encoded vector, and the normalized scalar are concatenated to obtain the transaction event vector. The temporal Transformer includes a position encoder and a multi-head self-attention layer. The position encoder uses sinusoidal position encoding to inject the transaction event vector into the temporal position information. The multi-head self-attention layer sets up 8 attention heads and maps the transaction event vector into a query matrix, a key matrix, and a value matrix through linear mapping, and outputs transaction context features.

5. The deep learning-based performance evaluation system for store operations personnel according to claim 1, characterized in that, The cross-attention alignment module transforms the transaction context features through the first linear layer to generate a query matrix, and transforms the spatial behavior features through the second and third linear layers to generate a key matrix and a value matrix, respectively. Perform matrix multiplication on the query matrix and the transposed key matrix, divide by the square root of the key matrix dimension, and input the result into the softmax function to obtain the cross-modal similarity matrix. Preliminary fusion features are obtained by performing matrix multiplication on the cross-modal similarity matrix and the value matrix; The preliminary fused features are added element-wise to the query matrix after the fourth linear layer transformation to achieve residual connection. The residual connection result is input to the normalization layer and the joint feature vector is output.

6. The deep learning-based performance evaluation system for store operations personnel according to claim 1, characterized in that, After calculating the loss function between the performance score output by the fully connected layer and the true performance label, the backpropagation process is performed. During backpropagation, the gradient matrix of the loss function with respect to the timestamp dimension of the transaction event vector is extracted. The gradient values ​​in the gradient matrix corresponding to time steps where no transaction occurred are set to zero, while the gradient values ​​corresponding to time steps where a transaction occurred are retained to form a truncated gradient matrix. After multiplying the truncated gradient matrix by a preset learning rate, update the weights of the temporal and physical connections in the spatiotemporal graph corresponding to the time steps in which the transaction occurred.

7. The deep learning-based performance evaluation system for store operations personnel according to claim 2, characterized in that, When performing forward kinematic interpolation completion on occluded keypoints, extract the coordinates of the parent node and child node of the occluded keypoints, and calculate the bone vector pointing from the parent node to the child node. Obtain the proportion of the bone length and the deflection angle of the occluded keypoint relative to the parent node in the standard pose; Starting from the coordinates of the parent node, the bone vector is rotated and scaled according to the bone length ratio and the deflection angle to calculate the predicted coordinates of the occluded key points. The predicted coordinates are used to replace the original coordinates of the occluded key points in the skeleton node sequence and output to the spatiotemporal graph construction process.

8. The deep learning-based performance evaluation system for store operations personnel according to claim 3, characterized in that, After the output of the third graph convolutional layer is flattened and input into the one-dimensional convolutional layer, the output feature map of the one-dimensional convolutional layer is input into the max pooling layer. The max pooling layer sets the pooling window size to 2 and the stride to 2, and downsamples the temporal dimension of the output feature map. The downsampled output feature map is input into a one-dimensional dilated convolutional layer. The one-dimensional dilated convolutional layer is set with a dilation rate of 2 and a kernel size of 3. Long-distance temporal dependency features are extracted from the downsampled output feature map, and the long-distance temporal dependency features are reorganized into a spatial behavior feature matrix.

9. The deep learning-based performance evaluation system for store operations personnel according to claim 4, characterized in that, The multi-head self-attention layer introduces a time interval mask matrix, the dimension of which is consistent with the sequence length of the transaction event vector; Calculate the relative time difference between two adjacent vectors in the transaction event vector. When the relative time difference is greater than the preset time interval threshold, set the element at the corresponding row and column position in the time interval mask matrix to negative infinity. When the relative time difference is less than or equal to the preset time interval threshold, set the element at the corresponding position to 0. The product of the time interval mask matrix, the query matrix, and the key matrix is ​​added element by element and then input into the softmax function.

10. The deep learning-based performance evaluation system for store operations personnel according to claim 6, characterized in that, When updating the edge weights in the spatiotemporal graph corresponding to the time step in which the transaction occurred, the truncated gradient matrix of the time step in which the transaction occurred is input into the first fully connected layer and the second fully connected layer. The first fully connected layer maps the truncated gradient matrix to the temporal connection edge weight update amount, and the second fully connected layer maps the truncated gradient matrix to the physical connection edge weight update amount. The updated weights of the temporal connection edges are added to the initial weights of the temporal connection edges in the spatiotemporal graph, and the updated weights of the physical connection edges are added to the initial weights of the physical connection edges in the spatiotemporal graph. The sum is used as the edge weights of the spatiotemporal graph in the next round of forward propagation.

Images