Cloud duty customer behavior identification method based on deep learning

Abstract

The invention discloses a cloud guarding customer behavior identification method based on deep learning, and the method comprises the following steps: S1, collecting and processing a monitoring video stream, and forming a continuous frame sequence; s2, inputting an action proposal network, and obtaining candidate time periods and time masks; s3, constructing a local attention map, and highlighting a key frame region; s4, fusing the attention map and the original frame to generate an enhanced frame sequence; s5, inputting the enhanced frame into an improved SlowFast dual-path network, and extracting short-time and long-time features; s6, fusing the features and the mask information to generate customer behavior representation vectors; and S7, outputting a behavior category and a time positioning result through a behavior classification module. According to the invention, the customer behavior identification accuracy in an unattended environment can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and computer vision technology, and particularly relates to a cloud value-keeping customer behavior recognition method based on deep learning. BACKGROUND

[0002] With the continuous progress of artificial intelligence technology, video monitoring is widely used in unattended scenes such as retail and security. As an important branch of video intelligent analysis, customer behavior recognition can assist in realizing automatic operation management and abnormal behavior early warning, and is gradually attracting industry attention. However, in actual deployment, customer behavior has characteristics such as small action, variable scene, and different time span, which puts higher requirements on the accuracy and robustness of the recognition system.

[0003] In the prior art, the mainstream customer behavior recognition method usually relies on convolutional neural networks or recurrent neural networks to extract features and classify behaviors from continuous video frames. These methods have the following shortcomings in actual application: Difficulty in extracting key periods: traditional methods lack effective action proposal mechanisms and cannot accurately determine the time period when customer behavior occurs, resulting in too wide recognition range or missing important actions. Insufficient attention to behavior areas: most existing technologies process the entire frame image and lack spatial attention modeling for behavior-related areas, making it difficult to highlight the key areas where behavior occurs. Limited spatio-temporal feature processing capability: most models use a single path structure for feature extraction, making it difficult to capture both short-term dynamic changes and long-term context information, affecting recognition stability. Rough fusion strategy: in multi-channel or multi-path information fusion, existing methods generally rely on simple concatenation or averaging operations and fail to introduce a guiding mechanism to optimize feature fusion results. Low behavior positioning accuracy: due to the lack of fine-grained time mask to assist classification and discrimination, existing systems can only output behavior categories and cannot achieve accurate time positioning.

[0004] Therefore, how to provide a cloud value-keeping customer behavior recognition method based on deep learning is a problem that those skilled in the art need to solve. SUMMARY

[0005] One object of the present application is to provide a cloud value-keeping customer behavior recognition method based on deep learning. The present application fully combines action proposal networks, local attention mechanisms, and improved SlowFast dual-path structures, and describes in detail the recognition process of spatio-temporal feature extraction, accurate positioning, and classification of customer behavior in unattended environments. The present application has the advantages of high recognition accuracy, accurate time positioning, and adaptability to complex behavior scenarios.

[0006] According to the cloud value-keeping customer behavior recognition method based on deep learning of the present application, the following steps are included:

[0007] S1, acquire a continuous video stream collected by a monitoring camera, perform frame-level processing on the continuous video stream, and form a time-sequentially continuous video frame sequence;

[0008] S2, input the time-sequentially continuous video frame sequence into a motion proposal network, and output candidate time period information and a time mask corresponding to the candidate time period from the motion proposal network;

[0009] S3, match the time mask with the time-sequentially continuous video frame sequence, and generate a local attention map corresponding to the candidate time period;

[0010] S4, fuse the local attention map with the time-sequentially continuous video frame sequence, and obtain a video frame sequence enhanced for the candidate time period;

[0011] S5, input the video frame sequence enhanced for the candidate time period into an improved SlowFast dual-path action recognition network, process short-time features in a fast path and long-time features in a slow path of the improved SlowFast dual-path action recognition network, and synchronously receive the local attention map as a guide signal;

[0012] S6, fuse output features of the fast path and the slow path, and generate a customer behavior representation vector in combination with the time mask and the local attention map;

[0013] S7, input the customer behavior representation vector into a behavior classification module, and output a customer behavior category prediction result from the behavior classification module, to output a recognition positioning result of a customer behavior category and corresponding start and end time in combination with the candidate time period.

[0014] The application proposes a cloud value guard customer behavior recognition method based on deep learning, and a complete recognition process from video acquisition, time period proposal, spatial attention modeling, spatio-temporal feature extraction to behavior recognition and time positioning is constructed, effectively solving the problems of inaccurate behavior time period positioning, non-centralized regional response and insufficient feature expression in traditional customer behavior recognition. The method generates a candidate time period and a time mask through a motion proposal network, focuses on a key region by using the mask to construct a local attention map, and generates an enhanced frame sequence by fusing original video frames, guides an improved SlowFast dual-path network to process short-time actions and long-time behavior patterns respectively, generates a highly representative behavior vector in combination with attention and mask information, and finally realizes accurate classification and start and end time positioning of customer behavior. The method has the advantages of high recognition accuracy, acute spatial response and accurate time positioning, is suitable for unmanned retail, intelligent security and other cloud value guard scenes, and has good practical application prospect and deployment value.

[0015] Optionally, the S1 specifically includes: dividing a continuous video stream collected by a monitoring camera according to a fixed time interval Δt to generate a frame sequence {F1, F2,..., F n}, wherein n is the number of frames, and Δt satisfies f is a sampling frame rate; after time sequence ordering of the frame sequence, image normalization processing and size adjustment are performed, unified to a preset resolution WxH, and the processed video frame sequence is arranged in time sequence to form a tensor input structure, and a time-continuous video frame sequence T is constructed, satisfying wherein C is the number of channels.

[0016] In the original video frame preprocessing stage, the present application divides the continuous video stream by frame at a fixed time interval Δt, and uniformly performs image normalization and size adjustment operations, ensuring that each frame of image has consistent time sequence and spatial scale, thereby constructing a high-quality video frame input sequence satisfying the tensor structure The method effectively solves the problems of unstable frame rate, inconsistent image resolution and chaotic input data structure in the traditional video processing process.

[0017] Optionally, the S2 specifically comprises:

[0018] S21, the action proposal network is composed of the following modules:

[0019] S211, a three-dimensional convolution feature extraction module, composed of at least two three-dimensional convolution units, each three-dimensional convolution unit including a three-dimensional convolution layer, a normalization layer and an activation function layer, the convolution operation extracting features in the time dimension and the spatial dimension;

[0020] S212, a time compression module, adopting a one-dimensional pooling structure, compressing the convolution feature sequence length in the time dimension, and outputting a frame number lower than the input frame number;

[0021] S213, a context modeling module, composed of a bidirectional gated recurrent unit, receiving the time-compressed feature sequence, and outputting a frame-level feature representation containing context dependence;

[0022] S214, a proposal segment generation module, including a time sequence sliding window mechanism and a plurality of linear transformation modules, outputting classification confidence and start-stop boundary regression values of each candidate time segment;

[0023] S215, a time mask mapping module, converting the proposal segment boundary information into a frame-level binary sequence, representing the corresponding frame position of the candidate action in the time dimension;

[0024] S22, inputting the time-continuous video frame sequence into the three-dimensional convolution feature extraction module to extract an initial spatio-temporal feature tensor F1;

[0025] S23, inputting the feature tensor F1 into the time compression processing module, performing a maximum pooling operation along the time dimension to obtain a compressed feature tensor F2;

[0026] S24, expand F2 into a time sequence and input into a context modeling module, output a context feature sequence: S={s1, s2,...,s m}, where m=n / 2, each represents the semantic state vector of frame i after splicing in the forward and backward time directions;

[0027] S25, in the proposal segment generation module, the sequence S is executed with a window length w j , each segment feature is input into a full connection network after one-dimensional convolution, and the confidence σ j of the proposal segment is output. s , Δ e , the proposal score is calculated as:

[0028] score(a j )=σ j ·exp(-α·|Δ e -Δ s |);

[0029] Wherein, σ j represents the classification confidence of the proposal segment a j , Δ s , Δ e are the regression offset values of the start frame and the end frame of the proposal segment respectively, and α is a hyperparameter for controlling the decay rate of the score.

[0030] S26, map the start and end frame numbers of each proposal segment to a binary time mask vector M j with a length of n, which is defined as:

[0031]

[0032] Wherein, t is the time frame index, [s j , e j ] represents the frame index interval of the jth proposal segment.

[0033] S27, perform a bitwise logical or operation on all time mask vectors M j , and integrate into the final output time mask M, which satisfies: Wherein k is the number of candidate proposal segments, and the final output time mask M∈{0,1} n .

[0034] The application establishes a complete process from time sequence feature extraction to candidate time period positioning and frame-level mask generation by constructing an action proposal network integrating three-dimensional convolution, time compression, context modeling and candidate segment generation, effectively solving the problems of difficult accurate determination of behavior start and end, insufficient context understanding and low time positioning precision in traditional action detection methods. The three-dimensional convolution is used to extract inter-frame dynamic features in the space-time dimension, the time pooling compression is used to reduce the calculation redundancy, and the bidirectional gate recurrent unit is introduced to model the semantic dependency relationship before and after the behavior sequence, which significantly enhances the expression ability of the network to the behavior continuity and boundary transition. The action candidate segment is generated based on the sliding window mechanism and the boundary regression strategy, and the frame-level binary time mask is formed by combining the mask mapping operation, which provides the time focusing mechanism for subsequent attention map construction and behavior recognition.

[0035] Optionally, the S3 specifically comprises:

[0036] S31, obtaining a time sequence continuous video frame sequence formed after frame-level processing from a video stream collected by a monitoring camera;

[0037] S32, obtaining a time mask vector M consistent with the frame number of the video frame sequence, the time mask vector M being a one-dimensional binary sequence with a length of n, wherein the value of each element is 1, indicating that the corresponding frame belongs to the candidate time period, and the value of each element is 0, indicating that the corresponding frame does not belong to the candidate time period;

[0038] S33, performing a matching operation on each frame in the video frame sequence according to the value of each position in the time mask vector M, extracting all frames with a mask value of 1 to form a candidate frame subsequence, and the frame number of the candidate frame subsequence is not greater than n;

[0039] S34, performing a point-by-point accumulation operation on all frames in the candidate frame subsequence along the time dimension, keeping the spatial size and the number of channels unchanged, and generating a preliminary attention feature map, wherein the attention feature map represents the cumulative response degree of the candidate time period in the spatial position;

[0040] S35, performing mean normalization processing on the preliminary attention feature map in the channel dimension to obtain a two-dimensional spatial attention map, wherein the value of each spatial position in the spatial attention map is located in the interval of 0 to 1, and the value represents the relative response intensity of the spatial position in the candidate time period;

[0041] S36, copying and extending the two-dimensional spatial attention map in the time dimension to n frames to form a local attention map sequence consistent with the shape of the original video frame sequence, and the local attention map sequence is one-to-one corresponding to the original video frame sequence.

[0042] The application constructs a local attention map generation mechanism corresponding to a candidate time period by matching a video frame with a time mask, effectively solving the problem of non-concentrated response to a key behavior region in the existing behavior recognition method. In a specific implementation, the system extracts a candidate frame subsequence according to a time mask vector, and forms a preliminary attention feature map reflecting behavior intensity distribution by frame-by-frame accumulation, and then performs normalization processing in the channel dimension to generate a two-dimensional spatial attention map between 0 and 1, realizing explicit expression of response to a key spatial region. By extending the attention map to the complete frame number and corresponding to the original video frame sequence one by one, a precise mapping in the time-space dimension is established, providing fine attention region information for subsequent feature fusion and behavior recognition. The method improves the focusing ability of the system on the high-response action region, enhances the discriminability and positioning accuracy of behavior recognition, and at the same time maintains the integrity of the frame structure.

[0043] Optionally, the S4 specifically comprises:

[0044] S41, a time-sequentially continuous video frame sequence is acquired, the video frame sequence is composed of n frames, each frame is a tensor with a spatial size WxH and a channel number C, and the frame numbers are in a time sequence {1, 2, …, n};

[0045] S42, a local attention map sequence with the same number of frames as the video frame sequence is acquired, each image in the local attention map sequence is consistent with the corresponding numbered video frame in the spatial dimension, and the value of each pixel position in the image is in the range of [0, 1];

[0046] S43, the video frame sequence and the local attention map sequence are aligned according to the frame numbers, each frame of the video frame is matched with the same numbered local attention map in the two-dimensional spatial coordinates to form frame-level paired data;

[0047] S44, for each frame of paired data, the channel pixel value of the video frame tensor and the corresponding position value of the local attention map are extracted at each spatial coordinate position, and point-by-point multiplication calculation is performed in the channel order to generate a fusion pixel value tensor that is invariant in the channel;

[0048] S45, all processed video frame fusion tensors are combined into a tensor sequence with n frames in the original time sequence, and each frame in the sequence is consistent with the original video frame in the spatial size and the channel structure;

[0049] S46, the tensor sequence is output as a fusion result, constituting video frame enhancement data for a candidate time period.

[0050] The application establishes a fine video frame enhancement method by designing a frame-by-frame fusion mechanism of video frames and local attention maps, effectively solving the problems of insufficient response to key areas and strong background interference in existing behavior recognition systems. In specific implementation, the system accurately aligns the local attention map and the original video frame in frame number and spatial coordinates, and performs point-by-point multiplication at each spatial position in channel order to realize the saliency enhancement of behavior-sensitive areas and the weakening of background areas in the original frame. The fusion operation maintains the original image channel structure and time sequence while enhancing the feature expression ability of the behavior occurrence area, thereby generating a frame-level enhanced tensor sequence with higher discriminability.

[0051] Optionally, the S5 specifically includes:

[0052] S51, input the candidate time period enhanced video frame sequence into the improved SlowFast dual-path action recognition network, respectively into the fast path sampling branch and the slow path sampling branch in the sampling module, the fast path sampling interval is r1, the slow path is r2, and r1

[0053] S52, input the fast path sampling frame into the fast path feature extraction module, and sequentially process through the three-dimensional convolution structure unit, output the fast path time sequence feature tensor F f (t);

[0054] S53, input the slow path sampling frame into the slow path feature extraction module, gradually obtain the original slow path feature tensor F s (t), and obtain the context enhanced feature tensor by combining the average value of the previous frames in the history window T

[0055] S54, input the fast path feature tensor into the action prior trigger module, construct the adjacent frame channel difference f (t)=F f (t)-F f (t-1), and generate the action prior trigger vector by the activation judgment logic:

[0056]

[0057] Wherein, θ, δ are set threshold parameters;

[0058] S55, input the action prior trigger vector G f (t) into the channel gating module, apply gating control to the context enhanced feature , execute the time step channel suppression operation, and output the gated slow path tensor

[0059] S56, respectively mapping the sequence of local attention maps to and performing a channel-wise point-wise multiplication operation on the corresponding time step, outputting a fast-path tensor F' f (t) and a slow-path tensor F" s (t).

[0060] The application realizes multi-time domain feature extraction and dynamic response control of customer behavior by designing an improved SlowFast dual-path action recognition network. In the specific method, the fast path and the slow path sample frame sequences of different frequencies respectively, extract short-time dynamic features and long-time context features, and introduce a frame-level context buffer in the slow path to enhance the expression of time sequence semantics. The action prior trigger module is innovatively introduced, the activation vector is generated through channel difference and threshold judgment, and the channel gating module is controlled to implement dynamic suppression of slow path features, effectively highlighting the key frame response area. The local attention map is combined to guide the fusion of fast and slow path tensors at the channel level, enhancing the spatial specificity and action sensitivity of feature expression, and significantly improving the recognition accuracy and discrimination ability of the system in complex behavior scenes.

[0061] Optionally, the improved SlowFast dual-path action recognition network specifically comprises:

[0062] a sampling module, the sampling module comprising a fast-path sampling branch and a slow-path sampling branch, the fast-path sampling branch sampling the original video frame sequence at a first frame interval, the slow-path sampling branch sampling at a second frame interval, the first frame interval being smaller than the second frame interval, and outputting a fast-path input sequence and a slow-path input sequence;

[0063] a fast-path feature extraction module, the fast-path feature extraction module comprising a plurality of three-dimensional convolution structure units arranged in series, each three-dimensional convolution structure unit comprising a three-dimensional convolution layer, a normalization layer and an activation function layer in sequence, the structure unit performing frame-by-frame processing on the fast-path input sequence to extract a short-time action feature tensor;

[0064] a slow-path feature extraction module, the slow-path feature extraction module comprising a plurality of three-dimensional convolution structure units arranged in series, each structure unit comprising a three-dimensional convolution layer, a normalization layer, an activation function layer and a frame-level context buffer unit in sequence, the frame-level context buffer unit being used to introduce the average value of historical frame features within a fixed time window and perform a point-by-point multiplication operation with the current frame features to generate a context-enhanced feature tensor;

[0065] An action prior trigger module is connected with the output of the fast-path feature extraction module, and includes a spatial difference calculation unit and an activation judgment logic unit, the spatial difference calculation unit calculates the channel-by-channel difference of the feature tensor between adjacent frames, and the activation judgment logic unit generates an action prior trigger vector based on the comparison between the difference value and the set threshold value;

[0066] A channel gating module is used for receiving the action prior trigger vector, and performing channel-selective masking or reservation on the slow-path feature tensor in the time step dimension, and outputs the gated slow-path feature tensor.

[0067] The improved SlowFast dual-path action recognition network is constructed, the short-time change and long-time context features of the customer behavior are finely modeled, and the response sensitivity and discrimination accuracy of the behavior recognition system are effectively improved. The network adopts a dual-sampling branch to perform high-frequency and low-frequency frame extraction on the original video frame sequence, the fast path extracts short-time dynamic features through a multi-layer three-dimensional convolution structure unit, and the slow path introduces historical average features into each frame feature by combining a frame-level context buffer unit, so that the context enhancement is realized. On this basis, the network introduces an action prior trigger module, calculates the channel difference of adjacent frames of the fast-path output, and performs threshold judgment to generate an action trigger vector, and guides a channel gating module to perform channel-selective activation or inhibition on the slow path in the time dimension. The structure realizes the enhancement of the key frame feature response and the inhibition of the redundant frame feature, effectively focuses on the customer behavior occurrence area, improves the real-time performance, accuracy and stability of the recognition, and is particularly suitable for intelligent monitoring systems in complex time sequence behavior recognition scenes.

[0068] Optionally, the S6 specifically includes:

[0069] S61, acquiring the fast-path feature tensor and the slow-path feature tensor output by the improved SlowFast dual-path action recognition network, the time dimensions of the fast-path feature tensor and the slow-path feature tensor are different, and the channel numbers are C f and C s , respectively.

[0070] S62, according to the time mask vector, projecting or interpolating the mask to the time dimensions of the fast path and the slow path to obtain a fast-path mask sequence and a slow-path mask sequence;

[0071] S63, in the fast-path and slow-path time dimensions, respectively, taking the mask sequence as a weight, performing weighted averaging on the respective feature tensors along the time dimension to obtain fast-path aggregated features and slow-path aggregated features, each feature being a one-dimensional channel vector.

[0072] S64, accumulate all local attention maps of the candidate time period in time sequence, divide by the number of valid frames to obtain an average attention map, and then transform the average attention map into a fixed-length vector through convolution or full connection operation, denoted as F a ;

[0073] S65, the fast path aggregation feature, the slow path aggregation feature and the attention feature vector are spliced in the channel dimension to obtain a customer behavior representation vector V, wherein the length of V is equal to the sum of the channel numbers of the three, and the output is used as an input feature of subsequent behavior classification.

[0074] The application designs a customer behavior representation vector generation method based on multi-source information fusion, constructs a unified fusion mechanism of fast path features, slow path features and spatial attention features, and effectively improves the expression integrity and discrimination accuracy of behavior recognition. In specific implementation, the system first weights and aggregates the time dimension of the time sequence feature tensor output by the fast and slow paths according to the time mask sequence, and extracts one-dimensional channel vectors of short-time dynamic features and long-time context features respectively; at the same time, the local attention map in the candidate time period is accumulated and averaged in time sequence, the response features of the key spatial region are extracted, and the fixed-length vector is transformed through convolution or full connection. Finally, the three types of features are spliced in the channel dimension to form a unified behavior representation vector, realizing the deep fusion of time domain and spatial information. This method effectively solves the problems of single information dimension and lack of regional guidance in traditional representation methods, significantly enhances the sensitivity of feature expression to behavior category differences, and provides a more discriminative input basis for subsequent classifiers.

[0075] Optionally, the S7 specifically comprises:

[0076] S71, obtaining a customer behavior representation vector, the customer behavior representation vector being a fixed-length one-dimensional channel feature tensor containing feature components spliced from multiple source channels;

[0077] S72, inputting the customer behavior representation vector into a behavior classification module, the behavior classification module including an input layer, two full connection layers and an output layer, each layer being connected with a batch normalization layer and a nonlinear activation unit in turn, and the output dimension being a preset category number K;

[0078] S73, calculating the confidence probability of each customer behavior category in the output layer to obtain a behavior prediction probability vector P = [p1, p2,..., p K ], wherein each component p k represents the probability value of predicting belonging to category k, satisfying

[0079] S74, selecting the category index k *Output the corresponding behavior category name in the behavior label table as the customer behavior prediction category number;

[0080] S75, in the candidate time period set, select the time period with the highest confidence score * The time period with the highest confidence score, whose frame index is the start frame number s * And the end frame number e * The confidence score is calculated according to the similarity measure value between the feature vector in the candidate segment and the behavior representation vector;

[0081] S76, output the customer behavior prediction result and the time positioning result triplet (k * , s * , e * ) as the customer behavior recognition and positioning result.

[0082] The application realizes the unified output of customer behavior category recognition and start-end time period positioning by constructing a behavior representation vector fused with multi-source features and designing a refined classification and time positioning module. In the specific method, the representation vector fused by the fast path, the slow path and the local attention map is input into the classification module composed of multiple full connection networks, and after batch normalization and nonlinear activation processing, the prediction probability of each behavior is output. The system determines the customer behavior type according to the category index corresponding to the maximum probability value, and selects the time period with the highest confidence degree of matching the category by calculating the similarity between the candidate time period feature and the representation vector, to realize the accurate time positioning of the behavior event. This method solves the problem of separation of behavior classification and positioning in existing recognition systems, and improves the automatic response ability and behavior audit accuracy of the system in the actual cloud value guard scene.

[0083] The beneficial effects of the application are:

[0084] Unlike traditional customer behavior recognition methods that rely on static rules or single-path network structures, the application introduces an action proposal network to dynamically generate candidate time periods based on video frame sequences, and combines time masks to accurately filter key behavior segments, significantly improving the accuracy and context relevance of behavior positioning. Cooperating with the local attention map construction mechanism, the active areas in the candidate frames are effectively highlighted, so that the system has the ability to focus on the key parts of the behavior, and the recognition and discrimination ability of the model to small actions and complex behaviors is enhanced.

[0085] The improved SlowFast double-path structure is adopted, short-time dynamic features are captured in the fast path, long-time context semantics are modeled in the slow path, a channel gating mechanism and an action prior trigger module are introduced, dynamic control of time-sensitive channels and reservation of important information are realized, and the effectiveness of spatiotemporal feature fusion is improved. The attention map serves as an explicit guidance signal throughout the feature extraction and fusion process, further enhancing the feature representation capability of the behavior highly relevant area.

[0086] In the behavior representation stage, the fast and slow path features and the average attention map are fused by time mask weighting to form a compact structure and rich semantic one-dimensional behavior vector, and the behavior classification module is combined to realize fine-grained behavior category prediction and time period positioning. The overall method has stronger time positioning ability, higher behavior discrimination accuracy and better model generalization performance, and is significantly better than the existing recognition scheme based on fixed window or overall feature mean processing. BRIEF DESCRIPTION OF DRAWINGS

[0087] The accompanying drawings are included to provide a further understanding of the application, and constitute a part of the specification, together with the embodiments of the application, to explain the application, and do not constitute a limitation on the application. In the drawings:

[0088] Fig. 1 A flowchart of a cloud value guard customer behavior recognition method based on deep learning is provided for the application;

[0089] Fig. 2 A structural diagram of the action proposal network is provided for the application;

[0090] Fig. 3 An improved SlowFast double-path action recognition network structure diagram is provided for the application. DETAILED DESCRIPTION

[0091] The application will now be described in further detail with reference to the drawings. These drawings are simplified schematic diagrams, and only illustrate the basic structure of the application in a schematic manner, and therefore only show the components related to the application.

[0092] REFERENCE Figs. 1-3 A cloud value guard customer behavior recognition method based on deep learning, comprising the following steps:

[0093] S1, acquiring a continuous video stream collected by a monitoring camera, performing frame-level processing on the continuous video stream, and forming a time-sequentially continuous video frame sequence;

[0094] S2, inputting the time-sequentially continuous video frame sequence into an action proposal network, and outputting candidate time period information and a time mask corresponding to the candidate time period from the action proposal network;

[0095] S3, matching the temporal mask with the temporally continuous video frame sequence to generate a local attention map corresponding to the candidate time period;

[0096] S4, fusing the local attention map with the temporally continuous video frame sequence to obtain a video frame sequence enhanced for the candidate time period;

[0097] S5. Input the enhanced video frame sequence for the candidate time period into the improved SlowFast dual-path action recognition network. The improved SlowFast dual-path action recognition network processes short-term features in the fast path and long-term features in the slow path, and simultaneously receives the local attention map as a guidance signal.

[0098] S6. Fuse the output features of the fast path and the slow path, and generate a customer behavior representation vector by combining the temporal mask and the local attention map.

[0099] S7. Input the customer behavior representation vector into the behavior classification module. The behavior classification module outputs the customer behavior category prediction result and outputs the identification and positioning result of the customer behavior category and the corresponding start and end time in combination with the candidate time period.

[0100] In this embodiment, the S1 specifically includes: dividing the continuous video stream collected by the surveillance camera into frame levels according to a fixed time interval Δt, generating a frame sequence {F1, F2, ..., F n}, where n is the number of frames, and Δt satisfies f is the sampling frame rate; after the frame sequence is time-sequentially sorted, image normalization and size adjustment are performed to unify it to the preset resolution W×H, and the processed video frame sequence is organized into a tensor input structure in time order to construct a time-sequential continuous video frame sequence T, satisfying Where C is the number of channels.

[0101] In this embodiment, S2 specifically includes:

[0102] S21. The action proposal network consists of the following modules:

[0103] S211, a 3D convolution feature extraction module, consisting of at least two layers of 3D convolution units, each of which includes a 3D convolution layer, a normalization layer, and an activation function layer. The convolution operation extracts features in the time dimension and the spatial dimension;

[0104] S212, a time compression module, uses a one-dimensional pooling structure to compress the length of the convolution feature sequence in the time dimension, and the number of output frames is lower than the number of input frames;

[0105] S213, a context modeling module, composed of a bidirectional gated recurrent unit, receives the time-compressed feature sequence and outputs a frame-level feature representation containing context dependencies;

[0106] S214, proposal segment generation module, including a time sliding window mechanism and a plurality of linear transformation modules, outputting classification confidence and start-end boundary regression value of each candidate time segment;

[0107] S215, time mask mapping module, converting proposal segment boundary information into frame-level binary sequence, indicating corresponding frame position of candidate action in time dimension;

[0108] S22, inputting time-continuous video frame sequence into three-dimensional convolution feature extraction module to extract initial space-time feature tensor F1;

[0109] S23, inputting feature tensor F1 into time compression processing module to perform maximum pooling operation along time dimension to obtain compressed feature tensor F2;

[0110] S24, expanding F2 into time sequence and inputting into context modeling module to output context feature sequence S={s1, s2,..., sn} where m=n / 2, each m represents semantic state vector of frame i after splicing in forward and backward time directions;

[0111] S25, in proposal segment generation module, performing sliding extraction operation on sequence S with window length w j , each segment of feature is inputted into full connection network after one-dimensional convolution to output classification confidence σ j and boundary offset value Δ s , Δ e of proposal segment, and calculating proposal score as:

[0112] score(a j )=σ j ·exp(-α·|Δ e -Δ s |);

[0113] wherein, σ j represents classification confidence of proposal segment a j , Δ s , Δ e are regression offset values of start frame and end frame of proposal segment respectively, and α is hyperparameter for controlling score decay rate;

[0114] S26, mapping start-end frame number of each proposal segment into binary time mask vector M j with length n, which is defined as:

[0115]

[0116] wherein, t is time frame index, [s j ,e j ​represents the frame index interval of the jth proposed segment;

[0117] S27, all time mask vectors M j Performing a bit-by-bit logical OR operation, integrating into the final output time mask M, satisfying: where k is the number of candidate proposed segments, and the final output time mask M∈{0,1} n .

[0118] In this embodiment, the S3 specifically comprises:

[0119] S31, obtaining a time-sequentially continuous video frame sequence formed after frame-level processing from a video stream collected by a monitoring camera;

[0120] S32, obtaining a time mask vector M consistent with the frame number of the video frame sequence, the time mask vector M being a one-dimensional binary sequence with a length of n, wherein each element has a value of 1 indicating that the corresponding frame belongs to the candidate time period, and a value of 0 indicating that the corresponding frame does not belong to the candidate time period;

[0121] S33, according to the value of each position in the time mask vector M, performing a matching operation on each frame in the video frame sequence, extracting all frames with a mask value of 1 to form a candidate frame subsequence, and the frame number of the candidate frame subsequence is not greater than n;

[0122] S34, performing a point-by-point accumulation operation on all frames in the candidate frame subsequence along the time dimension, keeping the spatial size and the number of channels unchanged, to generate a preliminary attention feature map, the attention feature map representing the cumulative response degree in the spatial position within the candidate time period;

[0123] S35, performing mean normalization processing on the preliminary attention feature map in the channel dimension to obtain a two-dimensional spatial attention map, each spatial position in the spatial attention map having a value in the interval of 0 to 1, and the value representing the relative response intensity of the spatial position within the candidate time period;

[0124] S36, copying and extending the two-dimensional spatial attention map in the time dimension to n frames to form a local attention map sequence consistent with the shape of the original video frame sequence, and the local attention map sequence is in one-to-one correspondence with the original video frame sequence by frame.

[0125] In this embodiment, the S4 specifically comprises:

[0126] S41, obtaining a time-sequentially continuous video frame sequence, the video frame sequence being composed of n frames, each frame being a tensor with a spatial size of WxH and a channel number of C, and the frame numbers being in time order {1, 2, …, n};

[0127] S42, acquire a sequence of local attention maps consistent with the number of frames in the sequence of video frames, each image in the sequence of local attention maps is consistent with the corresponding numbered video frame in the spatial dimension, and the value range of each pixel position in the image is [0, 1];

[0128] S43, align the sequence of video frames with the sequence of local attention maps according to the frame number, each frame of the video frame is matched with the local attention map with the same number in the two-dimensional space coordinates to form frame-level paired data;

[0129] S44, for each frame of paired data, at each spatial coordinate position, extract the channel pixel value of the video frame tensor and the corresponding position value of the local attention map, and perform point-by-point multiplication calculation in the channel order to generate a fusion pixel value tensor that is invariant to the channel;

[0130] S45, combine all the processed video frame fusion tensors into a tensor sequence with n frames according to the original time sequence, and each frame in the sequence is consistent with the original video frame in terms of spatial size and channel structure;

[0131] S46, output the tensor sequence as the fusion result to form the video frame enhancement data for the candidate time period.

[0132] In this embodiment, S5 specifically includes:

[0133] S51, input the enhanced video frame sequence of the candidate time period into the improved SlowFast dual-path action recognition network, respectively into the fast path sampling branch and the slow path sampling branch in the sampling module, the fast path sampling interval is r1, the slow path is r2, and r1

[0134] S52, input the fast path sampling frame into the fast path feature extraction module, and process it through the three-dimensional convolution structure unit in turn to output the fast path time sequence feature tensor F f (t);

[0135] S53, input the slow path sampling frame into the slow path feature extraction module, gradually acquire the original slow path feature tensor F s (t), and obtain the context enhanced feature tensor by combining the average value of the previous frames in the history window T

[0136] S54, input the fast path feature tensor into the action prior trigger module, construct the adjacent frame channel difference ΔF f (t) = F f (t) - F f (t-1), and then generate the action prior trigger vector by the activation judgment logic:

[0137]

[0138] wherein θ, δ are set threshold parameters;

[0139] S55, the action prior trigger vector G f (t) input channel gating module, the context enhanced feature apply gating control, execute the time step by time step channel suppression operation, output the slow path tensor after gating

[0140] S56, the local attention map sequence is mapped to and corresponding time step, execute the channel level point by point multiplication operation, output the fast path tensor F' f (t) and the slow path tensor F" s (t).

[0141] In the embodiment, the improved SlowFast dual-path action recognition network specifically comprises:

[0142] a sampling module, the sampling module includes a fast path sampling branch and a slow path sampling branch, the fast path sampling branch samples the original video frame sequence at a first frame interval, and the slow path sampling branch samples at a second frame interval, the first frame interval is smaller than the second frame interval, and the fast path input sequence and the slow path input sequence are output;

[0143] a fast path feature extraction module, the fast path feature extraction module includes a plurality of three-dimensional convolution structure units arranged in series, each three-dimensional convolution structure unit includes a three-dimensional convolution layer, a normalization layer and an activation function layer in turn, the structure unit processes the fast path input sequence frame by frame to extract a short-time action feature tensor;

[0144] a slow path feature extraction module, the slow path feature extraction module includes a plurality of three-dimensional convolution structure units arranged in series, each structure unit includes a three-dimensional convolution layer, a normalization layer, an activation function layer and a frame-level context buffer unit in turn, the frame-level context buffer unit is used to introduce the average value of historical frame features in a fixed time window and perform point-by-point multiplication operation with the current frame features to generate a context enhanced feature tensor;

[0145] an action prior trigger module, the action prior trigger module is connected with the output of the fast path feature extraction module, and includes a spatial difference calculation unit and an activation judgment logic unit, the spatial difference calculation unit calculates the channel-by-channel difference of feature tensors between adjacent frames, and the activation judgment logic unit generates an action prior trigger vector based on the comparison between the difference value and the set threshold value;

[0146] The channel gating module is used to receive the action prior trigger vector, and perform channel selective shielding or retention on the slow path feature tensor in the time step dimension, and output the gated slow path feature tensor.

[0147] In this embodiment, S6 specifically includes:

[0148] S61. Obtain the fast path feature tensor and slow path feature tensor output by the improved SlowFast dual-path action recognition network. The time dimensions of the fast path feature tensor and the slow path feature tensor may be different, and the number of channels is C f and C s ;

[0149] S62. Project or interpolate the mask onto the time dimensions of the fast path and the slow path according to the time mask vector to obtain a fast path mask sequence and a slow path mask sequence;

[0150] S63. In the time dimension of the fast path and the slow path, use the mask sequence as the weight, and perform weighted averaging on the feature tensors along the time dimension to obtain the fast path aggregate feature and the slow path aggregate feature, each of which is a one-dimensional channel vector.

[0151] S64: Accumulate all local attention maps of the candidate time period in chronological order, and then divide it by the number of valid frames to obtain the average attention map. Then, transform the average attention map into a fixed-length vector through convolution or full connection operation, denoted as F a ;

[0152] S65. Concatenate the fast path aggregation features, slow path aggregation features, and attention feature vectors in the channel dimension to obtain the customer behavior representation vector V, where the length of V is equal to the sum of the number of channels of the three. The output is used as the input feature for subsequent behavior classification.

[0153] In this embodiment, the S7 specifically includes:

[0154] S71. Obtain a customer behavior representation vector, which is a fixed-length one-dimensional channel feature tensor and includes feature components formed by concatenating multiple source channels.

[0155] S72. Input the customer behavior representation vector into a behavior classification module. The behavior classification module includes an input layer, two fully connected layers, and an output layer. Each layer is sequentially connected to a batch normalization layer and a nonlinear activation unit. The output dimension is a preset number of categories K.

[0156] S73. Calculate the confidence probability of each customer behavior category in the output layer to obtain the behavior prediction probability vector P = [p1, p2, ..., p K ], where each component p k Indicates the probability value of the prediction belonging to category k, satisfying

[0157] S74. Select the category index k corresponding to the maximum probability value * As the customer behavior prediction category number, match the corresponding behavior category name in the behavior label table and output;

[0158] S75. Select the predicted category k from the candidate time period set. * The time segment with the highest confidence score, whose frame index is the starting frame number s * and the end frame number e * , the confidence score is calculated based on the similarity measure between the feature vector in the candidate segment and the behavior representation vector;

[0159] S76, output customer behavior prediction result and time positioning result triple (k * ,s * ,e * ), as the result of customer behavior identification and positioning.

[0160] Example 1:

[0161] To validate the performance of this invention, we deployed it in a certain brand's unmanned convenience store, open 24 hours a day, equipped with three high-definition cameras covering the entrance, main shelf area, and self-service checkout area. The store saw an average daily customer flow of approximately 250, with an average stay time of 4.5 minutes. Previously, the store primarily detected abnormal behavior through backstage video playback, which presented issues such as delayed recognition, heavy manual workload, and low anomaly recognition rates.

[0162] The method is integrated into an in-store edge smart box powered by the NVIDIA Jetson Xavier NX chip. This box also features the proposed behavior recognition system, which is integrated with a cloud server to provide inference optimization and data logging. The system workflow includes video stream acquisition, action proposal generation, local attention map construction, SlowFast dual-path recognition, and behavior classification output. The entire process is fully automated, requiring no human intervention.

[0163] For example, the behavior of repeatedly picking up and placing items without ultimately purchasing them is difficult to accurately identify in traditional systems, relying solely on manual post-analysis. With this new technology, the system can automatically capture the time points of a customer's interaction with the shelf in video, extract relevant time periods, and locate the customer's hand and body positions based on their attention areas. It can then further extract continuous features for identification and classification. If a customer repeatedly picks up and places items three or more times without completing a purchase, the system will label the action as "suspected abandoned purchase" and record it.

[0164] To objectively evaluate the performance of the method in practical application, the research team cooperated with the store manager to carry out a 21-day on-site test, and compared it with the YOLO+LSTM rule matching recognition system used by the convenience store before. During the test, the system analyzed a total of 19,750 video segments, covering the entire process of customer entry, standing, selection, queuing, payment, etc.

[0165] Table 1 compares the real performance data collected in the test

[0166]

[0167] As can be seen from Table 1, the method of the present application has significantly improved in various indicators compared with the original system. The recognition accuracy is improved by 14.2 percentage points, and the F1 score (a performance indicator that combines accuracy and recall) is improved to 0.89, meaning that the system's recognition of various behaviors is more stable and reliable. At the same time, the average behavior recognition delay is reduced from 9.3 seconds to 3.1 seconds, making the system basically have quasi-real-time analysis capability. The system can recognize 13 types of behaviors, including but not limited to: standing and staying, taking and putting goods, repeatedly trying to leave without settlement, long stay, non-payment departure, etc. high-risk behaviors, significantly enhancing the modeling capability of complex customer behavior.

[0168] In terms of operational effectiveness, the system's support for operational strategies also shows practical results. After the store placed cameras in front of certain beverage products to monitor them and identified a large number of "stand-by-take-and-put-back" behaviors, the store's operational staff adjusted the placement of the products and the discount information. After observing for a week, the sales of this type of product increased from an average of 8 bottles per day to 13 bottles, an increase of 62.5%. In addition, the system identified four customers who tried to leave without paying, and the management backend received an alarm message and intervened in time through the remote shouting function, successfully preventing possible property loss incidents.

[0169] Table 2 shows the statistical data of the recognized behavior types during the test period

[0170] Behavior type Recognition times Recognition accuracy (%) Average positioning error (frames) Normal browsing behavior 4,218 94.1 5.3 Taking and placing goods behavior 3,802 92.3 6.1 Queuing and waiting for payment behavior 2,176 90.8 6.7 Abnormal stay behavior 512 88.6 8.5 Unpaid departure attempt behavior 131 86.9 9.2

[0171] As shown in Table 2, the system has high accuracy in recognizing high-frequency regular behaviors, with "normal browsing" and "taking and putting goods" accuracy rates exceeding 92%, and positioning errors controlled within 6 frames. In high-risk behaviors related to safety, such as "abnormal stay" and "unpaid departure attempt", the system's accuracy rate is close to 90%, providing a reliable risk warning basis for management personnel.

[0172] The system also provides data access capability based on a cloud API interface, so that the headquarter management system can aggregate customer behavior data of multiple stores, and provide basic support for retail strategy decision. For example, the headquarter can perform horizontal analysis on data such as commodity preferences in different regions, average stay time in stores, and abnormal behavior density.

[0173] To sum up, the embodiment shows the deployment and running process of the application in a real unmanned convenience store scene. Through technical features such as dynamic proposal, spatial attention mechanism, SlowFast double-path feature fusion, and time mask positioning, the system effectively improves the accuracy, response efficiency, and risk prevention and control capability of behavior recognition, and the data performance is objective and verifiable. The verification results show that the application technology has high engineering practicability and commercial landing potential, and can be widely applied to intelligent video analysis tasks in the fields of retail, security, transportation hubs, and the like.

[0174] The above describes only the preferred specific embodiments of the application, but the protection scope of the application is not limited thereto, and any person skilled in the art can make equivalent replacements or changes within the technical scope disclosed by the application and according to the technical solutions and inventive concepts of the application, which should be covered within the protection scope of the application.

Claims

1. A cloud-based customer behavior recognition method based on deep learning, characterized in that: The steps include: S1. Obtain a continuous video stream captured by a surveillance camera and perform frame-level processing on the continuous video stream to form a temporally continuous video frame sequence; S2. Input the temporally continuous video frame sequence into the action proposal network, which outputs candidate time segment information and the time mask corresponding to the candidate time segment; S3, matching the temporal mask with the temporally continuous video frame sequence to generate a local attention map corresponding to the candidate time period; S4, fusing the local attention map with the temporally continuous video frame sequence to obtain a video frame sequence enhanced for the candidate time period; S5. Input the enhanced video frame sequence for the candidate time period into the improved SlowFast dual-path action recognition network. The improved SlowFast dual-path action recognition network processes short-term features in the fast path and long-term features in the slow path, and simultaneously receives the local attention map as a guidance signal. S6. Fuse the output features of the fast path and the slow path, and generate a customer behavior representation vector by combining the temporal mask and the local attention map. S7. Input the customer behavior representation vector into the behavior classification module. The behavior classification module outputs the customer behavior category prediction result and outputs the identification and positioning result of the customer behavior category and the corresponding start and end time in combination with the candidate time period.

2. The method for identifying customer behavior in a cloud-based environment based on deep learning according to claim 1, characterized in that: The S1 specifically includes: dividing the continuous video stream collected by the surveillance camera into frame levels according to a fixed time interval Δt, generating a frame sequence {F1, F2, ..., F n }, where n is the number of frames, and Δt satisfies f is the sampling frame rate; after the frame sequence is time-sequentially sorted, image normalization and size adjustment are performed to unify it to the preset resolution W×H, and the processed video frame sequence is organized into a tensor input structure in time order to construct a time-sequential continuous video frame sequence T, satisfying Where C is the number of channels.

3. The method for identifying customer behavior in a cloud-based environment based on deep learning according to claim 1, characterized in that: The S2 specifically includes: S21. The action proposal network consists of the following modules: S211, a 3D convolution feature extraction module, consisting of at least two layers of 3D convolution units, each of which includes a 3D convolution layer, a normalization layer, and an activation function layer. The convolution operation extracts features in the time dimension and the spatial dimension; S212, time compression module, uses a one-dimensional pooling structure to compress the length of the convolution feature sequence in the time dimension, and the number of output frames is lower than the number of input frames; S213, a context modeling module, composed of a bidirectional gated recurrent unit, receives the time-compressed feature sequence and outputs a frame-level feature representation containing context dependencies; S214, a proposed segment generation module, including a temporal sliding window mechanism and multiple linear transformation modules, outputs the classification confidence and start and end boundary regression values ​​of each candidate time segment; S215, a temporal mask mapping module converts the proposed segment boundary information into a frame-level binary sequence, representing the corresponding frame position of the candidate action in the temporal dimension; S22, inputting the temporally continuous video frame sequence into a three-dimensional convolutional feature extraction module to extract an initial spatiotemporal feature tensor F1; S23, inputting the feature tensor F1 into the time compression processing module, performing a maximum pooling operation along the time dimension, and obtaining a compressed feature tensor F2; S24, expand F2 into a time series and input it into the context modeling module, outputting the context feature sequence: S = {s1, s2, ..., s m }, where m = n / 2, each represents the semantic state vector of frame i after concatenation in the forward and backward time directions; S25. In the proposed segment generation module, the sequence S is processed with a window length w. j Perform sliding extraction operation, each segment feature is input into the fully connected network after one-dimensional convolution, and output the confidence σ of the proposed segment j Offset value Δ from the boundary s ,Δ e , calculate the proposal score as: score(a j )=s j ·exp(-α·|Δ e -D s |); Among them, σ j Indicates proposed paragraph a j The classification confidence, Δ s ,Δ e are the regression offset values ​​of the starting and ending frames of the proposed segment, respectively, and α is a hyperparameter that controls the score decay rate; S26. Map the start and end frame numbers of each proposed segment to a binary time mask vector M of length n. j , which is defined as: Where t is the time frame index, [s j ,e j ] represents the frame index interval of the j-th proposed segment; S27, all time mask vectors M j Perform bit-by-bit logical OR operation to synthesize the final output time mask M, satisfying: Where k is the number of candidate proposal segments, and the final output temporal mask M∈{0,1} n .

4. The method for identifying customer behavior in a cloud-based environment based on deep learning according to claim 1, characterized in that: The S3 specifically includes: S31, obtaining a temporally continuous video frame sequence formed after frame-level processing from a video stream collected by a surveillance camera; S32. Obtain a time mask vector M that is consistent with the number of frames in the video frame sequence. The time mask vector M is a one-dimensional binary sequence with a length of n, wherein the value of each element is 1, indicating that the corresponding frame belongs to the candidate time period, and the value of each element is 0, indicating that the corresponding frame does not belong to the candidate time period. S33, performing a matching operation on each frame in the video frame sequence according to the value of each position in the temporal mask vector M, extracting all frames with a mask value of 1 to form a candidate frame subsequence, where the number of frames in the candidate frame subsequence is no greater than n; S34, performing a point-by-point accumulation operation on all frames in the candidate frame subsequence along the time dimension, keeping the spatial size and the number of channels unchanged, and generating a preliminary attention feature map, which represents the cumulative response degree at the spatial position within the candidate time period; S35. Perform mean normalization processing on the preliminary attention feature map in the channel dimension to obtain a two-dimensional spatial attention map, where the value of each spatial position in the spatial attention map is in the range of 0 to 1, and the value represents the relative response strength of the spatial position in the candidate time period; S36. Copy and expand the two-dimensional spatial attention map in the time dimension to n frames to form a local attention map sequence with the same shape as the original video frame sequence, and correspond the local attention map sequence to the original video frame sequence frame by frame.

5. The method for identifying customer behavior in a cloud-based environment based on deep learning according to claim 1, wherein: The S4 specifically includes: S41. Acquire a temporally continuous video frame sequence, where the video frame sequence consists of n frames, each frame is a tensor with a spatial size of W×H and a number of channels C, and the frame numbers are {1, 2, …, n} in chronological order; S42, obtaining a local attention map sequence with the same number of frames as the video frame sequence, wherein each image in the local attention map sequence is consistent with the corresponding numbered video frame in the spatial dimension, and the value range of each pixel position in the map is [0, 1]; S43, aligning the video frame sequence with the local attention map sequence according to the frame number, and matching each video frame with the local attention map with the same number in two-dimensional space coordinates to form frame-level pairing data; S44. For each frame of paired data, at each spatial coordinate position, extract the channel pixel value of the video frame tensor and the corresponding position value of the local attention map, perform point-by-point multiplication calculation in channel order, and generate a channel-invariant fused pixel value tensor; S45. Combining all processed video frame fusion tensors into a tensor sequence with n frames in the original time sequence, where each frame in the sequence is consistent with the original video frame in terms of spatial size and channel structure; S46. Output the tensor sequence as a fusion result to form video frame enhancement data for the candidate time period.

6. The method for identifying customer behavior in a cloud-based environment based on deep learning according to claim 1, characterized in that: The S5 specifically includes: S51. Input the enhanced video frame sequence of the candidate time period into the improved SlowFast dual-path action recognition network, and send it into the fast-path sampling branch and the slow-path sampling branch in the sampling module respectively. The fast-path sampling interval is r1, the slow-path is r2, and r1 < r2. The output frame index sets are respectively S52: Input the fast path sampling frame into the fast path feature extraction module, process it through the three-dimensional convolution structure unit in sequence, and output the fast path temporal feature tensor F f (t); S53: Input the slow path sampling frame into the slow path feature extraction module to gradually obtain the original slow path feature tensor F s (t), and combined with the average value of the previous frame in the historical window T through the frame-level context buffer unit Get the context-enhanced feature tensor S54, send the fast path feature tensor to the action prior trigger module to construct the adjacent frame channel difference ΔF f (t) = F f (t)-F f (t-1), and then the activation judgment logic generates the action prior trigger vector: Among them, θ and δ are the threshold parameters; S55, the action prior trigger vector G f (t) Input channel gating module to enhance the context features Apply gate control, perform time-step channel suppression operations, and output gated slow path tensors S56, map the local attention map sequence to and At the corresponding time step, perform channel-level point-by-point multiplication operations and output the fast path tensor F′ f (t) and the slow path tensor F″ s (t).

7. The method for identifying customer behavior in a cloud-based environment based on deep learning according to claim 1, wherein: The improved SlowFast dual-path action recognition network specifically includes: A sampling module, the sampling module includes a fast path sampling branch and a slow path sampling branch, the fast path sampling branch samples the original video frame sequence at a first frame interval, and the slow path sampling branch samples at a second frame interval, the first frame interval is smaller than the second frame interval, and outputs a fast path input sequence and a slow path input sequence; A fast-path feature extraction module, comprising a plurality of three-dimensional convolutional structural units arranged in series, each of which sequentially comprises a three-dimensional convolutional layer, a normalization layer, and an activation function layer. The structural unit processes the fast-path input sequence frame by frame to extract short-term action feature tensors; A slow-path feature extraction module includes multiple three-dimensional convolutional structural units arranged in series. Each structural unit sequentially includes a three-dimensional convolutional layer, a normalization layer, an activation function layer, and a frame-level context buffer unit. The frame-level context buffer unit is used to introduce the average value of historical frame features within a fixed time window and perform point-by-point multiplication with the current frame features to generate a context-enhanced feature tensor; An action priori trigger module, connected to the output of the fast path feature extraction module, includes a spatial difference calculation unit and an activation judgment logic unit. The spatial difference calculation unit calculates the channel-by-channel difference of the feature tensor between adjacent frames, and the activation judgment logic unit generates an action priori trigger vector based on the comparison between the difference value and a set threshold. The channel gating module is used to receive the action prior trigger vector, and perform channel selective shielding or retention on the slow path feature tensor in the time step dimension, and output the gated slow path feature tensor.

8. The method for identifying customer behavior in a cloud-based environment based on deep learning according to claim 1, characterized in that: The S6 specifically includes: S61. Obtain the fast path feature tensor and slow path feature tensor output by the improved SlowFast dual-path action recognition network. The time dimensions of the fast path feature tensor and the slow path feature tensor may be different, and the number of channels is C f and C s ; S62. Project or interpolate the mask onto the time dimensions of the fast path and the slow path according to the time mask vector to obtain a fast path mask sequence and a slow path mask sequence; S63. In the time dimension of the fast path and the slow path, use the mask sequence as the weight, and perform weighted averaging on the feature tensors along the time dimension to obtain the fast path aggregate feature and the slow path aggregate feature, each of which is a one-dimensional channel vector. S64: Accumulate all local attention maps of the candidate time period in chronological order, and then divide it by the number of valid frames to obtain the average attention map. Then, transform the average attention map into a fixed-length vector through convolution or full connection operation, denoted as F a ; S65. Concatenate the fast path aggregation features, slow path aggregation features, and attention feature vectors in the channel dimension to obtain the customer behavior representation vector V, where the length of V is equal to the sum of the number of channels of the three. The output is used as the input feature for subsequent behavior classification.

9. The method for identifying customer behavior in a cloud-based environment based on deep learning according to claim 1, characterized in that: The S7 specifically includes: S71. Obtain a customer behavior representation vector, which is a fixed-length one-dimensional channel feature tensor and includes feature components formed by concatenating multiple source channels. S72. Input the customer behavior representation vector into a behavior classification module. The behavior classification module includes an input layer, two fully connected layers, and an output layer. Each layer is sequentially connected to a batch normalization layer and a nonlinear activation unit. The output dimension is a preset number of categories K. S73. Calculate the confidence probability of each customer behavior category in the output layer to obtain the behavior prediction probability vector P = [p1, p2, ..., p K ], where each component p k Indicates the probability value of the prediction belonging to category k, satisfying S74. Select the category index k corresponding to the maximum probability value * As the customer behavior prediction category number, match the corresponding behavior category name in the behavior label table and output; S75. Select the predicted category k from the candidate time period set. * The time segment with the highest confidence score, whose frame index is the starting frame number s * and the end frame number e * , the confidence score is calculated based on the similarity measure between the feature vector in the candidate segment and the behavior representation vector; S76, output customer behavior prediction result and time positioning result triple (k * ,s * ,e * ), as the result of customer behavior identification and positioning.

Images