Theft behavior identification method and system based on computer vision

Abstract

The invention provides a theft behavior recognition method and system based on computer vision, and relates to the technical field of machine vision, and the method comprises the steps: preprocessing a video stream, detecting a target through a YOLOv11 algorithm, extracting features through a feature pyramid network and a path aggregation network, extracting a posture skeleton and micro expression features through an Openpose algorithm and wavelet transform, and carrying out the recognition of a theft behavior through a computer. And inputting a deep learning model to generate a behavior feature vector. And inputting the feature vector into an analysis engine fusing a convolutional neural network and a Transform structure, calculating the similarity with a preset template and generating a confidence score, thereby realizing accurate identification and timely early warning of the theft behavior.

Description

Technical Field

[0001] The present invention relates to machine vision technology, and in particular to a theft behavior recognition method and system based on computer vision. Background Art

[0002] With the rapid development of social economy, the security management issues in public places such as shopping malls, supermarkets, and exhibition halls have become increasingly prominent, among which the prevention and timely detection of theft is particularly important. Traditional security monitoring systems mainly rely on manual duty and simple video monitoring, which has problems such as high labor costs and low monitoring efficiency. In recent years, with the development of computer vision and deep learning technology, behavior recognition systems based on intelligent visual analysis have gradually become a research hotspot in the field of security. The current intelligent visual analysis system mainly uses technologies such as video image processing, target detection, and behavior analysis to achieve automatic recognition and early warning of suspicious behaviors.

[0003] The existing theft behavior recognition technology has the following main shortcomings: The existing technology has a low recognition accuracy in complex scenes, especially in crowded environments with drastic light changes, which is prone to misjudgment and missed judgment. This is mainly due to the lack of adaptability of existing algorithms to environmental interference factors and the lack of in-depth analysis of behavioral characteristics.

[0004] Current recognition systems often only focus on a single behavioral feature, such as human posture or motion trajectory, and ignore the analysis of detailed features such as micro-expressions and hand movements during the theft process, resulting in the system being unable to fully and accurately judge the true intention of the behavior.

[0005] Existing technologies lack an effective multi-feature fusion mechanism and fail to organically combine multi-dimensional information such as human posture, facial expressions, and behavior timing. At the same time, they lack a reasonable confidence assessment system, making it difficult to accurately classify suspicious behaviors of different degrees and issue timely warnings. Summary of the invention

[0006] The embodiments of the present invention provide a theft behavior recognition method and system based on computer vision, which can solve the problems in the prior art.

[0007] According to a first aspect of the embodiments of the present invention, Provides a theft behavior recognition method based on computer vision, including: Collecting a video data stream, performing frame rate conversion on the video data stream to obtain a video stream with a target frame rate, applying Gaussian filtering and median filtering to the video stream with the target frame rate to eliminate noise, and performing image segmentation on the filtered video stream using a threshold segmentation algorithm to obtain a preprocessed video frame; The YOLOv11 deep learning target detection algorithm is used to perform target detection and positioning on the preprocessed video frame to obtain bounding box information of the behavior target, the bounding box information is input into a feature pyramid network for multi-scale feature extraction, and a high-level semantic feature and a low-level detail feature are fused through a path aggregation network to obtain a target feature map; based on the target feature map, the Openpose algorithm is used to extract the coordinate data of the hand key points and the trunk key points of the behavior target, and the partial affinity field technology is used to establish the association relationship between the hand key points and the trunk key points to construct a posture skeleton, wavelet transform and spatiotemporal attention mechanism are used to extract features of the facial area of ​​the behavior target to obtain micro-expression features, the posture skeleton and the micro-expression features are input into a deep learning decision classification model, and a behavior feature vector including the distance data between the hand and the trunk, the hand movement trajectory data, and the behavior time window data is output; The behavior feature vector is input into the theft behavior scoring mechanism analysis engine for behavior judgment. The theft behavior scoring mechanism analysis engine integrates convolutional neural network and Transformer structure, calculates the similarity between the behavior feature vector and the preset behavior template and generates a confidence score. When the confidence score is greater than 0.8, it is judged as theft behavior and triggers an alarm signal. When the confidence score is between 0.5 and 0.8, it is judged as abnormal behavior and a tracking mark is recorded. When the confidence score is less than 0.5, it is judged as normal behavior.

[0008] The preprocessed video frames are subjected to target detection and positioning using the YOLOv11 deep learning target detection algorithm to obtain bounding box information of the behavioral target, and the bounding box information is input into the feature pyramid network for multi-scale feature extraction and the target feature map is obtained by fusing high-level semantic features and low-level detail features through the path aggregation network, including: Divide the preprocessed video frame into a number of grid units, perform bounding box prediction on the grid units, and obtain the center coordinates, width, height and confidence of the bounding box, where the confidence is calculated by multiplying the probability of the object existing in the box by the intersection and union ratio of the predicted box and the real box; A feature pyramid network is constructed based on the bounding box, the preprocessed video frame is downsampled by a convolution layer to obtain five bottom-level feature maps of different scales, the sizes of the five bottom-level feature maps of different scales are halved successively, the high-level feature map is upsampled and one-dimensionally convolved with the horizontally connected feature map of the same scale to obtain a pyramid feature map; Input the pyramid feature map into a path aggregation network, construct a top-down enhancement path and a bottom-up enhancement path in the path aggregation network, the top-down enhancement path obtains a top-down feature map through upsampling and feature concatenation, the bottom-up enhancement path obtains a bottom-up feature map through downsampling and feature concatenation, and the top-down feature map and the bottom-up feature map are fused through a learnable fusion weight coefficient to obtain a fused feature map; The fused feature map is respectively subjected to attention enhancement in the channel dimension and the spatial dimension. The attention enhancement in the channel dimension is performed by performing average pooling and maximum pooling on the fused feature map and obtaining a channel attention weight through a multilayer perceptron. The attention enhancement in the spatial dimension is performed by performing average pooling and maximum pooling on the fused feature map and obtaining a spatial attention weight through a convolution operation. The fused feature map is multiplied by the channel attention weight and the spatial attention weight to obtain an attention enhancement feature map. The information entropy of the attention enhanced feature map at different scales is calculated, and the dynamic fusion weights of different scales are calculated based on the information entropy. The attention enhanced feature map and the corresponding dynamic fusion weights are weightedly summed to obtain a final feature map, wherein the final feature map contains multi-scale semantic information and detail features of the target.

[0009] Based on the target feature graph, the coordinate data of the hand key points and the trunk key points of the behavior target are extracted by using the Openpose algorithm, and the association relationship between the hand key points and the trunk key points is established by using the partial affinity field technology to construct the posture skeleton, including: Based on the target feature map, a two-branch convolutional neural network is used to generate a confidence map through the first branch, and a partial affinity field is generated through the second branch. The distance from the pixel point in the target feature map to the real position of the key point is calculated using a Gaussian kernel function to obtain an initial confidence map of multi-level key points, and the candidate positions of the key points of multiple human targets are determined based on the initial confidence map; Based on the candidate positions of the key points, the initial coordinates of the wrist key points are obtained, the hand region is adaptively cropped with the initial coordinates of the wrist key points as the center to obtain the hand region of interest, the hand region of interest is input into the hand key point extraction network, a hand key point heat map is generated through multi-layer convolution operations, and the accurate coordinates of the hand fine key points are extracted from the hand key point heat map; A hierarchical detection strategy is adopted for the trunk key points in the key point candidate positions, a reference position of the trunk main key point is determined based on the maximum response value of the initial confidence map, the reference position is input into a position regression network to calculate the spatial offset of the secondary key point relative to the reference position, and the precise position of the trunk secondary key point is obtained according to the spatial offset and the reference position; A bidirectional partial affinity field is constructed based on the accurate coordinates of the hand fine key points and the precise positions of the trunk secondary key points, a feature vector integral value of the forward partial affinity field is calculated along the key point connection direction, a feature vector integral value of the reverse partial affinity field is calculated along the reverse direction of the key point connection, and the forward partial affinity field and the reverse partial affinity field are fused through adaptive direction weights to obtain an enhanced partial affinity field; The enhanced partial affinity field is used to calculate the connection relationship scores between adjacent key points, and the optimal skeleton topology structure is obtained by maximizing the global accumulation of the connection relationship scores. A temporal smoothing constraint is applied to the optimal skeleton topology structure, and the current frame posture is adaptively weighted fused with the previous frame historical posture and motion predicted posture to obtain a smoothed posture skeleton that is consistent in time and space. The smoothed posture skeleton contains a stable connection relationship between the hand and torso key points.

[0010] A hierarchical detection strategy is adopted for the trunk key points in the key point candidate positions, a reference position of the trunk trunk key points is determined based on the maximum response value of the initial confidence map, the reference position is input into the position regression network to calculate the spatial offset of the secondary key points relative to the reference position, and the precise position of the trunk secondary key points is obtained according to the spatial offset and the reference position, including: Performing weighted summation on the heat map responses of multiple channels to obtain a confidence response map, setting a response threshold based on the confidence response map to screen and obtain a candidate region of a backbone key point, and performing Gaussian smoothing on the candidate region of the backbone key point to obtain a smoothed response map; Determine the maximum response position in the smooth response map as the reference position of the backbone key point, calculate the sub-pixel offset using the response difference in the horizontal direction and the vertical direction, and add the reference position and the sub-pixel offset to obtain the refined backbone key point coordinates; Construct a multi-level feature pyramid network, perform upsampling and convolution operations on the input feature map to obtain feature maps of different levels, cascade the feature maps of different levels into a position regression network, and predict the spatial offset of the secondary key point relative to the refined main key point coordinates; Establishing bone length constraints and joint angle constraints based on human anatomical features, adding the refined trunk key point coordinates to the spatial offset to obtain the secondary key point initial position, and performing constraint optimization on the secondary key point initial position to obtain the optimized secondary key point position that satisfies the bone length constraints and the joint angle constraints; A temporal smoothness constraint is introduced to calculate the displacement difference between the secondary key point position of the current frame and the secondary key point position of the previous two frames, and the movement speed difference between the current frame and the previous frame. The spatial positioning error, displacement difference and speed difference are used as optimization targets, and the timing consistency optimization is performed on the optimized secondary key point position to obtain the final precise position of the secondary key point.

[0011] Wavelet transform and spatiotemporal attention mechanism are used to extract features from the facial area of ​​the behavior target to obtain micro-expression features, and the posture skeleton and the micro-expression features are input into a deep learning decision classification model, and the output behavior feature vector including the distance data between the hand and the torso, the hand movement trajectory data, and the behavior time window data includes: Performing a multi-scale two-dimensional discrete wavelet transform on the facial region image of the behavioral target to obtain wavelet coefficients of different scales, calculating the wavelet energy value at each scale based on the wavelet coefficients and determining the energy proportion, thereby obtaining an initial feature map of the facial region; Input the initial feature map into a dual-stream spatiotemporal attention network, perform average pooling and maximum pooling on the initial feature map through the spatial attention branch to obtain a spatial attention weight, perform a three-dimensional convolution operation on the initial feature map through the temporal attention branch to obtain a temporal attention weight, and multiply the initial feature map with the spatial attention weight and the temporal attention weight to obtain an enhanced feature map; The Euclidean distance between the hand key points and the trunk key points in the posture skeleton is calculated to construct the hand-torso distance matrix. The speed, acceleration and direction angle of the hand movement are calculated based on the continuous frame position information of the hand key points to obtain the hand movement trajectory descriptor. A sliding time window is used to perform feature aggregation on the enhanced feature map, the temporal correlation of the feature sequence in the time window is calculated, and the feature sequence is weighted and summed with the time weight to obtain the time window feature; The enhanced feature map, the hand-torso distance matrix, the hand motion trajectory descriptor and the time window feature are concatenated to construct a multimodal feature vector, and the multimodal feature vector is simultaneously input into a multilayer perceptron and a bidirectional long short-term memory network for feature fusion. The fused features are passed through a softmax classifier to obtain the behavior category probability distribution, and the classification model is optimized and trained based on the cross entropy loss function and the regularization term to obtain the final behavior classification result.

[0012] Inputting the behavior feature vector into the theft behavior scoring mechanism analysis engine for behavior determination, the theft behavior scoring mechanism analysis engine integrates the convolutional neural network and the Transformer structure, calculates the similarity between the behavior feature vector and the preset behavior template and generates a confidence score including: Reorganize the input behavior feature vector into a spatiotemporal feature matrix, embed sine position coding and cosine position coding into the spatiotemporal feature matrix to obtain a coded spatiotemporal feature matrix; Using convolution branches with different convolution kernel sizes to perform parallel convolution operations on the encoded spatiotemporal feature matrix to obtain multi-scale convolution features, and performing weighted fusion on the multi-scale convolution features to obtain fused convolution features; Performing query matrix transformation, key matrix transformation and value matrix transformation on the encoded spatiotemporal feature matrix to obtain a query matrix, a key matrix and a value matrix, calculating an attention weight based on the product of the query matrix and the key matrix, multiplying the attention weight by the value matrix to obtain an attention feature, and concatenating a plurality of the attention features to obtain a multi-head attention output feature; Calculate the cosine similarity between the fused convolution feature and the preset behavior template to obtain a similarity feature, and perform dynamic time warping calculation on the fused convolution feature and the preset behavior template to obtain a time warping distance feature; Performing weighted summation on the fused convolution feature, the multi-head attention output feature, the similarity feature and the time-warped distance feature to obtain an integrated feature, and performing a sigmoid activation operation on the integrated feature to obtain an initial confidence score; The initial confidence score is subjected to exponential smoothing to obtain a smoothed confidence score, the temporal consistency of the smoothed confidence score is calculated, and a weighted sum of the smoothed confidence score and the temporal consistency is taken as a final theft behavior determination result.

[0013] A second aspect of an embodiment of the present invention provides a theft behavior recognition system based on computer vision, comprising: The first unit is used to collect a video data stream, perform frame rate conversion on the video data stream to obtain a video stream with a target frame rate, apply Gaussian filtering and median filtering to the video stream with the target frame rate to eliminate noise, and use a threshold segmentation algorithm to perform image segmentation on the filtered video stream to obtain a preprocessed video frame; The second unit is used to use the YOLOv11 deep learning target detection algorithm to perform target detection and positioning on the preprocessed video frame to obtain the bounding box information of the behavior target, input the bounding box information into the feature pyramid network to perform multi-scale feature extraction, and fuse the high-level semantic features and low-level detail features through the path aggregation network to obtain the target feature map; based on the target feature map, the Openpose algorithm is used to extract the coordinate data of the hand key points and the trunk key points of the behavior target, and the partial affinity field technology is used to establish the association relationship between the hand key points and the trunk key points to construct a posture skeleton, and the wavelet transform and spatiotemporal attention mechanism are used to extract the features of the facial area of ​​the behavior target to obtain micro-expression features, and the posture skeleton and the micro-expression features are input into the deep learning decision classification model, and the behavior feature vector containing the distance data between the hand and the trunk, the hand movement trajectory data, and the behavior time window data is output; The third unit is used to input the behavior feature vector into the theft behavior scoring mechanism analysis engine for behavior judgment. The theft behavior scoring mechanism analysis engine integrates the convolutional neural network and the Transformer structure, calculates the similarity between the behavior feature vector and the preset behavior template and generates a confidence score. When the confidence score is greater than 0.8, it is judged as theft behavior and triggers an alarm signal. When the confidence score is between 0.5 and 0.8, it is judged as abnormal behavior and a tracking mark is recorded. When the confidence score is less than 0.5, it is judged as normal behavior.

[0014] A third aspect of the embodiments of the present invention An electronic device is provided, comprising: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0015] According to a fourth aspect of the embodiments of the present invention, A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0016] The beneficial effects of this application are as follows: The present invention uses Gaussian filtering and median filtering to eliminate noise, and combines the threshold segmentation algorithm for image preprocessing, which significantly improves the accuracy and robustness of subsequent target detection. At the same time, the YOLOv11 deep learning algorithm is used for target detection and positioning, and the feature pyramid network and path aggregation network are combined for multi-scale feature extraction and fusion, which effectively improves the target recognition ability in complex scenes.

[0017] This paper innovatively combines the Openpose algorithm and partial affinity field technology to accurately extract the key points of the hands and torso of the behavior target, construct a posture skeleton, and extract micro-expression features through wavelet transform and spatiotemporal attention mechanism. This multimodal feature fusion method greatly enhances the comprehensiveness and accuracy of behavior analysis and can capture subtle behavioral characteristics and emotional changes.

[0018] The present invention designs a theft behavior scoring mechanism analysis engine that integrates convolutional neural networks and Transformer structures. By calculating the similarity between the behavior feature vector and the preset behavior template, a confidence score is generated, and behavior judgment is performed according to different score ranges. This method can not only accurately identify theft behavior, but also mark and track abnormal behavior, greatly improving the practicality and reliability of the system, and providing an efficient and intelligent solution for the security monitoring field. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Schematic diagram of the flow of the theft behavior recognition method based on computer vision according to an embodiment of the present invention; Figure 2 Schematic diagram of the structure of a theft behavior recognition system based on computer vision according to an embodiment of the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0022] Figure 1 FIG. 1 is a flow chart of a method for identifying theft based on computer vision according to an embodiment of the present invention. Figure 1 As shown, the method includes: S101. Collecting a video data stream, performing frame rate conversion on the video data stream to obtain a video stream with a target frame rate, applying Gaussian filtering and median filtering to the video stream with the target frame rate to eliminate noise, and performing image segmentation on the filtered video stream using a threshold segmentation algorithm to obtain preprocessed video frames; S102. Use the YOLOv11 deep learning target detection algorithm to perform target detection and positioning on the preprocessed video frame to obtain the bounding box information of the behavior target, input the bounding box information into the feature pyramid network to perform multi-scale feature extraction, and fuse the high-level semantic features and low-level detail features through the path aggregation network to obtain the target feature map; based on the target feature map, use the Openpose algorithm to extract the coordinate data of the hand key points and the trunk key points of the behavior target, and use the partial affinity field technology to establish the association relationship between the hand key points and the trunk key points to construct a posture skeleton, use wavelet transform and spatiotemporal attention mechanism to extract features from the facial area of ​​the behavior target to obtain micro-expression features, input the posture skeleton and the micro-expression features into the deep learning decision classification model, and output the behavior feature vector including the distance data between the hand and the trunk, the hand movement trajectory data, and the behavior time window data; S103. Input the behavior feature vector into the theft behavior scoring mechanism analysis engine for behavior judgment. The theft behavior scoring mechanism analysis engine integrates the convolutional neural network and the Transformer structure, calculates the similarity between the behavior feature vector and the preset behavior template and generates a confidence score. When the confidence score is greater than 0.8, it is judged as theft behavior and triggers an alarm signal. When the confidence score is between 0.5 and 0.8, it is judged as abnormal behavior and a tracking mark is recorded. When the confidence score is less than 0.5, it is judged as normal behavior.

[0023] In an optional implementation, the preprocessed video frame is subjected to target detection and positioning using the YOLOv11 deep learning target detection algorithm to obtain bounding box information of the behavioral target, the bounding box information is input into a feature pyramid network for multi-scale feature extraction, and a target feature map is obtained by fusing high-level semantic features and low-level detail features through a path aggregation network: Divide the preprocessed video frame into a number of grid units, perform bounding box prediction on the grid units, and obtain the center coordinates, width, height and confidence of the bounding box, where the confidence is calculated by multiplying the probability of the object existing in the box by the intersection and union ratio of the predicted box and the real box; A feature pyramid network is constructed based on the bounding box, the preprocessed video frame is downsampled by a convolution layer to obtain five bottom-level feature maps of different scales, the sizes of the five bottom-level feature maps of different scales are halved successively, the high-level feature map is upsampled and one-dimensionally convolved with the horizontally connected feature map of the same scale to obtain a pyramid feature map; Input the pyramid feature map into a path aggregation network, construct a top-down enhancement path and a bottom-up enhancement path in the path aggregation network, the top-down enhancement path obtains a top-down feature map through upsampling and feature concatenation, the bottom-up enhancement path obtains a bottom-up feature map through downsampling and feature concatenation, and the top-down feature map and the bottom-up feature map are fused through a learnable fusion weight coefficient to obtain a fused feature map; The fused feature map is respectively subjected to attention enhancement in the channel dimension and the spatial dimension. The attention enhancement in the channel dimension is performed by performing average pooling and maximum pooling on the fused feature map and obtaining a channel attention weight through a multilayer perceptron. The attention enhancement in the spatial dimension is performed by performing average pooling and maximum pooling on the fused feature map and obtaining a spatial attention weight through a convolution operation. The fused feature map is multiplied by the channel attention weight and the spatial attention weight to obtain an attention enhancement feature map. The information entropy of the attention enhanced feature map at different scales is calculated, and the dynamic fusion weights of different scales are calculated based on the information entropy. The attention enhanced feature map and the corresponding dynamic fusion weights are weightedly summed to obtain a final feature map, wherein the final feature map contains multi-scale semantic information and detail features of the target.

[0024] The video behavior target feature extraction method based on the YOLOv11 deep learning target detection algorithm first preprocesses the input video frames, including image normalization, data enhancement, etc. The preprocessed video frames are input into the target detection module for processing.

[0025] In the object detection phase, the preprocessed video frames are divided into grid cells of size 13×13. Each grid cell is responsible for predicting three bounding boxes of different scales, each of which contains center coordinates, width, and height information. The overlapping boxes are filtered by the non-maximum suppression algorithm with a threshold of 0.5 to improve the detection accuracy. The confidence of the bounding box is obtained by multiplying the intersection of the predicted box and the true box by the probability of the object existing in the box, and the confidence threshold is set to 0.6.

[0026] The feature pyramid network adopts a five-layer structure design, and downsamples the input feature map through a 3×3 convolution kernel and a maximum pooling layer with a stride of 2. Taking the input size of 416×416 as an example, the feature map sizes obtained after downsampling are 208×208, 104×104, 52×52, 26×26, and 13×13 respectively. When fusion of features, a 1×1 convolution layer is used to adjust the number of channels to ensure that the number of channels of feature maps at different levels is consistent. The upsampling adopts the nearest neighbor interpolation method with a sampling rate of 2 to double the size of the feature map.

[0027] In the path aggregation network, the top-down path achieves feature enhancement through 2x upsampling and feature cascading. For the top feature map of size 13×13, a 26×26 feature map is obtained after upsampling, and the enhanced feature is obtained by cascading with the feature map of the same scale. The bottom-up path uses the maximum pooling with a step size of 2 to achieve feature downsampling and cascade with the feature map of the same scale. The feature fusion weights are adaptively adjusted through trainable parameters, and the initial weights are all set to 0.5.

[0028] The attention mechanism enhancement module is divided into two branches: channel attention and spatial attention. The channel attention branch performs global average pooling and maximum pooling on the feature map to obtain two one-dimensional vectors, which are passed through two fully connected layers to obtain weight coefficients. The spatial attention branch also performs double pooling operations and obtains a two-dimensional attention map through a 7×7 convolution layer. The feature map is multiplied by the two attention weights to obtain enhanced features.

[0029] Finally, the information entropy of feature maps of different scales is calculated, and the fusion weight is dynamically assigned according to the entropy value. The larger the information entropy, the richer the feature information, and the larger the corresponding weight. The enhanced feature map is weighted and summed with the dynamic weight to obtain the final feature map that integrates multi-scale semantic information and detail features.

[0030] This application can achieve: Through multi-scale feature extraction and fusion strategies, the multi-level feature information of the target is effectively extracted, which improves the feature expression ability and detection accuracy. The bidirectional path enhancement and adaptive feature fusion mechanism are adopted to make full use of the complementarity of features at different levels and enhance the feature discrimination ability. The channel and space dual attention mechanism is introduced to highlight important feature information, suppress irrelevant background interference, and improve the robustness and accuracy of target detection.

[0031] In an optional implementation, based on the target feature graph, the coordinate data of the hand key points and the trunk key points of the behavior target are extracted using the Openpose algorithm, and the association relationship between the hand key points and the trunk key points is established using the partial affinity field technology to construct the posture skeleton, including: Based on the target feature map, a two-branch convolutional neural network is used to generate a confidence map through the first branch, and a partial affinity field is generated through the second branch. The distance from the pixel point in the target feature map to the real position of the key point is calculated using a Gaussian kernel function to obtain an initial confidence map of multi-level key points, and the candidate positions of the key points of multiple human targets are determined based on the initial confidence map; Based on the candidate positions of the key points, the initial coordinates of the wrist key points are obtained, the hand region is adaptively cropped with the initial coordinates of the wrist key points as the center to obtain the hand region of interest, the hand region of interest is input into the hand key point extraction network, a hand key point heat map is generated through multi-layer convolution operations, and the accurate coordinates of the hand fine key points are extracted from the hand key point heat map; A hierarchical detection strategy is adopted for the trunk key points in the key point candidate positions, a reference position of the trunk main key point is determined based on the maximum response value of the initial confidence map, the reference position is input into a position regression network to calculate the spatial offset of the secondary key point relative to the reference position, and the precise position of the trunk secondary key point is obtained according to the spatial offset and the reference position; A bidirectional partial affinity field is constructed based on the accurate coordinates of the hand fine key points and the precise positions of the trunk secondary key points, a feature vector integral value of the forward partial affinity field is calculated along the key point connection direction, a feature vector integral value of the reverse partial affinity field is calculated along the reverse direction of the key point connection, and the forward partial affinity field and the reverse partial affinity field are fused through adaptive direction weights to obtain an enhanced partial affinity field; The enhanced partial affinity field is used to calculate the connection relationship scores between adjacent key points, and the optimal skeleton topology structure is obtained by maximizing the global accumulation of the connection relationship scores. A temporal smoothing constraint is applied to the optimal skeleton topology structure, and the current frame posture is adaptively weighted fused with the previous frame historical posture and motion predicted posture to obtain a smoothed posture skeleton that is consistent in time and space. The smoothed posture skeleton contains a stable connection relationship between the hand and torso key points.

[0032] The human posture estimation method based on the target feature map first uses a two-branch convolutional neural network for feature extraction. The first branch network contains multiple convolutional layers and pooling layers, and the output size is a confidence map with a width multiplied by a height multiplied by the number of channels. Each channel of the confidence map corresponds to a key point type. The pixels in the target feature map are processed by the Gaussian kernel function, and the distance from the pixel to the true position of the key point is calculated to generate an initial confidence map. The non-maximum suppression method is used on the initial confidence map to extract the candidate positions of the key points.

[0033] For the precise positioning of the key points of the hand, the hand region is adaptively cropped based on the candidate position of the wrist key point. The size of the cropped area is dynamically adjusted according to the image features of the area around the wrist key point. Generally, a square area centered on the wrist key point is taken, with a side length of 2 to 3 times the width of the wrist. The cropped hand region of interest is input into a dedicated hand key point extraction network, which adopts a stacked hourglass network structure and includes downsampling and upsampling processes. A heat map of the hand key points is generated through multi-layer convolution operations, and each channel corresponds to a hand key point. The sub-pixel positioning method is used on the heat map to extract the precise coordinates of the hand key points.

[0034] The trunk key points are located using a hierarchical detection strategy. First, the trunk key points, including the neck, shoulder and hip key points, are determined by the maximum response value on the initial confidence map. These key points are input into the position regression network as the reference position, which predicts the spatial offset of the secondary key points relative to the reference position. The position regression network adopts a residual structure and outputs the x and y direction components of the offset. The predicted offset is superimposed on the reference position to obtain the precise position of the secondary trunk key points.

[0035] After obtaining the precise positions of the key points of the hands and torso, a bidirectional partial affinity field is constructed to establish the connection relationship between the key points. Along the connection direction between the key point pairs, the feature vector is integrated in this direction to obtain the forward partial affinity field value. Similarly, the integration is performed in the reverse direction to obtain the reverse partial affinity field value. The weights of the forward and reverse fields are adaptively adjusted according to the complexity of the image content, and they are fused to obtain the enhanced partial affinity field.

[0036] Finally, the enhanced partial affinity field is used to calculate the connection relationship score between adjacent key points. The Hungarian algorithm is used to maximize the sum of the scores of all possible connections to obtain the optimal skeleton topology. In order to ensure the temporal consistency of the pose estimation results, the pose of the current frame is weightedly fused with the historical pose of the previous frame and the motion pose predicted based on the historical pose. The weight is adaptively determined according to the credibility of the pose of each frame, and finally a smoothed pose skeleton consistent in time and space is obtained.

[0037] This application can achieve: The candidate positions of key points are extracted by combining a dual-branch convolutional neural network with a Gaussian kernel function, which improves the accuracy and robustness of the initial key point positioning. The hand area is accurately positioned using an adaptive cropping strategy, which effectively solves the scale change problem in hand key point detection.

[0038] The hierarchical trunk key point detection strategy makes full use of the prior knowledge of human body structure and predicts the spatial offset through the position regression network, which significantly improves the accuracy of trunk key point positioning. The construction method of the bidirectional partial affinity field enhances the ability to express the connection relationship between key points.

[0039] The temporal smoothing constraint and adaptive weight fusion mechanism effectively suppress the jitter of the pose estimation results and ensure the stability of the pose skeleton in continuous video sequences. The overall solution has strong environmental adaptability and high real-time performance in complex scenes.

[0040] In an optional implementation, a hierarchical detection strategy is adopted for the trunk key points in the key point candidate positions, a reference position of the trunk trunk key point is determined based on the maximum response value of the initial confidence map, the reference position is input into a position regression network to calculate the spatial offset of the secondary key point relative to the reference position, and the precise position of the trunk secondary key point is obtained according to the spatial offset and the reference position, including: Performing weighted summation on the heat map responses of multiple channels to obtain a confidence response map, setting a response threshold based on the confidence response map to screen and obtain a candidate region of a backbone key point, and performing Gaussian smoothing on the candidate region of the backbone key point to obtain a smoothed response map; Determine the maximum response position in the smooth response map as the reference position of the backbone key point, calculate the sub-pixel offset using the response difference in the horizontal direction and the vertical direction, and add the reference position and the sub-pixel offset to obtain the refined backbone key point coordinates; Construct a multi-level feature pyramid network, perform upsampling and convolution operations on the input feature map to obtain feature maps of different levels, cascade the feature maps of different levels into a position regression network, and predict the spatial offset of the secondary key point relative to the refined main key point coordinates; Establishing bone length constraints and joint angle constraints based on human anatomical features, adding the refined trunk key point coordinates to the spatial offset to obtain the secondary key point initial position, and performing constraint optimization on the secondary key point initial position to obtain the optimized secondary key point position that satisfies the bone length constraints and the joint angle constraints; A temporal smoothness constraint is introduced to calculate the displacement difference between the secondary key point position of the current frame and the secondary key point position of the previous two frames, and the movement speed difference between the current frame and the previous frame. The spatial positioning error, displacement difference and speed difference are used as optimization targets, and the timing consistency optimization is performed on the optimized secondary key point position to obtain the final precise position of the secondary key point.

[0041] First, a multi-channel heat map is generated for the input image. The input image is feature extracted through a convolutional neural network to generate a 17-channel key point heat map, each channel corresponding to a human key point type. These heat maps are weighted and summed, and the weights can be set according to the importance of the key points. For example, the weight of the torso can be set to 0.6, and the weight of the limbs can be set to 0.4. After weighted summation, the confidence response map is obtained, and a response threshold of 0.3 is used for screening to obtain candidate areas that may contain trunk key points. These candidate areas are smoothed using a Gaussian kernel with a standard deviation of 1.5 to reduce the impact of noise.

[0042] In the smoothed response graph, search for the position with the largest response value as the reference position of the backbone key point. In order to obtain more accurate coordinates, calculate the response difference in the horizontal and vertical directions. For example, if the response value of a certain position is 0.8, and the response values ​​of its left and right adjacent positions are 0.7 and 0.6 respectively, then the sub-pixel offset of the point in the horizontal direction can be calculated as 0.2 pixels based on the response difference. Similarly, calculate the offset in the vertical direction, and add the reference position and the offset to obtain the refined coordinates of the backbone key point.

[0043] A feature pyramid network is constructed for secondary keypoint detection. The network consists of 4 levels, corresponding to the 1 / 2, 1 / 4, 1 / 8, and 1 / 16 scales of the input feature map. Each level extracts features through 3x3 convolution, and then uses bilinear interpolation to perform 2x upsampling and concatenate with the feature map of the previous level. The multi-scale feature map obtained in this way is input into the position regression network, which consists of 3 fully connected layers and outputs the spatial offset of the secondary keypoint relative to the backbone keypoint.

[0044] Constraints are set based on human anatomical features. For bone length constraints, the upper arm length range is set to 20-35 cm, and the forearm is set to 22-32 cm. For joint angle constraints, the elbow joint range is 0-145 degrees, and the shoulder joint horizontal direction is -40 to 180 degrees. The coordinates of the trunk key points are added to the predicted offset to obtain the initial position of the secondary key points, and then iterative optimization is performed to make them meet these constraints.

[0045] Finally, the timing constraint is introduced to achieve smoothing. Calculate the position difference between the current frame and the previous two frames. For example, if the coordinates of a key point in the current frame are (100, 100), the previous frame is (98, 99), and the previous two frames are (95, 97), then the displacement difference is 5 pixels. At the same time, calculate the speed change. If the speed of the previous frame is 3 pixels / frame and the current frame is 4 pixels / frame, the speed difference is 1 pixel / frame. Comprehensively consider the spatial positioning error, displacement difference and speed difference for optimization to obtain the final secondary key point position.

[0046] This application can achieve: Through hierarchical detection strategy and multi-scale feature extraction, the accuracy of torso key point detection is significantly improved, the detection error is reduced, and the key point positioning is made more accurate and reliable.

[0047] The introduction of human anatomical constraints and temporal smoothing constraints effectively suppresses the key point jump and drift phenomena, enhances the stability of the detection results, and makes the posture estimation more consistent with the human body movement characteristics.

[0048] The use of sub-pixel precise positioning and multi-layer optimization strategies has improved the robustness of the algorithm, making it able to adapt to different scenarios and posture changes, and has strong practical value and prospects for promotion and application.

[0049] In an optional implementation, wavelet transform and spatiotemporal attention mechanism are used to extract features of the facial area of ​​the behavior target to obtain micro-expression features, the posture skeleton and the micro-expression features are input into a deep learning decision classification model, and the output behavior feature vector including the distance data between the hand and the torso, the hand motion trajectory data, and the behavior time window data includes: Performing a multi-scale two-dimensional discrete wavelet transform on the facial region image of the behavioral target to obtain wavelet coefficients of different scales, calculating the wavelet energy value at each scale based on the wavelet coefficients and determining the energy proportion, thereby obtaining an initial feature map of the facial region; Input the initial feature map into a dual-stream spatiotemporal attention network, perform average pooling and maximum pooling on the initial feature map through the spatial attention branch to obtain a spatial attention weight, perform a three-dimensional convolution operation on the initial feature map through the temporal attention branch to obtain a temporal attention weight, and multiply the initial feature map with the spatial attention weight and the temporal attention weight to obtain an enhanced feature map; The Euclidean distance between the hand key points and the trunk key points in the posture skeleton is calculated to construct the hand-torso distance matrix. The speed, acceleration and direction angle of the hand movement are calculated based on the continuous frame position information of the hand key points to obtain the hand movement trajectory descriptor. A sliding time window is used to perform feature aggregation on the enhanced feature map, the temporal correlation of the feature sequence in the time window is calculated, and the feature sequence is weighted and summed with the time weight to obtain the time window feature; The enhanced feature map, the hand-torso distance matrix, the hand motion trajectory descriptor and the time window feature are concatenated to construct a multimodal feature vector, and the multimodal feature vector is simultaneously input into a multilayer perceptron and a bidirectional long short-term memory network for feature fusion. The fused features are passed through a softmax classifier to obtain the behavior category probability distribution, and the classification model is optimized and trained based on the cross entropy loss function and the regularization term to obtain the final behavior classification result.

[0050] This paper proposes a behavior recognition method based on multimodal feature fusion, which combines wavelet transform, spatiotemporal attention mechanism, posture skeleton analysis and deep learning technology to achieve accurate recognition of behavior targets. The specific implementation steps are as follows: First, extract features from the facial region of the behavioral target. Use multi-scale two-dimensional discrete wavelet transform to process the facial region image and obtain wavelet coefficients of different scales. Specifically, you can choose Haar wavelet or Daubechies wavelet as the basis function to decompose the image into 3 or 4 layers. For each scale, calculate the wavelet energy value and determine the energy proportion. For example, for a facial image of 256x256 pixels, after 3 layers of wavelet decomposition, 10 sub-band images can be obtained. Calculate the energy value of each sub-band image and normalize it to obtain the energy proportion. Combine these features into an initial feature map, whose dimension may be 10x32x32.

[0051] Next, the initial feature map is fed into a two-stream spatiotemporal attention network for enhancement. In the spatial attention branch, the initial feature map is average pooled and max pooled to obtain two 32x32 feature maps, which are then concatenated and passed through a 1x1 convolution layer to obtain the spatial attention weights. In the temporal attention branch, the initial feature map of multiple consecutive frames is processed using 3D convolution, for example, using 16 frames of 3x3x3 convolution kernels with a stride of 1 to obtain the temporal attention weights. The initial feature map is multiplied by the two attention weights to obtain an enhanced feature map, whose dimension remains unchanged at 10x32x32.

[0052] At the same time, the posture skeleton is analyzed. The Euclidean distance between the key points of the hand (such as the center of the wrist and palm) and the key points of the trunk (such as the center of the shoulder and chest) is calculated to construct the hand-torso distance matrix. For example, for each frame of the image, a 4x2 distance matrix can be obtained, which represents the distance between the left and right hands and the shoulder and chest. Based on the position information of the key points of the hand in consecutive frames, the speed, acceleration and direction angle of the hand movement are calculated. Specifically, a sliding window of 5 frames can be used to calculate the displacement, speed change and angle change of the center frame relative to the previous and next frames to obtain a 15-dimensional hand motion trajectory descriptor.

[0053] In order to capture the temporal information, a sliding time window is used to aggregate the features of the enhanced feature map. 16 frames are selected as the time window size, and the temporal correlation is calculated for the feature sequence in each time window. The self-attention mechanism or temporal convolutional network can be used to model the temporal relationship. Then, the feature sequence is weighted and summed with the learned temporal weight to obtain a 160-dimensional time window feature.

[0054] The enhanced feature map (10x32x32), hand-torso distance matrix (4x2), hand motion trajectory descriptor (15 dimensions) and time window feature (160 dimensions) are concatenated to construct a multimodal feature vector. This feature vector contains information such as facial micro-expressions, body posture and motion trajectory.

[0055] Finally, the multimodal feature vectors are simultaneously input into a multilayer perceptron (MLP) and a bidirectional long short-term memory network (Bi-LSTM) for feature fusion. The MLP can contain three fully connected layers, with 512, 256, and 128 neurons in each layer. The Bi-LSTM can use 128 hidden units. The output features of the MLP and Bi-LSTM are concatenated, reduced to 64 dimensions through a fully connected layer, and then input into a softmax classifier to obtain the probability distribution of the behavior categories.

[0056] In the training phase, the cross entropy loss function is used as the objective function, and the L2 regularization term is added to prevent overfitting. The Adam optimizer is used to update the model parameters, and the initial learning rate is set to 0.001, which decays by 10% every 50 epochs. The training dataset can contain a variety of daily behaviors, such as walking, running, sitting, standing up, etc. Each type of behavior contains at least 1000 samples. After achieving the best performance on the validation set, the model parameters are saved as the final behavior classification model.

[0057] This application can achieve: By combining wavelet transform and spatiotemporal attention mechanism, this method can effectively extract micro-expression features in facial areas, capture subtle changes in expression, and improve the accuracy and robustness of behavior recognition.

[0058] By introducing posture skeleton analysis and hand motion trajectory description, this method can comprehensively characterize the motion characteristics of the human body, not only considering static posture but also including dynamic motion information, so as to more accurately distinguish different types of behaviors.

[0059] By adopting multimodal feature fusion and deep learning models, this method can adaptively learn the correlation between different features, make full use of various information sources, and achieve more intelligent and accurate behavior recognition, which is suitable for complex practical application scenarios.

[0060] In an optional implementation, the behavior feature vector is input into the theft behavior scoring mechanism analysis engine for behavior determination, and the theft behavior scoring mechanism analysis engine integrates a convolutional neural network and a Transformer structure, calculates the similarity between the behavior feature vector and a preset behavior template, and generates a confidence score, including: Reorganize the input behavior feature vector into a spatiotemporal feature matrix, embed sine position coding and cosine position coding into the spatiotemporal feature matrix to obtain a coded spatiotemporal feature matrix; Using convolution branches with different convolution kernel sizes to perform parallel convolution operations on the encoded spatiotemporal feature matrix to obtain multi-scale convolution features, and performing weighted fusion on the multi-scale convolution features to obtain fused convolution features; Performing query matrix transformation, key matrix transformation and value matrix transformation on the encoded spatiotemporal feature matrix to obtain a query matrix, a key matrix and a value matrix, calculating an attention weight based on the product of the query matrix and the key matrix, multiplying the attention weight by the value matrix to obtain an attention feature, and concatenating a plurality of the attention features to obtain a multi-head attention output feature; Calculate the cosine similarity between the fused convolution feature and the preset behavior template to obtain a similarity feature, and perform dynamic time warping calculation on the fused convolution feature and the preset behavior template to obtain a time warping distance feature; Performing weighted summation on the fused convolution feature, the multi-head attention output feature, the similarity feature and the time-warped distance feature to obtain an integrated feature, and performing a sigmoid activation operation on the integrated feature to obtain an initial confidence score; The initial confidence score is subjected to exponential smoothing to obtain a smoothed confidence score, the temporal consistency of the smoothed confidence score is calculated, and a weighted sum of the smoothed confidence score and the temporal consistency is taken as a final theft behavior determination result.

[0061] The present invention provides a theft behavior scoring mechanism analysis engine, which combines convolutional neural networks and Transformer structures to calculate the similarity between behavior feature vectors and preset behavior templates and generate confidence scores. The specific implementation process is as follows: First, the input behavior feature vector is reorganized into a spatiotemporal feature matrix. Assuming that the input behavior feature vector contains 100 features, it can be reorganized into a 10x10 spatiotemporal feature matrix. Sine position coding and cosine position coding are embedded in this matrix to enhance the model's perception of feature positions. Specifically, position coding can be generated using sine and cosine functions of different frequencies, and then added to the original feature matrix to obtain the encoded spatiotemporal feature matrix.

[0062] Next, convolution branches with different convolution kernel sizes are used to perform parallel convolution operations on the encoded spatiotemporal feature matrix to obtain multi-scale convolution features. For example, 3x3, 5x5, and 7x7 convolution kernels can be used to perform convolution operations on the feature matrix respectively, and the number of output channels of each convolution kernel can be set to 64. Then, these multi-scale convolution features are weighted fused to obtain fused convolution features. Weighted fusion can be achieved by learning the weight coefficients of each scale, such as using a 1x1 convolution layer to learn the weights.

[0063] At the same time, the encoded spatiotemporal feature matrix is ​​transformed into a query matrix, a key matrix, and a value matrix respectively to obtain a query matrix, a key matrix, and a value matrix. These transformations can be implemented through a fully connected layer, and the output dimension can be set to 64. The attention weight is calculated based on the product of the query matrix and the key matrix, and the attention feature is obtained by multiplying the attention weight with the value matrix. Eight attention heads can be set, each with a dimension of 8. Finally, multiple attention features are concatenated to obtain a multi-head attention output feature with a dimension of 64.

[0064] Then, the cosine similarity between the fused convolution feature and the preset behavior template is calculated to obtain the similarity feature. The preset behavior template can be a set of typical theft behavior feature vectors. By calculating the similarity between the current feature and these templates, the abnormality of the behavior can be judged. At the same time, dynamic time warping is performed on the fused convolution feature and the preset behavior template to obtain the time warping distance feature. Dynamic time warping can handle the comparison between sequences of different lengths and can capture local deformations and global trends in time series.

[0065] The fused convolutional features, multi-head attention output features, similarity features, and time-warped distance features are weighted and summed to obtain integrated features. The weight coefficients can be learned during model training and can be initially set to equal weights. The integrated features are activated by sigmoid to obtain the initial confidence score, which ranges from 0 to 1.

[0066] In order to improve the stability of the judgment result, the initial confidence score is exponentially smoothed to obtain a smoothed confidence score. Exponential smoothing can select an appropriate smoothing coefficient, such as 0.8, to balance the impact of historical data and current data. Then the temporal consistency of the smoothed confidence score is calculated, which can be measured by calculating the variance of the confidence scores at multiple consecutive time points. Finally, the weighted sum of the smoothed confidence score and the temporal consistency is used as the final theft judgment result. The weighting coefficient can be adjusted according to the actual application scenario. For example, the weight of the confidence score can be set to 0.7 and the weight of the temporal consistency can be set to 0.3.

[0067] In actual applications, a threshold value, such as 0.8, can be set. When the final judgment result exceeds this threshold value, the system will issue a theft alarm. At the same time, different response strategies can be set according to different score intervals. For example, when the score is between 0.6-0.8, it will be monitored closely, and when the score is below 0.6, no action will be taken.

[0068] The theft scoring mechanism analysis engine of the present invention can achieve: First, by integrating the convolutional neural network and the Transformer structure, the advantages of the two models are fully utilized. The convolutional neural network can effectively extract local features and multi-scale information, while the Transformer's self-attention mechanism can capture long-distance dependencies, enabling the model to comprehensively analyze behavioral characteristics and improve the accuracy and robustness of theft behavior recognition.

[0069] Secondly, dynamic time warping and exponential smoothing are introduced to effectively deal with the inconsistency and noise problems of time series data. Dynamic time warping can adapt to behavioral sequences of different lengths, while exponential smoothing can smooth short-term fluctuations and improve the model's sensitivity to long-term trends, thereby enhancing the stability and reliability of theft behavior judgment.

[0070] Finally, a multi-feature fusion and weighting strategy was adopted, which comprehensively considered convolutional features, attention features, similarity features, and time regularization features, and introduced temporal consistency evaluation to make the judgment results more comprehensive and reliable. This multi-dimensional analysis method greatly reduced the misjudgment rate and improved the model's ability to identify complex theft behaviors, providing a more reliable basis for decision-making in practical applications.

[0071] Figure 2 FIG. 1 is a schematic diagram of the structure of the theft behavior recognition system based on computer vision according to an embodiment of the present invention. Figure 2 As shown, the system comprises: The first unit is used to collect a video data stream, perform frame rate conversion on the video data stream to obtain a video stream with a target frame rate, apply Gaussian filtering and median filtering to the video stream with the target frame rate to eliminate noise, and use a threshold segmentation algorithm to perform image segmentation on the filtered video stream to obtain a preprocessed video frame; The second unit is used to use the YOLOv11 deep learning target detection algorithm to perform target detection and positioning on the preprocessed video frame to obtain the bounding box information of the behavior target, input the bounding box information into the feature pyramid network to perform multi-scale feature extraction, and fuse the high-level semantic features and low-level detail features through the path aggregation network to obtain the target feature map; based on the target feature map, the Openpose algorithm is used to extract the coordinate data of the hand key points and the trunk key points of the behavior target, and the partial affinity field technology is used to establish the association relationship between the hand key points and the trunk key points to construct a posture skeleton, and the wavelet transform and spatiotemporal attention mechanism are used to extract the features of the facial area of ​​the behavior target to obtain micro-expression features, and the posture skeleton and the micro-expression features are input into the deep learning decision classification model, and the behavior feature vector containing the distance data between the hand and the trunk, the hand movement trajectory data, and the behavior time window data is output; The third unit is used to input the behavior feature vector into the theft behavior scoring mechanism analysis engine for behavior judgment. The theft behavior scoring mechanism analysis engine integrates the convolutional neural network and the Transformer structure, calculates the similarity between the behavior feature vector and the preset behavior template and generates a confidence score. When the confidence score is greater than 0.8, it is judged as theft behavior and triggers an alarm signal. When the confidence score is between 0.5 and 0.8, it is judged as abnormal behavior and a tracking mark is recorded. When the confidence score is less than 0.5, it is judged as normal behavior.

[0072] According to a third aspect of the embodiments of the present invention, An electronic device is provided, comprising: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0073] According to a fourth aspect of the embodiments of the present invention, A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0074] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A theft behavior recognition method based on computer vision, characterized in that: include: Collecting a video data stream, performing frame rate conversion on the video data stream to obtain a video stream with a target frame rate, applying Gaussian filtering and median filtering to the video stream with the target frame rate to eliminate noise, and performing image segmentation on the filtered video stream using a threshold segmentation algorithm to obtain a preprocessed video frame; Using the YOLOv11 deep learning target detection algorithm to perform target detection and positioning on the preprocessed video frame to obtain bounding box information of the behavioral target, inputting the bounding box information into a feature pyramid network to extract multi-scale features, and fusing high-level semantic features and low-level detail features through a path aggregation network to obtain a target feature map; Based on the target feature map, the Openpose algorithm is used to extract the coordinate data of the hand key points and the trunk key points of the behavior target, and the association relationship between the hand key points and the trunk key points is established by using the partial affinity field technology to construct a posture skeleton, and the wavelet transform and spatiotemporal attention mechanism are used to extract the features of the facial area of ​​the behavior target to obtain micro-expression features, and the posture skeleton and the micro-expression features are input into the deep learning decision classification model, and the behavior feature vector containing the distance data between the hand and the trunk, the hand movement trajectory data, and the behavior time window data is output; The behavior feature vector is input into the theft behavior scoring mechanism analysis engine for behavior judgment. The theft behavior scoring mechanism analysis engine integrates convolutional neural network and Transformer structure, calculates the similarity between the behavior feature vector and the preset behavior template and generates a confidence score. When the confidence score is greater than 0.8, it is judged as theft behavior and triggers an alarm signal. When the confidence score is between 0.5 and 0.8, it is judged as abnormal behavior and a tracking mark is recorded. When the confidence score is less than 0.5, it is judged as normal behavior.

2. The method according to claim 1, characterized in that: The preprocessed video frames are subjected to target detection and positioning using the YOLOv11 deep learning target detection algorithm to obtain bounding box information of the behavioral target, and the bounding box information is input into the feature pyramid network for multi-scale feature extraction and the target feature map is obtained by fusing high-level semantic features and low-level detail features through the path aggregation network, including: Divide the preprocessed video frame into a number of grid units, perform bounding box prediction on the grid units, and obtain the center coordinates, width, height and confidence of the bounding box, where the confidence is calculated by multiplying the probability of the object existing in the box by the intersection and union ratio of the predicted box and the real box; A feature pyramid network is constructed based on the bounding box, the preprocessed video frame is downsampled by a convolution layer to obtain five bottom-level feature maps of different scales, the sizes of the five bottom-level feature maps of different scales are halved successively, the high-level feature map is upsampled and one-dimensionally convolved with the horizontally connected feature map of the same scale to obtain a pyramid feature map; Input the pyramid feature map into a path aggregation network, construct a top-down enhancement path and a bottom-up enhancement path in the path aggregation network, the top-down enhancement path obtains a top-down feature map through upsampling and feature concatenation, the bottom-up enhancement path obtains a bottom-up feature map through downsampling and feature concatenation, and the top-down feature map and the bottom-up feature map are fused through a learnable fusion weight coefficient to obtain a fused feature map; The fused feature map is respectively subjected to attention enhancement in the channel dimension and the spatial dimension. The attention enhancement in the channel dimension is performed by performing average pooling and maximum pooling on the fused feature map and obtaining a channel attention weight through a multilayer perceptron. The attention enhancement in the spatial dimension is performed by performing average pooling and maximum pooling on the fused feature map and obtaining a spatial attention weight through a convolution operation. The fused feature map is multiplied by the channel attention weight and the spatial attention weight to obtain an attention enhancement feature map. The information entropy of the attention enhanced feature map at different scales is calculated, and the dynamic fusion weights of different scales are calculated based on the information entropy. The attention enhanced feature map and the corresponding dynamic fusion weights are weightedly summed to obtain a final feature map, wherein the final feature map contains multi-scale semantic information and detail features of the target.

3. The method according to claim 1, characterized in that Based on the target feature graph, the coordinate data of the hand key points and the trunk key points of the behavior target are extracted by using the Openpose algorithm, and the association relationship between the hand key points and the trunk key points is established by using the partial affinity field technology to construct the posture skeleton, including: Based on the target feature map, a two-branch convolutional neural network is used to generate a confidence map through the first branch, and a partial affinity field is generated through the second branch. The distance from the pixel point in the target feature map to the real position of the key point is calculated using a Gaussian kernel function to obtain an initial confidence map of multi-level key points, and the candidate positions of the key points of multiple human targets are determined based on the initial confidence map; Based on the candidate positions of the key points, the initial coordinates of the wrist key points are obtained, the hand region is adaptively cropped with the initial coordinates of the wrist key points as the center to obtain the hand region of interest, the hand region of interest is input into the hand key point extraction network, a hand key point heat map is generated through multi-layer convolution operations, and the accurate coordinates of the hand fine key points are extracted from the hand key point heat map; A hierarchical detection strategy is adopted for the trunk key points in the key point candidate positions, a reference position of the trunk main key point is determined based on the maximum response value of the initial confidence map, the reference position is input into a position regression network to calculate the spatial offset of the secondary key point relative to the reference position, and the precise position of the trunk secondary key point is obtained according to the spatial offset and the reference position; A bidirectional partial affinity field is constructed based on the accurate coordinates of the hand fine key points and the precise positions of the trunk secondary key points, a feature vector integral value of the forward partial affinity field is calculated along the key point connection direction, a feature vector integral value of the reverse partial affinity field is calculated along the reverse direction of the key point connection, and the forward partial affinity field and the reverse partial affinity field are fused through adaptive direction weights to obtain an enhanced partial affinity field; The enhanced partial affinity field is used to calculate the connection relationship scores between adjacent key points, and the optimal skeleton topology structure is obtained by maximizing the global accumulation of the connection relationship scores. A temporal smoothing constraint is applied to the optimal skeleton topology structure, and the current frame posture is adaptively weighted fused with the previous frame historical posture and motion predicted posture to obtain a smoothed posture skeleton that is consistent in time and space. The smoothed posture skeleton contains a stable connection relationship between the hand and torso key points.

4. The method according to claim 3, characterized in that A hierarchical detection strategy is adopted for the trunk key points in the key point candidate positions, a reference position of the trunk trunk key points is determined based on the maximum response value of the initial confidence map, the reference position is input into the position regression network to calculate the spatial offset of the secondary key points relative to the reference position, and the precise position of the trunk secondary key points is obtained according to the spatial offset and the reference position, including: Performing weighted summation on the heat map responses of multiple channels to obtain a confidence response map, setting a response threshold based on the confidence response map to screen and obtain a candidate region of a backbone key point, and performing Gaussian smoothing on the candidate region of the backbone key point to obtain a smoothed response map; Determine the maximum response position in the smooth response map as the reference position of the backbone key point, calculate the sub-pixel offset using the response difference in the horizontal direction and the vertical direction, and add the reference position and the sub-pixel offset to obtain the refined backbone key point coordinates; Construct a multi-level feature pyramid network, perform upsampling and convolution operations on the input feature map to obtain feature maps of different levels, cascade the feature maps of different levels into a position regression network, and predict the spatial offset of the secondary key point relative to the refined main key point coordinates; Establishing bone length constraints and joint angle constraints based on human anatomical features, adding the refined trunk key point coordinates to the spatial offset to obtain the secondary key point initial position, and performing constraint optimization on the secondary key point initial position to obtain the optimized secondary key point position that satisfies the bone length constraints and the joint angle constraints; A temporal smoothness constraint is introduced to calculate the displacement difference between the secondary key point position of the current frame and the secondary key point position of the previous two frames, and the movement speed difference between the current frame and the previous frame. The spatial positioning error, displacement difference and speed difference are used as optimization targets, and the timing consistency optimization is performed on the optimized secondary key point position to obtain the final precise position of the secondary key point.

5. The method according to claim 1, characterized in that Wavelet transform and spatiotemporal attention mechanism are used to extract features from the facial area of ​​the behavior target to obtain micro-expression features, and the posture skeleton and the micro-expression features are input into a deep learning decision classification model, and the output behavior feature vector including the distance data between the hand and the torso, the hand movement trajectory data, and the behavior time window data includes: Performing a multi-scale two-dimensional discrete wavelet transform on the facial region image of the behavioral target to obtain wavelet coefficients of different scales, calculating the wavelet energy value at each scale based on the wavelet coefficients and determining the energy proportion, thereby obtaining an initial feature map of the facial region; Input the initial feature map into a dual-stream spatiotemporal attention network, perform average pooling and maximum pooling on the initial feature map through the spatial attention branch to obtain a spatial attention weight, perform a three-dimensional convolution operation on the initial feature map through the temporal attention branch to obtain a temporal attention weight, and multiply the initial feature map with the spatial attention weight and the temporal attention weight to obtain an enhanced feature map; The Euclidean distance between the hand key points and the trunk key points in the posture skeleton is calculated to construct the hand-torso distance matrix. The speed, acceleration and direction angle of the hand movement are calculated based on the continuous frame position information of the hand key points to obtain the hand movement trajectory descriptor. A sliding time window is used to perform feature aggregation on the enhanced feature map, the temporal correlation of the feature sequence in the time window is calculated, and the feature sequence is weighted and summed with the time weight to obtain the time window feature; The enhanced feature map, the hand-torso distance matrix, the hand motion trajectory descriptor and the time window feature are concatenated to construct a multimodal feature vector, and the multimodal feature vector is simultaneously input into a multilayer perceptron and a bidirectional long short-term memory network for feature fusion. The fused features are passed through a softmax classifier to obtain the behavior category probability distribution, and the classification model is optimized and trained based on the cross entropy loss function and the regularization term to obtain the final behavior classification result.

6. The method according to claim 1, characterized in that Inputting the behavior feature vector into the theft behavior scoring mechanism analysis engine for behavior determination, the theft behavior scoring mechanism analysis engine integrates the convolutional neural network and the Transformer structure, calculates the similarity between the behavior feature vector and the preset behavior template and generates a confidence score including: Reorganize the input behavior feature vector into a spatiotemporal feature matrix, embed sine position coding and cosine position coding into the spatiotemporal feature matrix to obtain a coded spatiotemporal feature matrix; Using convolution branches with different convolution kernel sizes to perform parallel convolution operations on the encoded spatiotemporal feature matrix to obtain multi-scale convolution features, and performing weighted fusion on the multi-scale convolution features to obtain fused convolution features; Performing query matrix transformation, key matrix transformation and value matrix transformation on the encoded spatiotemporal feature matrix to obtain a query matrix, a key matrix and a value matrix, calculating an attention weight based on the product of the query matrix and the key matrix, multiplying the attention weight by the value matrix to obtain an attention feature, and concatenating a plurality of the attention features to obtain a multi-head attention output feature; Calculate the cosine similarity between the fused convolution feature and the preset behavior template to obtain a similarity feature, and perform dynamic time warping calculation on the fused convolution feature and the preset behavior template to obtain a time warping distance feature; Performing weighted summation on the fused convolution feature, the multi-head attention output feature, the similarity feature and the time-warped distance feature to obtain an integrated feature, and performing a sigmoid activation operation on the integrated feature to obtain an initial confidence score; The initial confidence score is subjected to exponential smoothing to obtain a smoothed confidence score, the temporal consistency of the smoothed confidence score is calculated, and a weighted sum of the smoothed confidence score and the temporal consistency is taken as a final theft behavior determination result.

7. A theft behavior recognition system based on computer vision, used to implement the method according to any one of claims 1 to 6, characterized in that: include: The first unit is used to collect a video data stream, perform frame rate conversion on the video data stream to obtain a video stream with a target frame rate, apply Gaussian filtering and median filtering to the video stream with the target frame rate to eliminate noise, and use a threshold segmentation algorithm to perform image segmentation on the filtered video stream to obtain a preprocessed video frame; The second unit is used to detect and locate the target on the preprocessed video frame using the YOLOv11 deep learning target detection algorithm to obtain the bounding box information of the behavior target, input the bounding box information into the feature pyramid network to extract multi-scale features, and fuse the high-level semantic features and low-level detail features through the path aggregation network to obtain the target feature map; Based on the target feature map, the Openpose algorithm is used to extract the coordinate data of the hand key points and the trunk key points of the behavior target, and the association relationship between the hand key points and the trunk key points is established by using the partial affinity field technology to construct a posture skeleton, and the wavelet transform and spatiotemporal attention mechanism are used to extract the features of the facial area of ​​the behavior target to obtain micro-expression features, and the posture skeleton and the micro-expression features are input into the deep learning decision classification model, and the behavior feature vector containing the distance data between the hand and the trunk, the hand movement trajectory data, and the behavior time window data is output; The third unit is used to input the behavior feature vector into the theft behavior scoring mechanism analysis engine for behavior judgment. The theft behavior scoring mechanism analysis engine integrates the convolutional neural network and the Transformer structure, calculates the similarity between the behavior feature vector and the preset behavior template and generates a confidence score. When the confidence score is greater than 0.8, it is judged as theft behavior and triggers an alarm signal. When the confidence score is between 0.5 and 0.8, it is judged as abnormal behavior and a tracking mark is recorded. When the confidence score is less than 0.5, it is judged as normal behavior.

8. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images