Crossing behavior detection method based on fusion features and related device

Abstract

The application discloses a method and device for detecting climbing behavior based on fused features. The method comprises the following steps: performing target detection on an image sample by using a Yolov5 network to obtain a pedestrian detection frame and a gate detection frame; then sequentially passing through a graph attention network, an OpenPose algorithm and a camera model to obtain pedestrian appearance features, gate appearance features and spatial features of the pedestrian and the gate; finally, based on the pedestrian appearance features, the gate appearance features and the spatial features of the pedestrian and the gate, performing climbing behavior detection on the pedestrian and the gate by using a multilayer perceptron to obtain a detection result. The application is based on the Yolov5 network, the graph attention network, the OpenPose algorithm and the camera model, fully extracts the features of the pedestrian and the gate in the detection image, fully considers the spatial relationship between the pedestrian and the gate compared with the prior art, and greatly improves the interaction behavior detection accuracy of the pedestrian and the gate.

Description

A method and related device for detecting climbing behavior based on fused features Technical Field

[0001] The embodiments of the present invention relate to the field of visual detection technology, and in particular to a method and apparatus for detecting vaulting behavior based on fused features. Background Technology

[0002] Security and prevention work is mainly carried out by humans through real-time video surveillance to identify abnormal behavior, which consumes a lot of human resources and is inefficient. The traditional video surveillance management method that relies on human decision-making is clearly unsustainable. Instead, intelligent video surveillance technology with artificial intelligence, computer vision, and deep learning as its core has emerged.

[0003] Climbing over turnstiles refers to the act of entering a premises by climbing over turnstiles or other security facilities without proper legal procedures. This behavior not only violates the premises' safety regulations but may also cause harm to the individual. Currently, to prevent pedestrians from climbing over turnstiles, gravity sensors are installed to detect human weight and height. If someone attempts to climb over the turnstile, an alarm is automatically triggered and the system stops operating. Infrared sensors are also used to detect human position and movement; if someone is detected attempting to climb over or cross the turnstile, an alarm is automatically triggered and the system stops operating. While these methods can detect illegal turnstile climbing, they cannot accurately identify the specific unauthorized entrant, making subsequent tracking difficult. Summary of the Invention

[0004] This invention provides a method and related apparatus for detecting trespassing behavior based on fused features, aiming to solve the technical problem that the prior art cannot accurately detect specific unauthorized trespassers.

[0005] In a first aspect, embodiments of the present invention provide a method for detecting vaulting behavior based on fused features, comprising:

[0006] The video stream data of the gate entrance and exit monitoring camera is acquired, and the video stream data is processed by frame extraction to obtain an image sample set, the detection image set including multiple image samples;

[0007] The image samples are input into a preset Yolov5 network in chronological order for target detection to obtain corresponding detection images, which include pedestrian detection boxes and gate detection boxes.

[0008] The similarity of pedestrian detection boxes in two adjacent detection images is calculated. Based on the calculation results, it is determined whether the pedestrians in the pedestrian detection boxes in the two adjacent detection images are the same person, and the pedestrian detection boxes of the same person are marked accordingly.

[0009] The pedestrian detection box and the gate detection box in the detection image are learned by a pre-trained graph attention network, and the learning results are used to perform relational reasoning to obtain the appearance features of the pedestrian and the gate.

[0010] The OpenPose algorithm is used to detect key points of pedestrians in the pedestrian detection box to obtain the coordinates of key points of various parts of the human body. The distance between each part of the human body and the gate is calculated based on the camera model and the coordinates of the key points. The calculation results are converted into vectors and then merged to obtain the spatial features of the pedestrian and the gate.

[0011] Based on a pre-trained multilayer perceptron and the pedestrian appearance features, gate appearance features, and spatial features, the system detects the climbing behavior of pedestrians and gates in the detected image, and obtains the detection results.

[0012] Secondly, embodiments of the present invention provide a vaulting behavior detection device based on fused features, comprising:

[0013] The frame extraction module is used to acquire video stream data from the monitoring camera at the gate entrance and exit, and to perform frame extraction processing on the video stream data to obtain an image sample set. The detection image set includes multiple image samples.

[0014] The target detection module is used to input the image samples into a preset Yolov5 network in chronological order to perform target detection and obtain the corresponding detection image, which includes pedestrian detection boxes and gate detection boxes.

[0015] The similarity detection module is used to calculate the similarity of pedestrian detection boxes in two adjacent detection images, determine whether the pedestrians in the pedestrian detection boxes in the two adjacent detection images are the same person based on the calculation results, and mark the pedestrian detection boxes of the same person accordingly.

[0016] The appearance feature extraction module is used to learn multi-attention features of the pedestrian detection box and the gate detection box in the detection image through a pre-trained graph attention network, and to perform relational reasoning on the learning results to obtain the appearance features of the pedestrian and the gate.

[0017] The spatial feature extraction module is used to perform key point detection on pedestrians in the pedestrian detection box using the OpenPose algorithm, obtain the key point coordinates of each part of the human body, calculate the distance between each part of the human body and the gate based on the camera model and the key point coordinates, and then convert the calculation results into vectors and merge them to obtain the spatial features of pedestrians and gates.

[0018] The climbing behavior detection module is used to detect climbing behavior of pedestrians and turnstiles in the detection image based on a pre-trained multilayer perceptron and the appearance features of pedestrians, turnstiles, and spatial features, and to obtain detection results.

[0019] Thirdly, embodiments of the present invention provide a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the hop-over behavior detection method based on fused features described in the first aspect.

[0020] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, causes the processor to perform the hop-over behavior detection method based on fused features described in the first aspect.

[0021] This invention provides a method and apparatus for detecting climbing behavior based on fused features. The method utilizes a YOLOv5 network to perform target detection on image samples, obtaining pedestrian detection boxes and turnstile detection boxes. Then, it sequentially passes through a graph attention network, the OpenPose algorithm, and a camera model to obtain pedestrian appearance features, turnstile appearance features, and spatial features between pedestrians and turnstiles, respectively. Finally, based on these features, a multilayer perceptron is used to detect climbing behavior between pedestrians and turnstiles, yielding the detection results. This application, based on a YOLOv5 network, graph attention network, OpenPose algorithm, and camera model, fully extracts the features of pedestrians and turnstiles in the detected image. Compared to existing methods, it fully considers the spatial relationship between pedestrians and turnstiles, significantly improving the accuracy of detecting pedestrian-turntile interaction behavior. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 is a flowchart illustrating an embodiment of the vaulting behavior detection method based on fused features provided by the present invention.

[0024] Figure 2 is a schematic diagram of a sub-process of step S140 in Figure 1;

[0025] Figure 3 is a schematic diagram of false detections in the vaulting behavior detection method based on fused features provided in the embodiment of the present invention;

[0026] Figure 4 is a schematic diagram of a climbing behavior detection device based on fused features provided in an embodiment of the present invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0029] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0030] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0031] Please refer to Figure 1, which is a flowchart of the method for detecting vaulting behavior based on fused features provided in an embodiment of the present invention. The method includes steps S110 to S160.

[0032] Step S110: Obtain video stream data from the gate entrance and exit monitoring camera, perform frame extraction processing on the video stream data to obtain an image sample set, the detection image set including multiple image samples;

[0033] In this embodiment, surveillance cameras installed at the gate entrance and exit acquire video stream data of pedestrians passing through the gate. After frame extraction processing of the video stream data, multiple image samples are obtained, thus obtaining an image sample set. The video stream data can be obtained from locally stored historical surveillance videos or extracted from real-time surveillance videos. For example, frame extraction processing is performed on the video stream data at 0.2-second intervals to obtain multiple image samples.

[0034] Step S120: Input the image samples into a preset Yolov5 network in chronological order for target detection to obtain the corresponding detection image, which includes pedestrian detection boxes and gate detection boxes.

[0035] In this embodiment, to better consider the temporal sequence of images and determine the positions of pedestrians and turnstiles in the image samples, the image samples are input into a pre-trained Yolov5 network for target detection based on the generation time of each frame in the image sample set, in chronological order, to obtain the corresponding detection images. The Yolov5 network mainly consists of a backbone network, a Neck network, and prediction layers. The backbone network comprises a Focus module, a CBL module, a CSP module, and an SPP (Spatial Pyramid Pooling) module. The Focus module samples image samples at equal intervals and then performs channel concatenation. The CBL module is an encapsulated convolutional module consisting of convolutional (Conv) layers, normalization (BN) layers, and an activation function (Leaky ReLU). The CSP module consists of a CBL module, a residual module, and convolutional layers. The SPP (Spatial Pyramid Pooling) module uses 1×1, 5×5, 9×9, and 13×13 max pooling, respectively. The Neck network is a pyramid structure based on the FPN framework. The prediction layer outputs the category information, coordinate information, and confidence score of the object within the detection box.

[0036] Step S130: Calculate the similarity of pedestrian detection boxes in two adjacent detection images, determine whether the pedestrians in the pedestrian detection boxes in the two adjacent detection images are the same person based on the calculation results, and mark the pedestrian detection boxes of the same person accordingly.

[0037] In this embodiment, considering that the pedestrian features in adjacent frames do not change significantly, a structural similarity calculation method is used to simulate human visual perception features, which is more consistent with the human visual system compared to other traditional methods. The similarity of pedestrian detection boxes in two adjacent detection images is calculated. A higher similarity indicates that the pedestrians within the detection boxes in the two images are more similar. When the similarity exceeds a preset threshold, it indicates that the pedestrians in the two detection boxes are the same person, and then the same ID number is assigned to the same person. The implementation process of the similarity calculation method is as follows:

[0038] (1) Convert two adjacent detection images into grayscale images and divide them into multiple sub-regions of equal size;

[0039] (2) Calculate the brightness, contrast and structural contrast of each sub-region in the two detection images respectively, and combine the calculated results into a comprehensive index to obtain the first comprehensive index and the second comprehensive index.

[0040] (3) The first and second comprehensive indices are weighted and averaged to obtain the SSIM value of the similarity between two adjacent detected images. The SSIM value is calculated using the following formula:

[0041]

[0042] Where, μ x This represents the average value of graph x. Let μ represent the variance of graph x. y This represents the average value of graph y. σ represents the variance of graph y. xy Let c represent the covariance of graphs x and y, and c1 and c2 both represent constants in the stability formula.

[0043] Step S140: The pedestrian detection box and the gate detection box in the detection image are learned by multi-attention feature learning through a pre-trained graph attention network, and the learning results are used to perform relational reasoning to obtain the appearance features of the pedestrian and the appearance features of the gate.

[0044] In this embodiment, considering that the target detection model generates a large number of pedestrian and turnstile detection boxes, which will cause huge computational redundancy for subsequent interaction detection, since an object can usually interact with multiple people, and a person can also interact with multiple objects at the same time, and there can also be interactions between people and between objects, this application only considers the interaction between pedestrians and turnstiles. Therefore, a graph attention network is used to learn multi-attention features for pedestrian and turnstile detection boxes in the detection image, and relation reasoning is performed on the learning results to obtain pedestrian appearance features and turnstile appearance features, thus filtering out a large number of non-human interaction detection boxes (i.e., between people and between objects).

[0045] As shown in Figure 2, in one embodiment, step S140 includes:

[0046] Step S210: Extract features from pedestrian and turnstile detection boxes in the detected image using a preset backbone extraction network to obtain pedestrian image features and turnstile image features. The backbone extraction network is a ResNet network.

[0047] Step S220: The graph attention network is used to aggregate the features of pedestrians and turnstiles in the pedestrian detection box and turnstile detection box in the detection image set, respectively, to obtain pedestrian aggregate features and turnstile aggregate features.

[0048] Step S230: Construct an initial feature matrix based on the pedestrian aggregation features and the gate aggregation features, and obtain an initial adjacency matrix based on the initial feature matrix;

[0049] Step S240: Calculate the interaction probability between the feature vectors corresponding to pedestrians and turnstiles in the initial adjacency matrix based on the multilayer perceptron and softmax function, and dynamically update the initial adjacency matrix according to the interaction probability to obtain the appearance features of pedestrians and turnstiles.

[0050] In this embodiment, a ResNet (Residual Network) is used as the backbone extraction network. Its basic structure is a deep neural network composed of multiple residual blocks. Each residual block consists of two 3x3 convolutional layers and a cross-layer connection. During training, each residual block learns a residual mapping function, and the output signal of the residual block is added to the input signal before being output to the next residual block. By introducing convolutional layers and residual blocks, the network depth of the backbone extraction network can be increased.

[0051] In this embodiment, to better extract the appearance features of pedestrians and turnstiles, a graph attention network is used to extract these features because it utilizes graph structure and node features to update node representations, thus effectively extracting feature information. Specifically, the graph is defined as G = (V, E), consisting of a set of V nodes and E edges. Using F... v The characteristics of a node are represented by F. e Describe the characteristics of an edge. Let v i Let i represent the i-th node in the graph, e ij =(v i ,v j ) represents node v i Pointing to node v j The edge of node v. i The feature vector is represented as Construct nodes to detect pedestrians and turnstiles in the image. For a given node v i Its characteristics can be expressed as Features of its neighboring nodes First, a learnable linear mapping W is applied. Then, the mapped features are input into k layers of graph convolution and aggregation functions for aggregation. Then, each node v... i aggregation features The calculation formula is as follows:

[0052]

[0053]

[0054] In the formula, This indicates the first initialization calculation, N. i Represents node v i All adjacent points, aij This indicates the preset weights. and This represents the mean function and the vector summation function;

[0055] Expanding and combining the above equations, the node representations are integrated into a matrix form, yielding the following equation:

[0056]

[0057] In the formula, σ represents the nonlinear mapping function ReLU. node v i Not adjacent node v j When connected, a ij The value is 0; when node v i With neighboring node v j During connection, a ij The value is 1.

[0058] After obtaining the pedestrian aggregation features and turnstile aggregation features using the above method, the pedestrian aggregation features and turnstile aggregation features are merged to obtain matrix F, and the corresponding initial adjacency matrix is ​​E, which can be represented as follows:

[0059]

[0060] Due to the lack of comprehensive supervised relationship labels, the model struggles to directly compute the interaction connections between pedestrians and turnstiles. Therefore, an attention mechanism is introduced to implicitly learn the interaction correlations between pedestrians and turnstiles, using a... o,h This indicates that a is used simultaneously. o,h The matrix E is dynamically updated to ensure sufficient information exchange between correlated pedestrians and turnstiles. To calculate a... o,h Using a multilayer perceptron R r (f o ,f h The features of pedestrians and turnstiles are calculated, and the classification result of the multilayer perceptron is output. The expression is:

[0061] S r =R r (f0,f h ),

[0062] In the formula f o Indicates the appearance characteristics of the turnstile, f h The pedestrian appearance features are represented, and the classification results are normalized using the softmax function to obtain the interaction probability, as shown in the following formula:

[0063]

[0064] In the formula, a o,h This represents the interaction correlation between pedestrians and turnstiles, where N represents the number of interactions between pedestrians and turnstiles in an image;

[0065] Calculate a o,h Then the dynamically updated adjacency matrix can be obtained, and then f can be updated. o and f h The corresponding expression is:

[0066]

[0067]

[0068] Where, N o and N p This represents the number of pedestrians and turnstiles in the image, where k represents the sum of the pedestrian detection boxes and the turnstile detection boxes.

[0069] Furthermore, the loss function during graph attention network training includes classification loss and regression loss, as shown in the following formula:

[0070]

[0071] In the formula, p i This indicates the probability that the bounding box is predicted to be the target. This represents the true label, with 1 indicating the presence of the target and 0 indicating the absence of the target; t i ={t x ,t y ,t w ,t h} is a vector representing the coordinate information of the predicted detection box. Represents the coordinate information of the actual bounding box; The log loss function represents the loss between two categories (target and non-target): This represents the regression loss function, typically N. cls Equals 256, N reg It equals 2400.

[0072] Step S150: Use the OpenPose algorithm to detect key points of pedestrians in the pedestrian detection box, obtain the key point coordinates of each part of the human body, calculate the distance between each part of the human body and the gate based on the camera model and the key point coordinates, and convert the calculation results into vectors and merge them to obtain the spatial features of pedestrians and gates.

[0073] In this embodiment, most existing methods use Euclidean distance to calculate the distance between people and objects in an image. This method takes into account the spatial position information between people and objects in the image, but does not take into account the shooting angle of the camera when shooting the entrance and exit gate. The Euclidean distance between pedestrians and gates in the image cannot truly reflect the distance between real people and gates in the image. Therefore, this application adopts the OpenPose algorithm and proposes to combine human key point detection and camera model to calculate the real distance between human key points and gates in order to obtain the spatial features between pedestrians and gates.

[0074] In one embodiment, the process of human keypoint detection includes: first, inferring about pedestrians in the pedestrian detection box using the OpenPose algorithm to generate heatmaps of human body parts and affinity vectors for all parts; then, continuously iterating over the human body part heatmaps and affinity vectors until the number of iterations reaches a preset condition, and outputting the keypoint coordinates. The inference process using the OpenPose algorithm includes two branches: one branch generates the human body part heatmaps, and the other branch generates affinity vectors for each part of the human body, expressed as follows:

[0075] S = (S1,S2,...,S) J ),

[0076] L = (L1, L2, ..., L C ),

[0077] Where S represents the generated heatmap, L represents the affinity vector, J represents the number of heatmaps, and C represents the number of affinity vectors. Through continuous iteration, the heatmap branch results and affinity vector branch results of the first layer of human body parts can be denoted as S0, S1, and S2, respectively. 1 =ψ 1 (F) and L 1 =φ 1 (F), where ψ 1 and φ 1 Let F represent the derivation of the first stage, and F represent the feature input. Each subsequent iteration is a repeated process, represented as:

[0078]

[0079]

[0080] In this embodiment, the coordinates of key points of various parts of the human body are output after 100 iterations. Considering that pedestrians mostly interact with the turnstile using their hands and legs, this application takes the calculation of the distance between the key point coordinates of the hands and legs and the turnstile as an example.

[0081] Furthermore, the camera model has three coordinate systems: world coordinate system, camera coordinate system, and image coordinate system. The definitions of these three coordinate systems are as follows: World coordinate system P... w (x w ,y w ,z w ), x w ,y w ,z w These represent the x-axis, y-axis, and z-axis of the world coordinate system, respectively; and the camera coordinate system P. c (x c ,y c ,z c ), x c ,y c ,z c These represent the x-axis, y-axis, and z-axis of the camera coordinate system, respectively. The image coordinate system is P0(x0,y0), where x0 and y0 represent the x-axis and y-axis, respectively.

[0082] This application uses the center point of the gate detection frame to represent the position of the gate, and uses a camera model to calculate the distances of the hand and leg from the center point of the gate. The implementation process is as follows:

[0083] (i) Save the image coordinates of the center point of the gate detection frame obtained by target detection, and save the image coordinates of the human hand and leg obtained by human key point detection.

[0084] (ii) Using formulas (1) and (2), map the image coordinate system points of the hand, leg, and gate detection frame center point to the world coordinate system. The formulas are as follows:

[0085] The transformation formula between the world coordinate system and the camera coordinate system is calculated as follows:

[0086]

[0087] In this formula, 'a' represents the angle between the world coordinate system and the camera coordinate system;

[0088] The formula for transforming the camera coordinate system into the image coordinate system is as follows:

[0089]

[0090] In the formula, f represents the focal length.

[0091] (III) According to the distance calculation formula:

[0092]

[0093]

[0094] In the formula x wh y wh z wh This indicates the position information of the hand key points in the x, y, and z axes of the world coordinate system; x wl y wl z wl This indicates the position information of the leg keypoints in the x, y, and z axes of the world coordinate system; x wz y wz z wz This indicates the position information of the center point of the gate detection frame in the x, y, and z axes of the world coordinate system; L h Indicates the distance between the hand and the center point of the gate's detection frame; L l This indicates the distance between the leg and the center point of the gate's detection frame.

[0095] (iv) L h and L l The obtained distance values ​​are vectorized, and then the two vectors are combined and denoted as the spatial feature f. d .

[0096] Step S160: Based on the pre-trained multilayer perceptron and the pedestrian appearance features, gate appearance features and spatial features, perform climbing behavior detection on the pedestrians and gates in the detection image to obtain the detection results.

[0097] In this embodiment, the detection of pedestrians and turnstiles in the detection image involves the following process: First, the appearance features of the pedestrian, the appearance features of the turnstile, and the spatial features are stitched together to obtain stitched features. Then, a multilayer perceptron (MLP) is used to perform weighted fusion and recognition of the stitched features, outputting the recognition probability that the stitched features belong to the climbing behavior, and determining whether the recognition probability is greater than a preset probability threshold. If yes, it is determined that climbing behavior has occurred in the detection image, and the pedestrian has illegally passed through the turnstile; if no, it is determined that climbing behavior has not occurred in the detection image, and the pedestrian has passed through the turnstile normally. The MLP contains three fully connected layers, and the recognition probability P is calculated using the following formula:

[0098] P(M|(b h ,b o ))=σ(MLP(a i f h ,a i f o ,a i f d )),

[0099] Where σ() represents the sigmoid function, f h f0 represents the appearance characteristics of pedestrians, and f0 represents the appearance characteristics of turnstiles.d Indicating spatial characteristics, a i The weights are represented by b, MLP stands for Multilayer Perceptron, and b represents weights. h b0 represents the coordinates of the pedestrian, and b0 represents the coordinates of the turnstile.

[0100] Furthermore, during training, the Multilayer Perceptron (MLP) uses a binary cross-entropy loss function for multi-label classification, as shown in the following formula:

[0101]

[0102]

[0103] In the formula, N represents the interaction pair between the pedestrian and the turnstile, and K represents the number of predicted actions. In this embodiment, only the acrobatic vaulting action is predicted, so K = 1; i Represents a binary tag, y i =1 indicates the label for the vaulting behavior, y i =0 indicates a label that passes normally, p(y i ) indicates that the output belongs to y i The probability of.

[0104] In one embodiment, after step S160, the following is included:

[0105] Step S310: Take a set of consecutive detection images as a detection image set;

[0106] Step S320: Determine whether there is a single frame image in the detection image set whose detection result is different from the detection results of other images;

[0107] Step S330: If the detection result of a single frame image in the detection image set is different from the detection result of other images, then the detection image set is time-series optimized by the triple exponential smoothing method, and it is determined whether there are multiple consecutive single frames in the optimized detection image set exhibiting jump behavior.

[0108] Step S340: If multiple consecutive single-frame images in the optimized detection image set exhibit vaulting behavior, then the multiple single-frame images exhibiting vaulting behavior in the detection image set are stitched together to obtain a vaulting behavior video, and an alarm message is generated.

[0109] In this embodiment, when detecting each single frame image, there may be missed detections or false detections. To improve this situation, considering that the pedestrian's crossing process is a continuous process, the recognition results of the single frame images obtained by frame extraction should meet the temporal characteristics. Therefore, a triple exponential smoothing method is used to optimize the situation shown in Figure 3. Specifically, a set of continuous detection images is taken as the detection image set; it is determined whether there are single frame images in the detection image set whose detection results are different from those of other images; if there are single frame images in the detection image set whose detection results are different from those of other images, the detection image set is temporally optimized using the triple exponential smoothing method, and it is determined whether there are multiple consecutive single frame images in the optimized detection image set showing the crossing behavior; if there are multiple consecutive single frame images in the optimized detection image set showing the crossing behavior, the multiple single frame images in the detection image set showing the crossing behavior are stitched together to obtain the crossing behavior video, and alarm information is generated.

[0110] As shown in Figure 3, from frame (n+1) to frame (n+4), except for frame (n+3) which is normal, all the others are jump-over behaviors. Therefore, the timing of frames (n+1) to (n+4) is optimized using the triple exponential smoothing method. Similarly, from frame (n+10) to frame (n+14), except for frame (n+3) which is normal, all the others are jump-over behaviors. Therefore, the timing of frames (n+10) to (n+14) is optimized using the triple exponential smoothing method.

[0111] Triple exponential smoothing is derived from single and double exponential smoothing. The single exponential smoothing model is as follows:

[0112]

[0113] In the formula, Y t Y represents the predicted value at time t. t-1 This represents the true value at time t-1. This represents the exponentially smoothed value at time t-1. Let a represent the exponentially smoothed value at time t-2, where a∈(0,1);

[0114] The quadratic exponential smoothing model is as follows:

[0115]

[0116] In the formula, F t+1 Indicates the predicted value, a t and b t Let L represent the model parameters, and L represent the number of forward predictions from time t. This represents the value after one exponential smoothing. Indicates the value of quadratic exponential smoothing;

[0117] The model at the triple exponential smoothing point is as follows:

[0118] F t+1 =a t +b t L+c t L 2 ,

[0119]

[0120] This method utilizes a YOLOv5 network to perform object detection on image samples, obtaining pedestrian and turnstile detection boxes. Then, it sequentially passes a graph attention network, the OpenPose algorithm, and a camera model to obtain pedestrian appearance features, turnstile appearance features, and spatial features between pedestrians and turnstiles, respectively. Finally, based on these features, a multilayer perceptron is used to detect pedestrian and turnstile crossing behaviors, yielding the detection results. This application, based on a YOLOv5 network, graph attention network, OpenPose algorithm, and camera model, fully extracts the features of pedestrians and turnstiles in the detected images. Compared to existing methods, it fully considers the spatial relationship between pedestrians and turnstiles, significantly improving the accuracy of pedestrian and turnstile interaction detection.

[0121] This invention also provides a feature-based vaulting behavior detection device, which is used to execute any of the aforementioned feature-based vaulting behavior detection methods. Specifically, please refer to FIG4, which is a schematic block diagram of the feature-based vaulting behavior detection device provided in this invention.

[0122] As shown in Figure 4, the vaulting behavior detection device 100 based on fusion features includes a frame extraction module 110, a target detection module 120, a similarity detection module 130, an appearance feature extraction module 140, a spatial feature extraction module 150, and a vaulting behavior detection module 160.

[0123] The frame extraction module 110 is used to acquire video stream data from the monitoring camera at the gate entrance and exit, perform frame extraction processing on the video stream data to obtain an image sample set, and the detection image set includes multiple image samples.

[0124] The target detection module 120 is used to input the image samples into a preset Yolov5 network in chronological order to perform target detection and obtain the corresponding detection image, which includes pedestrian detection boxes and gate detection boxes.

[0125] The similarity detection module 130 is used to calculate the similarity of pedestrian detection boxes in two adjacent detection images, determine whether the pedestrians in the pedestrian detection boxes in the two adjacent detection images are the same person based on the calculation results, and mark the pedestrian detection boxes of the same person accordingly.

[0126] The appearance feature extraction module 140 is used to learn multi-attention features of the pedestrian detection box and the gate detection box in the detection image through a pre-trained graph attention network, and to perform relational reasoning on the learning results to obtain the appearance features of the pedestrian and the appearance features of the gate.

[0127] The spatial feature extraction module 150 is used to perform key point detection on pedestrians in the pedestrian detection box using the OpenPose algorithm, obtain the key point coordinates of each part of the human body, calculate the distance between each part of the human body and the gate based on the camera model and the key point coordinates, and then convert the calculation results into vectors and merge them to obtain the spatial features of the pedestrian and the gate.

[0128] The climbing behavior detection module 160 is used to detect climbing behavior of pedestrians and turnstiles in the detection image based on a pre-trained multilayer perceptron and the appearance features of pedestrians, turnstiles, and spatial features, and to obtain detection results.

[0129] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the hop-over behavior detection method based on fused features as described above.

[0130] In another embodiment of the invention, a computer-readable storage medium is provided. This computer-readable storage medium may be a non-volatile computer-readable storage medium. The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the hop-over behavior detection method based on fused features as described above.

[0131] Those skilled in the art will readily understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0132] In the embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Units with the same function may be grouped into one unit. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, or it may be an electrical, mechanical, or other form of connection.

[0133] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention, depending on actual needs.

[0134] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0135] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks.

[0136] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for detecting vaulting behavior based on fused features, characterized in that, include: The video stream data from the monitoring camera at the gate entrance and exit is acquired, and the video stream data is processed by frame extraction to obtain an image sample set, which includes multiple image samples. The image samples are input into a preset Yolov5 network in chronological order for target detection to obtain corresponding detection images. The detection images include pedestrian detection boxes and turnstile detection boxes. The similarity of pedestrian detection boxes in two adjacent detection images is calculated. Based on the calculation results, it is determined whether the pedestrians in the pedestrian detection boxes of two adjacent detection images are the same person, and the pedestrian detection boxes of the same person are marked accordingly. The pedestrian detection box and the gate detection box in the detection image are learned by a pre-trained graph attention network, and the learning results are used to perform relational reasoning to obtain the appearance features of the pedestrian and the gate. The OpenPose algorithm is used to detect key points of pedestrians in the pedestrian detection box to obtain the coordinates of key points of various parts of the human body. The distance between each part of the human body and the gate is calculated based on the camera model and the coordinates of the key points. The calculation results are converted into vectors and then merged to obtain the spatial features of the pedestrian and the gate. Based on a pre-trained multilayer perceptron and the pedestrian appearance features, gate appearance features, and spatial features, the system detects the climbing behavior of pedestrians and gates in the detected image, and obtains the detection results. The step of using the OpenPose algorithm to detect key points of pedestrians in the pedestrian detection box and obtain the key point coordinates of various parts of the human body includes: using the OpenPose algorithm to infer the pedestrians in the pedestrian detection box and generating heatmaps of human body parts and affinity vectors of all parts of the pedestrians. The heat map and affinity vector of the human body parts are continuously iterated until the number of iterations reaches a preset condition, and then the coordinates of the key points are output.

2. The method for detecting vaulting behavior based on fused features according to claim 1, characterized in that, The similarity calculation of pedestrian detection boxes in two adjacent detection images includes: converting the two adjacent detection images into grayscale images and dividing them into multiple sub-regions of equal size; calculating the brightness, contrast, and structural contrast of each sub-region in the two detection images, and combining the calculated results into a comprehensive index to obtain a first comprehensive index and a second comprehensive index; and performing a weighted average of the first comprehensive index and the second comprehensive index to obtain the similarity SSIM value of the two adjacent detection images; wherein, the SSIM value is calculated according to the following formula: Where μx represents the average value of graph x. Let μ represent the variance of graph x, and μγ represent the mean of graph y. σ represents the variance of graph y. xy Let c represent the covariance of graphs x and y, and c1 and c2 both represent constants in the stability formula.

3. The method for detecting vaulting behavior based on fused features according to claim 1, characterized in that, The pre-trained graph attention network learns multi-attention features for pedestrian detection boxes and turnstile detection boxes in the detection image, respectively. The learning results are used to perform relational reasoning to obtain pedestrian appearance features and turnstile appearance features. This includes: extracting features from pedestrian and turnstile detection boxes in the detected images using a pre-set backbone extraction network (ResNet); aggregating the features of pedestrian and turnstile detection boxes in the detected image set using a graph attention network to obtain aggregated pedestrian and turnstile features; constructing an initial feature matrix based on the aggregated pedestrian and turnstile features, and obtaining an initial adjacency matrix based on the initial feature matrix; calculating the interaction probability between the feature vectors corresponding to pedestrians and turnstiles in the initial adjacency matrix using a multilayer perceptron and softmax function, and dynamically updating the initial adjacency matrix based on the interaction probability to obtain the pedestrian and turnstile appearance features; and introducing an attention mechanism to implicitly learn the interaction correlation between pedestrians and turnstiles, using a... o,h This indicates that a is used simultaneously. o,h Dynamically update matrix E; in order to calculate a o,h Using a multilayer perceptron R r (f o ,f h The multilayer perceptron calculates the features of pedestrians and turnstiles, and outputs the classification result of the multilayer perceptron, expressed as: S r =R r (f0,f h In the formula, f o Indicates the appearance characteristics of the turnstile, f h The pedestrian appearance features are represented, and the classification results are normalized using the softmax function to obtain the interaction probability, as shown in the following formula: In the formula, a o,h This represents the interaction correlation between pedestrians and turnstiles, where N represents the number of pedestrian-turnstile interactions in a single image; calculate a. o,h Then the dynamically updated adjacency matrix can be obtained.

4. The method for detecting vaulting behavior based on fused features according to claim 1, characterized in that, The pre-trained multilayer perceptron and the pedestrian appearance features, gate appearance features, and spatial features, The detection of pedestrians and turnstiles in the detected image is performed to obtain detection results, including: stitching together the appearance features of the pedestrians, appearance features of the turnstiles, and spatial features to obtain stitched features; weighting and fusing the stitched features through the multilayer perceptron and identifying them, outputting the recognition probability that the stitched features belong to the climbing behavior, and determining whether the recognition probability is greater than a preset probability threshold; if so, it is determined that the climbing behavior has occurred in the detected image.

5. The method for detecting vaulting behavior based on fused features according to claim 4, characterized in that, The step of weighted fusing and recognizing the spliced ​​features using the multilayer perceptron includes: calculating the recognition probability P according to the following formula: P(M|(b h ,b o ))=σ(MLP(a i f h ,a i f o ,a i f d ), where M represents the input b h b o Given the condition, the probability of event M occurring; σ() represents the sigmoid function, f h f0 represents the appearance characteristics of pedestrians, and f0 represents the appearance characteristics of turnstiles. d Indicating spatial characteristics, a i The weights are represented by b, MLP stands for Multilayer Perceptron, and b represents weights. h b0 represents the coordinates of the pedestrian, and b0 represents the coordinates of the turnstile.

6. The method for detecting vaulting behavior based on fused features according to claim 1, characterized in that, The method, based on a pre-trained multilayer perceptron and the pedestrian appearance features, gate appearance features, and spatial features, performs trespassing behavior detection on the detected images of pedestrians and gates. After obtaining the detection results, the method includes: taking a set of consecutive detected images as a detection image set; determining whether there are any single-frame images in the detection image set whose detection results differ from those of other images; if there are single-frame images in the detection image set whose detection results differ from those of other images, then performing temporal optimization on the detection image set using a triple exponential smoothing method, and determining whether there are any consecutive single-frame images in the optimized detection image set showing trespassing behavior; if there are any consecutive single-frame images in the optimized detection image set showing trespassing behavior, then stitching together the multiple single-frame images showing trespassing behavior in the detection image set to obtain a trespassing behavior video, and generating alarm information.

7. A device for detecting vaulting behavior based on fused features, characterized in that, include: The frame extraction module is used to acquire video stream data from the monitoring camera at the gate entrance and exit, and to perform frame extraction processing on the video stream data to obtain an image sample set, which includes multiple image samples. The target detection module is used to input the image samples into a preset Yolov5 network in chronological order to perform target detection and obtain the corresponding detection image, which includes pedestrian detection boxes and gate detection boxes. The similarity detection module is used to calculate the similarity of pedestrian detection boxes in two adjacent detection images, determine whether the pedestrians in the pedestrian detection boxes in the two adjacent detection images are the same person based on the calculation results, and mark the pedestrian detection boxes of the same person accordingly. The appearance feature extraction module is used to learn multi-attention features of the pedestrian detection box and the gate detection box in the detection image through a pre-trained graph attention network, and to perform relational reasoning on the learning results to obtain the appearance features of the pedestrian and the gate. space The feature extraction module is used to perform key point detection on pedestrians in the pedestrian detection box using the OpenPose algorithm, obtain the key point coordinates of each part of the human body, calculate the distance between each part of the human body and the gate based on the camera model and the key point coordinates, and then convert the calculation results into vectors and merge them to obtain the spatial features of the pedestrian and the gate. The hopping behavior detection module is used to detect hopping behavior of pedestrians and turnstiles in the detection image based on a pre-trained multilayer perceptron and the pedestrian appearance features, turnstile appearance features, and spatial features, and obtain the detection results. The spatial feature extraction module includes: an inference unit, used to infer the pedestrians in the pedestrian detection box using the OpenPose algorithm, and generate heatmaps of the pedestrian's body parts and affinity vectors of all parts respectively; and an iteration unit, used to continuously iterate the heatmaps of the body parts and affinity vectors until the number of iterations reaches a preset condition, and output the coordinates of the key points.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the vaulting behavior detection method based on fused features as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the hop-over behavior detection method based on fused features as described in any one of claims 1 to 6.

Images