Method, system and medium for identifying abnormal gate state based on convolutional neural network

Through the convolutional neural network, the gate video data is analyzed, combined with single frame and continuous frame detection, the gate status is judged, which solves the problem that the unattended gate abnormality cannot be discovered in time, real-time detection and abnormal reminder are realized, and safety is improved Management efficiency.

CN118762311BActive Publication Date: 2024-12-06CHINA AUTOMOBILE ZHILIAN TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411244315.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-06
Publication Date
2024-12-06
Estimated Expiration
2044-09-06

AI Technical Summary

Technical Problem

In the absence of a person, the gate abnormality cannot be discovered and dealt with in a timely manner, resulting in safety issues and personnel passage affected. The existing methods cannot effectively detect the gate abnormality and promptly notify management personnel.

Method used

Using a method based on a convolutional neural network, the gate video data is obtained through the image acquisition device, frame extraction, subcontracting and annotation, and the gate judgment model is trained. Combined with single-frame detection and continuous frame detection, the gate switch status is analyzed, and the gate is judged. Whether the machine is abnormal, and the cause of the abnormality is confirmed through detection of key points of human bones and multi-target tracking, reminding management personnel.

Benefits of technology

Real-time detection and abnormal judgment of the gate state is realized, the calculation complexity is reduced, the simplicity is high, and it is easy to deploy on various computers and on-board equipment. It promptly reminds managers to deal with abnormal situations, and improves safety prevention and management efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118762311B_ABST
    Figure CN118762311B_ABST
Patent Text Reader

Abstract

The present invention relates to the technical field of gates, and discloses a method, system and medium for identifying abnormal gate states based on a convolutional neural network, including: obtaining video data of a gate area collected by an image acquisition device, extracting frames, packaging and annotating the video data to obtain annotated image data; the annotated information includes single-frame switch attributes of the gate and continuous-frame attributes of the gate; using the annotated image data to train a gate judgment model; the gate judgment model is used to judge the single-frame switch state of the gate, and judge the continuous-frame switch state of the gate based on the single-frame switch state of the gate; using the gate judgment model to judge the continuous-frame switch state of the gate of the video data to be tested; according to the continuous-frame switch state of the gate, whether pedestrians swipe their cards to pass, and whether the gate components are abnormal, a gate abnormality judgment result is obtained. The advantage of the present invention is that when an abnormality occurs in the gate when no one is guarding, the staff can be reminded in time.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of gate machines, and in particular relates to a method, system and medium for identifying abnormal gate machine states based on a convolutional neural network. Background Art

[0002] In the field of intelligent connected vehicles, in order to solve the problem of identity legitimacy between on-board terminal devices, gates are widely used in parks, office buildings and other places as access control, which can enhance security precautions and facilitate management.

[0003] However, during the use of the gate, when the gate system is abnormal or there is a problem with the mechanical hardware, the gate may not be able to close or open normally, resulting in identity authentication failure, causing security issues or affecting the normal passage of personnel. In some application scenarios, the access control gate is in an unattended state, especially in large parks, so the staff cannot detect abnormalities in time. Although "a method, device, equipment and medium for detecting abnormal rotation of a gate" can detect situations where unauthorized personnel bend the gate wing to force through, it cannot handle situations where the gate itself has abnormalities. When there is no one to guard, there is currently no effective method to detect gate abnormalities and the cause of the abnormality and promptly notify management personnel to check.

[0004] In addition, judging whether the gate is abnormal also needs to be considered from the time dimension. Methods based on video sequence analysis often require the use of 3D convolution. The use of 3D convolution increases the complexity of the model, and requires sufficient computing resources and model capacity to support the training and reasoning process.

[0005] Therefore, based on the above situation, it is necessary to develop a method, system and medium for identifying abnormal gate status based on convolutional neural network to solve the above problems. Summary of the invention

[0006] In order to solve the above technical problems, the present invention provides a method, system and medium for identifying abnormal gate status based on a convolutional neural network, which can effectively solve the problem that when the gate is unattended, abnormalities occur and staff cannot be reminded in time, and it is easy to deploy on various computers for use.

[0007] In a first aspect, a first object of the present invention is to provide a method for identifying abnormal gate state based on a convolutional neural network, comprising the following steps:

[0008] Obtain video data of the gate area collected by the image acquisition device, extract frames, sub-packetize and annotate the video data to obtain annotated image data; the annotated information includes: gate single frame switch attributes, gate continuous frame attributes;

[0009] The gate judgment model is trained using the annotated image data; the gate judgment model is used to judge the single-frame switch state of the gate, and judge the continuous-frame switch state of the gate based on the single-frame switch state of the gate;

[0010] The gate judgment model is used to judge the gate continuous frame switch state of the video data to be tested; according to the gate continuous frame switch state, whether the pedestrian swipes the card to pass and whether the gate parts are abnormal, the gate abnormality judgment result is obtained.

[0011] In a second aspect, a second object of the present invention is to provide a system for identifying abnormal gate status using a convolutional neural network, comprising:

[0012] The data acquisition module is used to acquire the video data of the gate area acquired by the image acquisition device, extract frames, sub-packetize and annotate the video data to obtain annotated image data; the annotated information includes: gate single frame switch attributes, gate continuous frame attributes;

[0013] A training module, connected to the data acquisition module, is used to train a gate judgment model using the annotated image data; the gate judgment model is used to judge the single-frame switch state of the gate, and judge the continuous-frame switch state of the gate based on the single-frame switch state of the gate;

[0014] The output module is connected to the training module and is used to use the gate judgment model to judge the gate continuous frame switch state of the video data to be tested; according to the gate continuous frame switch state, whether the pedestrian swipes the card to pass and whether the gate parts are abnormal, the gate abnormality judgment result is obtained.

[0015] In a third aspect, the present application provides a computer-readable storage medium, which stores a computer program; the computer program includes a gate judgment model, a key point detection model and a card swiping behavior confirmation model; the computer program is executed by a processor to implement the method described in the first aspect.

[0016] The embodiments of the present invention have the following technical effects:

[0017] The present application provides a method for identifying gate status based on convolutional neural network, which is trained by annotated gate data and adopts a combination of single frame detection and continuous frame detection to realize the detection of gate status in video; wherein, during continuous frame detection, a 1D convolutional neural network is constructed to analyze the switch status of the gate in the time dimension in the video, a residual module is adopted, and a hole convolution is added to maintain the simplicity of the network, reduce the risk of overfitting, and avoid the use of 3D convolution to establish time and space connections, thereby reducing the amount of calculation and algorithm complexity;

[0018] In addition, a human skeleton key point detection model was trained to determine whether pedestrians swipe their cards. Then, a comprehensive analysis of the gate working status and pedestrian movement trajectory was conducted through multi-target tracking and other technologies to determine whether the gate is abnormal and the cause of the abnormality, and to promptly remind management personnel to conduct relevant inspections.

[0019] The present application provides a method for detecting gate abnormalities by comprehensively considering factors such as pedestrian status, card swiping conditions, and whether the gate is faulty. The method is stored in a storage medium and can be easily deployed to various computers and vehicle-mounted embedded devices for use, with flexible deployment. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 It is a flow chart of a method for identifying abnormal gate status based on a convolutional neural network provided by an embodiment of the present invention;

[0022] Figure 2 is a flow chart of obtaining annotated image data provided by an embodiment of the present invention;

[0023] Figure 3 is a structural schematic diagram of a single-frame detection model provided by an embodiment of the present invention;

[0024] Figure 4 A flowchart of a correction module provided by an embodiment of the present invention;

[0025] Figure 5 A flow chart of a card swiping behavior confirmation model provided by an embodiment of the present invention;

[0026] Figure 6 An application flow chart for identifying abnormal gate status provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0028] First, the following nouns are explained:

[0029] (1) trackID in this application is a unique code used to identify and track specific content.

[0030] (2) A single frame refers to a static image in a video sequence, while continuous frames refer to a series of static images arranged in chronological order.

[0031] (3) JSON (JavaScript Object Notation) file is a lightweight data exchange format based on a subset of ECMAScript (js specification developed by the European Computer Association) and uses a completely language-independent text format to store and represent data.

[0032] (4) One-stage method: directly extract features through convolutional neural networks and predict the classification and location of the target in one step, which is relatively fast. This method does not independently and explicitly extract candidate regions (RegionProposal), but directly obtains the category and location information of the existing objects from the input image.

[0033] (5) Two-stage method: First, region generation is performed, that is, candidate regions (RegionProposal) are generated, and then these candidate regions are classified and located through a convolutional neural network. This step-by-step processing method is relatively slow, but it can process each candidate region more finely.

[0034] (6) Bounding Box (BBox) regression, also known as bounding box regression, is a technique used in computer vision to predict the position and size of the bounding box of a target object. In the object detection task, in addition to identifying the object category in the image, it is also necessary to determine the exact position of the object, which is the task of bounding box regression; starting from the initial bounding box, the object is accurately located by learning the predicted correction value.

[0035] (7) The core idea of ​​the SORT algorithm is to track targets by combining target detection and motion prediction. It adopts the Tracking-by-Detection strategy, that is, the input data of the algorithm comes from the target detector, such as Faster R-CNN, and the algorithm itself does not participate in the target detection process.

[0036] (8) A 2D box is a rectangular box defined in two-dimensional space. This rectangular box is defined by the coordinates of its upper left corner and lower right corner (or sometimes by its center point and size). It is used to represent or select the position and size of an image, interface element, game object, or other visual element. Head refers to the detection head, which is a key component in the model structure.

[0037] (9) 1D in this application refers to one-dimensional convolution.

[0038] (10) Padding is used to define the space between an element's border and its content.

[0039] (11) Neck is a feature fusion component located between the backbone network and the head network. It is used to fuse features from different levels to make full use of multi-scale information.

[0040] (12) Head is the head network, which is the last part of the network and is responsible for generating the final output based on the features extracted and fused by the previous layers. In the object detection task, Head usually includes layers for 2D box regression and classification.

[0041] (13) ResNet-34 is a type of deep residual network that aims to solve the gradient vanishing or gradient exploding problems and network degradation problems that occur during the training of deep neural networks. ResNet-34 achieves this goal by introducing residual connections, which enables the network to learn identity mappings, making it easier to optimize deep networks.

[0042] (14) ResNet1d is a one-dimensional version of the ResNet architecture, designed specifically for processing one-dimensional time series data.

[0043] (15) The ReLU (Rectified Linear Unit) layer is a commonly used activation function layer in deep learning. It introduces nonlinear factors by nonlinearly mapping the input data, helping the network learn complex patterns.

[0044] (16) COCO (Common Objects in Context) is a large dataset widely used in the field of computer vision. It contains a large number of images that are detailedly annotated with information such as various objects, scenes, and human skeleton key points.

[0045] (17) Anchor-Free object detection methods do not rely on predefined anchor boxes, but instead achieve object detection by directly predicting the key points of the target (such as center points, corner points, etc.) or the parameters of the bounding box (such as center point coordinates, width and height, etc.) on the image or feature map.

[0046] (18) ROI image is a specific area selected from the original image. This area contains information or features that the user is particularly interested in. The ROI image can be a subset of the original image or an image that has been processed in some way (such as cropping, enlarging, enhancing, etc.).

[0047] Example 1

[0048] Figure 1 is a flow chart of a method for identifying abnormal gate status based on a convolutional neural network provided by an embodiment of the present invention. Figure 1 , specifically including:

[0049] S1. Obtain video data of the gate area collected by an image acquisition device, extract frames, packetize and annotate the video data to obtain annotated image data, wherein the annotated information includes: gate single-frame switch attributes and gate continuous frame attributes.

[0050] The image acquisition device here can be a surveillance camera set up in the surrounding area or a camera installed on an intelligent security patrol car, and the image of the gate area is obtained through the camera. Among them, the intelligent security patrol car has functions such as automatic driving and remote monitoring, and can replace people to realize real-time patrols around the clock and at all times. The camera is installed on the patrol car, and all the locations where the gates are installed are set as stations. The patrol car will stop every time it arrives at the gate station, monitor the gate status, and transmit the video data back to the background computer. The intelligent security patrol car here is a prior art, and this application will not elaborate on it in detail.

[0051] See also Figure 2 , labeling image information includes the following steps:

[0052] S11, sampling the video data to obtain adjacent frame images, and packetizing the adjacent frame images to obtain a plurality of continuous packets.

[0053] The gate video acquired by the camera is frame-sampled. For example, the original frame rate of the camera is 30fps, and frame sampling is performed every 4 frames. Then 224 adjacent frame images are combined into a continuous package, which takes about 30s.

[0054] S12, calculating a similarity risk value according to adjacent frame images in the continuous package; if the similarity risk value is greater than a first threshold, no marking is performed; if the similarity risk value is less than or equal to the first threshold, proceeding to the next step.

[0055] Since the gate is static, when no pedestrians pass through the gate, the gate status remains almost unchanged, so the data needs to be cleaned to avoid a large number of repeated annotations.

[0056] The structural similarity metric SSIM is an indicator for measuring the similarity between two images. It measures image similarity from three aspects: brightness, contrast, and structure. The SSIM value range is [0, 1]. The larger the value, the higher the similarity. Therefore, this application uses the SSIM metric to clean the data.

[0057] The SSIM calculation formula is as follows:

[0058] ,

[0059] Among them, x and y are the pixel values ​​of the two images respectively. is the mean value of x, is the mean value of y, is the variance of x, is the y variance, is the covariance of x and y; , is a constant used to maintain stability, L is the dynamic range of pixel values, .

[0060] Among them, the structural similarity measure of each frame image and the previous frame image is calculated based on the adjacent frame images in the continuous package; the structural similarity measure of each frame is added and averaged to obtain the similarity risk value of the continuous package. That is, the similarity between each frame image and the previous frame image is calculated. Add and average to get the similarity risk value of the continuous package. The value range of this value is [0,1]. The calculation formula is as follows:

[0061] ,

[0062] in, represents the structural similarity measure between the i-th frame image and its previous frame image. For example, When the similarity risk value is greater than the first threshold, such as 0.9, it is considered that there are a large number of repeated images in the continuous packet data and no marking is performed; otherwise, proceed to the next step.

[0063] S13, annotating adjacent frame images in the same continuous package to obtain annotated image data.

[0064] The annotation information includes the gate single frame switch attribute (open, closed), i.e. the gate single frame switch state, and the gate continuous frame attribute (normally open, normally closed, normal), i.e. the gate continuous frame state; it also includes the upper left corner coordinates of the gate rectangular box (x1, y1), the lower right corner coordinates of the gate rectangular box (x2, y2) and trackID (0, 1, 2, ..., n); among them, the gate continuous frame attribute needs to be annotated according to the gate switch state in the entire continuous package; if the gate is always in the open state in a continuous segment, the gate continuous frame attribute is annotated as "normally open"; if it is always in the closed state, it is annotated as "normally closed"; if there are both open and closed states, it is annotated as "normal". trackID is used to identify and track each gate. The 224 frames in a continuous package are tracked and annotated as a continuous segment. The trackID of the same target in different frames must be consistent. The upper left corner coordinates of the gate rectangular box and the lower right corner coordinates of the gate rectangular box represent the gate position information, which are used as true values ​​for training.

[0065] Export the annotation results of each frame, that is, save the annotation information in a JSON file, and save the data of each continuous package in the same folder for subsequent model training.

[0066] S2. Use the labeled image data to train the gate judgment model; the gate judgment model is used to judge the single-frame switch state of the gate, and judge the continuous-frame switch state of the gate based on the single-frame switch state of the gate.

[0067] The gate judgment model includes a single-frame detection model for identifying the video data to be tested and outputting the single-frame switch attributes of the gate and a continuous frame detection model for identifying the single-frame switch attributes of the gate output by the single-frame detection model and outputting the continuous frame attributes of the gate.

[0068] (1) Single-frame detection model

[0069] The single-frame detection model is used to judge the single-frame switch status of the gate based on the video data to be tested.

[0070] The labeled image data is used for training to obtain a single-frame detection model. The single-frame detection model belongs to the single-frame target detection task. The labeled information used in training is the single-frame switch attributes of the gate, the coordinates of the upper left corner of the gate rectangular frame, and the coordinates of the lower right corner of the gate rectangular frame.

[0071] Object detection models based on convolutional neural networks can be roughly divided into two categories: one-stage and two-stage. The main difference between the two is whether the target category detection and bounding box regression tasks are performed separately.

[0072] Taking into account the performance of the detection algorithm and the real-time deployment, this application builds a one-stage and anchor-free network model, including Backbone for feature extraction, Neck for feature fusion, and Head for 2D box regression.

[0073] The gates of the gate machine have various styles, usually made of transparent tempered glass or hollow curved stainless steel, and the feature difference between the open and closed states of the gate in the image is not obvious. Therefore, it is necessary to build a network with strong feature extraction capabilities to better detect the state of the gate machine.

[0074] Based on the above considerations, this application replaces the backbone network Backbone with a deeper ResNet-34 and adds jump connections based on the CenterNet-ResNet-18 model. Through jump connections, feature maps of different levels can be fused, which improves the model's ability to detect targets of different scales; among them, jump connections allow low-level features to be directly connected to high-level features, so that the network can better utilize low-level detail information to improve the expression ability of high-level features; it helps to improve the model's positioning accuracy and detection accuracy for targets. At the same time, an enhanced Neck design is adopted, that is, a layer of convolution is added to the Neck part, which further enhances the ability of feature extraction and feature fusion, meets real-time requirements, and improves the detection accuracy of the gate. In addition, in order to improve the robustness and generalization ability of the detection model, the labeled image data is enhanced, and the data enhancement methods include but are not limited to flipping, rotating, cropping, scaling, translating, adjusting HSV contrast, adding noise, etc.

[0075] Specifically, this application uses the residual network ResNet-34 to extract the features of the input image and removes the first downsampling in the first stage. The input of the network model is , after ResNet-34, the output of the last layer of the network is ; where B is the batch size, is the number of channels of the input image, is the number of channels of the output feature map. W and H are the width and height of the input image, which can be 512 and 512 respectively.

[0076] The Neck part is responsible for fusing the high-level abstract semantic features and low-level fine-grained spatial features used in the Head part. A set of convolutional layers is added before upsampling to further enhance the ability of feature extraction and feature fusion, and enrich the extracted high-level semantic information. The Head part is the detection head, which is used for classification and regression. Because the anchor-free method is adopted, the final detection head will have 3 branches, and the dimension head size is ; The heatmap header size is ; is the number of categories, es=2; offset header size is The model structure can be found in Figure 3 , the model training process is as follows:

[0077] S211, construct a single-frame detection model to be trained.

[0078] A network model combining one-stage and anchor-free is adopted. The network model includes ResNet-34 for feature extraction, Neck for feature fusion, and Head for 2D box regression.

[0079] S212: Divide the labeled image data into a training set and a test set, and perform preprocessing.

[0080] Specifically, all labeled image data are divided into training set and test set according to 8:2, and then the image size is adjusted to meet the size conditions of the model input, such as: (512, 512).

[0081] Preprocessing includes random data augmentation and normalization of images.

[0082] S213. Setting training hyperparameters according to the preprocessed training set and the single-frame detection model to be trained.

[0083] The hyperparameters here refer to the parameters that need to be manually set or adjusted before or during model training. The training hyperparameters can be set according to the training set, the single-frame detection model to be trained, the performance of the training equipment, etc. It has been empirically verified that the main training hyperparameters in this application can be set as follows: epochs=50, batch_size=16, img_size=512, lr=0.001, where epochs is the total number of rounds of model training, batch_size is the number of images contained in each batch, img_size is the size of the input image, and lr is the learning rate.

[0084] S214, substituting the preprocessed training set into the single-frame detection model to be trained for training and adjusting the model parameters of the single-frame detection model to be trained according to the loss function; substituting the preprocessed test set into the single-frame detection model to be trained to calculate the evaluation index.

[0085] S215, looping through S214 until the value of the loss function is less than the second threshold or the number of training times reaches the third threshold, selecting the model parameters when the evaluation index is optimal or the model parameters of the last round of training to obtain a single-frame detection model.

[0086] Load the training set into the single-frame detection model to be trained for training, and adjust the model parameters according to the loss function. After each round of training, load the test set into the model for inference and calculate various indicators until the number of training rounds reaches a third threshold, such as 50, or the loss function value is lower than the second threshold, then terminate the training. The second and third thresholds here can be set according to the actual situation, and this application does not limit it.

[0087] After all training is completed, the parameters of the last round of training and the parameters with the best evaluation indicators in the whole process will be saved

[0088] In summary, a single-frame detection model for gates is obtained. By inputting the video data to be tested, the sequence data of the switch state of each single-frame image of the gate can be obtained.

[0089] (2) Continuous frame detection model

[0090] The continuous frame detection model used to identify the sequence data output by the single frame detection model and output the continuous frame attributes of the gate includes the following training steps:

[0091] S221, constructing a continuous frame detection model to be trained.

[0092] A 1D convolutional neural network model is trained using the labeled image data. The model can analyze the switch status of the gate from the time dimension and divide the gate status into three categories: "normal", "normally open" and "normally closed".

[0093] Based on the setting of the aforementioned continuous package, the model input should be a one-dimensional sequence of length 224. The value of the sequence is the output sequence value of the single-frame detection model, that is, "open" is represented by 1 and "closed" is represented by 2. When the gate is not detected in a certain frame, the missing value is filled with 0.

[0094] Residual neural networks can effectively solve the problems of gradient vanishing and model degradation in deep network training and are suitable for sequence classification tasks. Therefore, a 1D convolutional residual neural network ResNet1d was built. In order to improve the model perception and capture long-term dependencies while maintaining the simplicity of the network and reducing the risk of overfitting, a dilated convolution is added to the residual module.

[0095] The model input layer accepts one-dimensional gate sequence data as input, represented as where batch_size is the batch size, channels is the number of channels of the input data, and sequence_length is the length of the gate sequence. In a specific embodiment, =32, channels=1, sequence_length=224.

[0096] The first convolutional layer uses a 1D convolution kernel to extract features from the input data. The convolution kernel of size 100, stride 2, padding 3, and output channels 64 are used to reduce the length of the feature map and extract low-level features. The Batch Normalization layer is connected to normalize the data distribution of the convolution layer output, which helps to accelerate the convergence of the network and improve the stability of the model. The ReLU layer is connected. The ReLU activation function is used to introduce nonlinear features, which helps the network learn more complex patterns and representations. Among them, the use of 1D convolutional network maintains the simplicity of the network, reduces the risk of overfitting, and reduces the amount of calculation and algorithm complexity.

[0097] The ReLU layer is connected to the residual module, which is the core component of the network. It consists of 4 residual blocks, which can ensure both accuracy and operating efficiency. Each residual block contains multiple residual units, which is set to 2 in this case. Atrous convolution is added to the last two residual blocks.

[0098] The first residual block contains two residual units, each of which consists of two 1D convolutional layers with the same output size and the convolution kernel size is , stride 1, padding 1, output channel number 64, followed by batch normalization layer and ReLU activation function. At the same time, the first residual unit of the residual block adopts the same size as the input Convolution kernels are used to match the dimensions of input and output. The residual units that follow use Convolution kernel. The residual module adds the input and output through the residual connection to learn the residual.

[0099] The second residual block contains two residual units. The convolution kernel size of the first convolution layer of the first residual unit is , stride is 2, padding is 1, the number of output channels is 128, and the convolution kernel size of other convolution layers is , the stride is 1, the padding is 1, and the number of output channels is 128.

[0100] Adding dilated convolutions to the last two residual blocks and using dilated convolutions in later residual modules can ensure that the network captures a wider range of input information at a deeper level; especially when processing long sequence data, it is crucial to capture global features. In data with long-term dependencies such as time series data, dilated convolutions can better capture long-distance dependencies.

[0101] The third residual block contains two residual units. The convolution kernel size of the first convolution layer of the first residual unit is , stride is 2, hole rate is 2, padding is 2, and the number of output channels is 256. The convolution kernel size of the second convolution layer is , stride is 1, dilation rate is 2, padding is 2, and the number of output channels is 256. The two convolution layers of the second residual unit are the same as the second convolution layer of the first residual unit of residual block 3.

[0102] The fourth residual block contains two residual units. The convolution kernel size of the first convolution layer of the first residual unit is , stride is 2, hole rate is 4, padding is 4, and the number of output channels is 512. The convolution kernel size of the second convolution layer is , stride 1, dilation rate 4, padding 4, and output channel number 512. The two convolution layers of the second residual unit are the same as the second convolution layer of the first residual unit of residual block 4.

[0103] It should be noted that the above parameter settings are all preferred values ​​and can be set according to actual conditions in actual applications.

[0104] After the last residual module, each channel of the feature map is averaged and pooled through the global average pooling layer to reduce the dimension of the feature map to a vector while retaining the important information of each channel. Finally, the output of the global average pooling layer is connected to one or more fully connected layers, and finally the softmax activation function is used to map it to the output category of the model to obtain the probability of belonging to a certain category.

[0105] Therefore, based on the above settings, the continuous frame detection model to be trained is obtained.

[0106] S222. Obtain a gate single-frame switch attribute sequence and a gate continuous frame attribute according to the labeled image data and use them as a training set for preprocessing.

[0107] Prepare training data, which is obtained from the previously labeled data. Each continuous segment is a unit, and a continuous segment has 224 frames. The switch status of each gate in each frame is obtained through trackID. "Open" is represented by 1 and "Closed" is represented by 2. When the gate is severely blocked in a certain frame and cannot be labeled, the missing value is filled with 0, and finally a sequence with a length of 224 frames is formed.

[0108] The label of the sequence is the marked gate continuous frame attribute "normal", "normally open" or "normally closed", represented by 0, 1, and 2 respectively. After processing all the continuous packets, n gate sequences with a length of 224 are finally obtained as the input of the training model, and the gate continuous frame attributes are used as the output of the model. In a specific embodiment, the n gate single frame switch state sequences can be expressed as ,..., , the training label is , with a length of n, and the training labels are the attributes of consecutive frames of n gates.

[0109] Random noise is added to the training data to improve the robustness and generalization of the model. In this application, the data in the sequence are randomly set to the opposite state with a probability of 0-5%, such as "on" is set to "off", and "off" is set to "on".

[0110] Then normalize the data and scale the sequence data to The range helps to accelerate the convergence of the model and improve the performance of the model.

[0111] S223, setting training hyperparameters, bringing the preprocessed training set into the judgment model to be trained for training, and obtaining a continuous frame detection model.

[0112] In a specific embodiment, epochs represents the number of times the entire data set will be traversed, batch_size represents the number of samples used when updating the model weights, sequence_length represents the number or length of sequences in a sequence, and lr represents the learning rate. The following settings can be made: epochs=30, batch_size=32, sequence_length=224, lr=0.001.

[0113] The model will save parameters after each round of training. After all training is completed, the parameters of the last round of training or the parameters with the best evaluation indicators in the whole process will be saved.

[0114] In summary, the training of the gate judgment model has been completed.

[0115] S3. Use the gate judgment model to judge the gate continuous frame switch state of the video data to be tested; according to the gate continuous frame switch state, whether the pedestrian swipes the card to pass and whether the gate parts are abnormal, obtain the gate abnormality judgment result.

[0116] (1) Use the gate judgment model to confirm the continuous frame attribute of the gate.

[0117] Based on the above settings, the image size of the video to be detected is first adjusted to the model input size (512, 512) and normalized; then the image data to be tested is sent to the gate single-frame detection model for forward reasoning; finally, the detection results are post-processed, and the non-maximum suppression (NMS) algorithm is used to further eliminate redundant frames, find the best object detection position and output it.

[0118] Generally, there are multiple gates at a location where a gate is installed. Therefore, it is necessary to obtain the trackID of each gate to form the sequence data of each gate, so as to obtain the status of each gate's continuous frame.

[0119] The SORT algorithm, which combines Kalman filtering and Hungarian matching algorithm, is used to track the status of each gate.

[0120] use Describe the object model, They are the x and y coordinate values ​​of the center point of the detection target frame BBox, is the size or area of ​​the BBox, is the ratio of the width to the height of the BBox. Since the gate and the camera are relatively stationary, the Kalman filter state transfer matrix is ​​set as follows:

[0121] ;

[0122] During the tracking process, the records are made based on the above settings, that is, the switch status of each gate tracked is recorded into a set, the gate status of "open" is recorded as 1, the gate status of "closed" is recorded as 2, and if the gate of the trackID is not detected in a certain frame, the missing value is filled with 0. The switch status sequence set of a gate is represented as , which stores the detection result of 224 frames of a gate, with a length of 224.

[0123] In order to avoid the adverse effects of missed or false detection of individual frames on the results, the output results of the single-frame detection model need to be corrected. The incorrectly identified categories can be corrected. Figure 4 , the steps are as follows:

[0124] Step 1: As a one-dimensional discrete point, find The first-order forward difference of :

[0125]

[0126] Here V(t) and V(t+1) represent V history The values ​​at time t and t+1 are obtained , we believe that the position detection result where the difference is not 0 has changed.

[0127] Step 2: Find The position and distance of the peak and trough of the wave can be used to find the position where the test result changes. Perform the sign function operation:

[0128] ,

[0129] Obtain .

[0130] beg The second-order forward difference of , the second-order forward difference solution formula is as follows:

[0131] =

[0132] Obtain Among them, the point with a result of 2 is the trough, and the point with a result of -2 is the peak.

[0133] Because we are looking for forward differences, the final trough index position is 5, the peak index position is 8, and the distance between adjacent peaks and troughs is .

[0134] Step 3: Correct the error frame according to the distance threshold. To represent, set it to 10. When , it is judged as a detection error, and the corresponding categories between adjacent peaks and troughs are corrected to the correct value according to the categories of the previous and next frames; Based on the above settings, the detection error of "0,0,0" in Vhistory is judged as "2,2,2" according to the previous and next frames. After correction, .

[0135] The corrected After that, the sequence is normalized and preprocessed before being sent to the gate continuous frame detection model for forward reasoning, so as to directly obtain the judgment result of the gate judgment model.

[0136] When the judgment result is "normally open", it jumps to the gate abnormality confirmation module to further confirm whether the gate is abnormal.

[0137] (2) Card swiping behavior confirmation module

[0138] Since the CenterNet network architecture can be used for both target detection and key point detection, and considering that key point detection is difficult and requires a backbone network with stronger feature extraction capabilities, the key point detection model used in this invention is CenterNet-DLA-34, and other models such as HRNet, SimplePose, etc. can also be used. The training data is the COCO human skeleton key point data set, including 17 joints, namely nose, left and right eyes, left and right ears, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, and left and right ankles.

[0139] The training method of this model is similar to that of the gate single-frame detection model, so I will not elaborate on it here. The network model output includes the pedestrian detection box bboxes and keypoints. The multi-target tracking algorithm SORT is still used to track the detected pedestrians, and the tracking results are stored in the historical state set:

[0140] ,

[0141] in, Indicates the ground point coordinates and width and height of the pedestrian detection box bboxes. Indicates the coordinates of the 17 key points detected. If the pedestrian is not detected for 30 consecutive frames, it will be deleted.

[0142] Before judging whether a pedestrian has swiped a card, the camera needs to be calibrated. The internal and external parameters of the camera and the position of the card swiping area on the image are obtained through calibration, so as to obtain the pedestrian's 3D trajectory information and whether the left and right wrist key points coincide with the card swiping area. In order to reduce interference, the ROI area is set to detect only the status of pedestrians within 5 meters before and after the gate.

[0143] Camera calibration is mainly to obtain the internal and external parameters of the camera, so as to convert the image pixel coordinates into world coordinates, so as to obtain the 3D trajectory information of the pedestrian. The internal parameter calibration adopts Zhang Zhengyou's chessboard calibration method, and the external parameter calibration method adopts the PNP algorithm. The ground is defined as Z=0, and the grounding point of the leftmost gate is set as the origin of the world coordinate system (0,0,0). The horizontal right is the positive direction of the x-axis, and the vertical forward direction is the positive direction of the y-axis. Collect 9 points on the ground and record the coordinates. These 9 points try to keep every three points parallel to the x-axis on a horizontal line, and there are also 3 points in the vertical direction, parallel to the y-axis. At the same time, record the image pixel coordinates and world coordinates of the corresponding points, and use the PNP algorithm to solve the camera's external parameters R and T. When the camera position is fixed, it only needs to be calibrated once. Through the internal and external parameters of the camera and the coordinates of the pedestrian's grounding point The 3D motion trajectory of the pedestrian can be obtained. The card swiping area calibration can be done by marking and recording the card swiping area on the image through a polygon.

[0144] Specifically, see Figure 5 , the card swiping behavior confirmation module includes the following steps:

[0145] Step 1: Calibrate the internal and external parameters of the image acquisition device and the position of the card swiping area, and obtain the ROI area image according to the card swiping area.

[0146] That is, camera calibration is performed. Zhang Zhengyou's chessboard calibration method and PNP algorithm are used to obtain the internal and external parameters of the camera, and the relationship between the image pixel coordinates and the world coordinates is established; the card swiping area is marked and recorded on the image through a polygon; if the image acquisition device, that is, the camera position is fixed, it only needs to be calibrated once. The intrinsic parameters of the camera are parameters that describe the internal characteristics of the camera, and no further explanation is given here. The ROI image area is set to detect the status of pedestrians within 5 meters before and after the gate.

[0147] Step 2: Use the key point detection model to detect the key points of pedestrians based on the ROI area image. The key point detection model is used to analyze and process the human body image to identify the key points of the human body.

[0148] The image data to be tested is obtained and cropped to obtain the ROI area image. The ROI area image is resized to the model input size (512, 512) and normalized. The image data to be tested is then sent to the key point detection model for forward reasoning. After post-processing, the detection results are obtained, including the bboxes and keypoints of the detected pedestrians.

[0149] Step 3: Perform multi-target tracking on the key points of pedestrians and record the tracking results.

[0150] Perform multi-target tracking on pedestrian key points and record the historical status of each target in the collection middle.

[0151] Step 4: Obtain the movement trajectory of the pedestrian based on the tracking result records, and confirm whether the card swiping behavior occurs based on the movement trajectory.

[0152] according to The information recorded in is used to convert the bboxes ground point to the world coordinate according to the internal and external parameters to obtain the 3D motion trajectory of the pedestrian.

[0153] Because the origin of the world coordinate system is at the gate, y>0 in front of the gate and y<0 behind the gate. When a pedestrian passes through the gate normally, the y value in the pedestrian's trajectory changes from negative to positive or from positive to negative. When the pedestrian does not pass through the gate and is only on one side of the gate, the y value is all positive or all negative.

[0154] When the motion trajectory is consistent with the normal behavior of passing through the gate, jump to Step 2 and continue to detect the key points of the human body in the next frame. or If the residence time within the range is greater than a set threshold, which in a specific embodiment is set to 5 seconds and the pedestrian does not pass through, it is determined that the pedestrian may not be able to pass through the gate normally.

[0155] Card swiping behavior can be confirmed in a variety of ways, including traditional key point analysis methods or deep learning methods, which can be used to model and train the key points of detection to achieve end-to-end judgment. For example, first obtain the coordinates of the left and right wrist key points and , and then determine whether it overlaps with the marked card swiping area. If any key point of the left or right wrist overlaps with it, it is determined that there is a card swiping behavior. If there is no card swiping behavior, jump to Step 2 and continue to detect the key points of the next frame. If it exists, at this time, the pedestrian swipes the card and does not pass the gate normally, then it is determined that the gate may be closed abnormally. When this happens, jump to the gate abnormality confirmation module to further confirm whether the gate is abnormal.

[0156] (3) Gate abnormality confirmation module

[0157] When the gate is "normally open", determine whether there are pedestrians at the same time. If there are pedestrians, continue to observe the gate status. If there are no pedestrians, continue to check whether the infrared receiver receives signals. If no signal is received, it is determined that there may be a problem with the infrared sensor. Further check whether the infrared sensor is blocked by foreign objects or there is a problem with the transmitter and receiver. If the infrared sensor can receive signals normally, it is further determined that there is a problem with the motor or controller.

[0158] Determine the switch status of the gate. If it is "normally closed", further investigate the cause of the abnormality. When a card swiping behavior is detected, the card swiping information of the gate system is read at the same time. If it is not read, it means that there may be an abnormality in the card reader of the gate and the card cannot be swiped normally; if the card swiping success information is read, but the gate does not open, there may be a problem with the motor or controller.

[0159] (4) Abnormal status prompt module

[0160] After the gate abnormality confirmation module determines that the gate is abnormal, the program issues a voice broadcast command to inform the staff of the gate abnormality and possible reasons, and reminds the staff to check the monitoring or go to the site to further investigate the cause of the abnormality.

[0161] Thus, the entire process of gate status judgment and prompting is completed from data acquisition to model construction to model training to abnormal status judgment and prompting.

[0162] In practice, after the single-frame detection model, continuous-frame detection model, key-point detection model, etc. are trained, they are deployed in the computing unit for use. In order to facilitate deployment and improve the inference speed of the model, the .pt file trained in Python is converted into an onnx file and then into an engine file. The model is deployed using the tensorrt acceleration technology route. The detection algorithm can be deployed on the server side or on the embedded device on the vehicle side, which is convenient and flexible to deploy.

[0163] It should be noted that in actual application, you only need to use the trained models, see Figure 6 The steps in , no longer need to be labeled and model training.

[0164] This application includes the training of a deep learning model and the deployment of the model. It also proposes a method for gate detection and correction. By detecting key points of the gate and pedestrians, the gate abnormality and possible causes of the abnormality can be inferred. The method is simple, efficient and easy to deploy, and can effectively solve the problem of the gate being unable to promptly alert staff when abnormalities occur when there is no one on duty.

[0165] Example 2

[0166] The present application provides a system for identifying abnormal gate status using a convolutional neural network, comprising:

[0167] The data acquisition module is used to collect video data of the gate area through the image acquisition device, extract frames, sub-packetize and annotate the video data to obtain annotated image data; the annotated information includes: gate single frame switch attributes, gate rectangular frame upper left corner coordinates, gate rectangular frame lower right corner coordinates, gate continuous frame attributes and trackID;

[0168] A training module, connected to the data acquisition module, is used to use the annotated image data to train a model for judging the single-frame switch state of the gate and the continuous-frame switch state of the gate, so as to obtain a gate judgment model;

[0169] The output module is connected to the training module and is used to use the gate judgment model to judge the gate continuous frame switch state of the video data to be tested; according to the gate continuous frame switch state, whether the pedestrian swipes the card to pass and whether the gate parts are abnormal, the gate abnormality judgment result is obtained.

[0170] For other contents, please refer to Example 1 and will not be described in detail here.

[0171] Example 3

[0172] The present application provides a computer-readable storage medium, which stores a computer program; the computer program includes a gate judgment model, a key point detection model, and a card swiping behavior confirmation model, etc.; the computer program is executed by a processor to implement the method in Example 1.

[0173] Based on the above technical solution, this application provides a solution to the problem that managers are unable to discover problems in time when the access control gate is abnormal, thereby improving the efficient management and security prevention of the park.

[0174] It should be noted that the terms used in the present invention are only for describing specific embodiments, rather than limiting the scope of the present application. As shown in the present specification, unless the context clearly indicates an exception, the words "one", "a", "a kind of" and / or "the" do not specifically refer to the singular, but may also include the plural. The terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method or device. In the absence of more restrictions, the elements defined by the sentence "include one..." do not exclude the presence of other identical elements in the process, method or device including the elements.

[0175] It should also be noted that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. Unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", etc. should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be a connection between the two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0176] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. A method for identifying abnormal gate status based on convolutional neural network, characterized in that: The following steps are involved: Acquire video data of the gate area collected by the image acquisition device, extract frames, divide packets and mark the video data to obtain marked image data; The marked information includes: single-frame switch attributes of the gate, and continuous-frame attributes of the gate; The gate judgment model is trained using the annotated image data; the gate judgment model is used to judge the single-frame switch state of the gate, and judge the continuous-frame switch state of the gate based on the single-frame switch state of the gate; The gate judgment model is used to judge the gate continuous frame switch state of the video data to be tested; according to the gate continuous frame switch state, whether the pedestrian swipes the card to pass and whether the gate parts are abnormal, the gate abnormality judgment result is obtained; The gate judgment model includes a single-frame detection model for identifying the video data to be tested and outputting the single-frame switch attributes of the gate and a continuous frame detection model for identifying the single-frame switch attributes of the gate output by the single-frame detection model and outputting the continuous frame attributes of the gate; The training steps of the single frame detection model include: Build a single-frame detection model to be trained; Dividing the labeled image data into a training set and a test set, and performing preprocessing; Set training hyperparameters based on the preprocessed training set and the single-frame detection model to be trained; Substituting the preprocessed training set into the single-frame detection model to be trained for training and adjusting the model parameters of the single-frame detection model to be trained according to the loss function; substituting the preprocessed test set into the single-frame detection model to be trained to calculate the evaluation index; The previous step is repeated until the value of the loss function is less than the second threshold or the number of training times reaches the third threshold, and the model parameters when the evaluation index is optimal or the model parameters of the last round of training are selected to obtain a single-frame detection model; The training steps of the continuous frame detection model include: Build a continuous frame detection model to be trained; According to the labeled image data, the gate single-frame switch attribute sequence and the gate continuous frame attributes are obtained and used as training sets for preprocessing; The training hyperparameters are set, and the preprocessed training set is brought into the continuous frame detection model to be trained to obtain the continuous frame detection model.

2. The method for identifying abnormal gate status based on convolutional neural network according to claim 1 is characterized in that: The step of extracting frames, packaging and labeling the video data to obtain labeled image data comprises the following steps: Sampling the video data frame by frame to obtain adjacent frame images, and packetizing the adjacent frame images to obtain a plurality of continuous packets; Calculating a similarity risk value according to adjacent frame images in the continuous package; if the similarity risk value is greater than a first threshold, no marking is performed; if the similarity risk value is less than or equal to the first threshold, proceeding to the next step; Adjacent frame images in the same continuous package are annotated to obtain the annotated image data; wherein, the structural similarity measure of each frame image and the previous frame image is calculated based on the adjacent frame images in the continuous package; the structural similarity measure of each frame is added and averaged to obtain the similarity risk value of the continuous package.

3. The method for identifying abnormal gate state based on convolutional neural network according to claim 1 is characterized in that: The single-frame detection model to be trained is constructed including a network model combining One-stage and Anchor-Free.

4. The method for identifying abnormal gate status based on convolutional neural network according to claim 1 is characterized in that: The continuous frame detection model to be trained includes: A residual neural network ResNet1d using 1D convolution; the first convolution layer of the residual neural network ResNet1d uses a 1D convolution kernel, the first convolution layer is connected to a Batch Normalization layer, the BatchNormalization layer is connected to a ReLU layer; the ReLU layer is connected to a residual module; the residual module is connected to a global average pooling layer; the global average pooling layer is connected to a fully connected layer; the fully connected layer is connected to a softmax activation function; Wherein, the residual module comprises: The four residual blocks are connected in sequence, each of which includes two residual units, and the last two residual blocks also include dilated convolutions.

5. The method for identifying abnormal gate status based on convolutional neural network according to claim 1 is characterized in that: A correction module is provided between the single frame detection model and the continuous frame detection model; The correction module is used to correct the single-frame switch attributes of the gate output by the single-frame detection model, and the specific steps include: The single-frame switch attributes of the gate output by the single-frame detection model are represented as one-dimensional discrete points; the first-order forward difference is calculated based on the one-dimensional discrete points; Calculating the positions of the peaks and troughs and the distances between the peaks and troughs according to the first-order forward difference; If the distance between the peak and the trough is less than the distance threshold, it is considered a false detection, and the one-dimensional discrete point is corrected according to the previous and next frames; The corrected one-dimensional discrete points are input into the continuous frame detection model of the gate judgment model for processing.

6. The method for identifying abnormal gate status based on convolutional neural network according to claim 1 is characterized in that: The output of the gate abnormality judgment result according to the gate continuous frame switch state, whether the pedestrian swipes the card to pass and whether the gate parts are abnormal includes: If the gate machine continuous frame switch state is "normally closed", the card swiping behavior confirmation model is used to determine whether the pedestrian swipes the card to pass; if the card is not passed, the gate machine abnormality confirmation module is used to determine whether the gate machine components are abnormal and output the gate machine abnormality judgment result; If the gate machine continuous frame switch state is "normally open", the gate machine abnormality confirmation module determines whether the gate machine components are abnormal and outputs the gate machine abnormality judgment result.

7. The method for identifying abnormal gate status based on convolutional neural network according to claim 6 is characterized in that: The card swiping behavior confirmation model includes the following contents: Calibrate the internal and external parameters of the image acquisition device and the position of the card swiping area, and obtain the ROI area image according to the card swiping area; According to the ROI area image, the key point detection model is used to detect the key point parts of the pedestrian; Perform multi-target tracking on the key points of pedestrians and record the tracking results; The movement trajectory of the pedestrian is obtained according to the tracking result, and whether the card swiping behavior occurs is confirmed according to the movement trajectory; wherein, The key point detection model is used to analyze and process human body images, thereby identifying key points of the human body.

8. A system for identifying abnormal gate state based on a convolutional neural network, used to implement the method for identifying abnormal gate state based on a convolutional neural network as described in any one of claims 1 to 7, characterized in that: Includes the following: A data acquisition module is used to acquire video data of the gate area acquired by the image acquisition device, extract frames, divide packets and mark the video data to obtain marked image data; The marked information includes: single-frame switch attributes of the gate, and continuous-frame attributes of the gate; A training module, connected to the data acquisition module, is used to train a gate judgment model using the annotated image data; the gate judgment model is used to judge the single-frame switch state of the gate, and judge the continuous-frame switch state of the gate based on the single-frame switch state of the gate; The output module is connected to the training module and is used to use the gate judgment model to judge the gate continuous frame switch state of the video data to be tested; according to the gate continuous frame switch state, whether the pedestrian swipes the card to pass and whether the gate parts are abnormal, the gate abnormality judgment result is obtained.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program; the computer program includes a gate judgment model, a key point detection model and a card swiping behavior confirmation model; the computer program is executed by a processor to implement the method as described in any one of claims 1 to 7.