Stall illegal operation identification method and system based on human key point detection

Abstract

The present invention provides a method and system for identifying illegal operation of stalls based on human key point detection, comprising: obtaining stall video information: obtaining illegal operation of stall identification results based on the acquired stall video information, and a preset stall identification model and a human key point detection model; wherein the stall identification model is obtained based on A-FAST-RCNN algorithm training, and is used for identifying the target edge of the stall; the human key point detection model is used to determine the lateral position of a person according to the identified edge, and to determine the longitudinal position of the person through the human key points and the stall target; and to identify the illegally operated stalls according to the lateral position and the longitudinal position; the present invention adopts the human key point detection model and the stall target detection model based on A-FAST-RCNN to identify illegal operation, and determines the illegally operated stalls by combining the stall detection and the human key point detection, so that the detection of the stalls is more accurate, and the problem of missed detection when the target is blocked can be effectively reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of smart city construction, and in particular relates to a method and system for identifying illegal operation of stalls based on key point detection of human body. Background Art

[0002] The concept of smart city originated from the field of media. It refers to the use of various information technologies or innovative concepts to connect and integrate the city's systems and services in order to improve the efficiency of resource utilization, optimize urban management and services, and improve the quality of life of citizens. Smart city is an advanced form of urban informatization based on the next generation of innovation in the knowledge society that makes full use of the new generation of information technology in all walks of life in the city, realizing the deep integration of informatization, industrialization and urbanization, which will help alleviate the "big city disease", improve the quality of urbanization, achieve refined and dynamic management, and enhance the effectiveness of urban management and improve the quality of life of citizens.

[0003] The inventors found that intelligent detection of illegal stall operations is a key link in smart city construction. There are the following problems when applying deep learning technology to identify stalls and personnel: the recognition accuracy of the one-stage target detection algorithm is low, and the mainstream two-stage algorithm Faster R-CNN has poor detection effect on small targets / occluded targets. Summary of the invention

[0004] In order to solve the above problems, the present invention proposes a method and system for identifying illegal stall operations based on human key point detection. The present invention is based on the A-Fast-RCNN algorithm and the human key point detection algorithm Keypoint R-CNN, and detects illegal stalls in the electronic fence area in the street, thereby improving the detection effect.

[0005] In order to achieve the above object, the present invention is implemented through the following technical solutions:

[0006] In a first aspect, the present invention provides a method for identifying illegal stall operations based on human key point detection, comprising:

[0007] Get booth video information:

[0008] Based on the acquired stall video information, as well as the preset stall recognition model and human key point detection model, the stall illegal operation recognition result is obtained;

[0009] Among them, the stall recognition model is trained based on the A-FAST-RCNN algorithm and is used to identify the target edges of the stall; the human key point detection model is used to determine the lateral position of a person based on the identified edges, and to determine the longitudinal position of a person through the human key points and the stall target; and to identify stalls that operate illegally based on the lateral position and longitudinal position.

[0010] Further, the training of the booth recognition model and the human key point detection model includes:

[0011] Get training booth video information;

[0012] Input the training booth video information into the convolutional neural network for feature extraction;

[0013] Through a RoI neural network using a RoI-pooling layer and multiple output fully connected layers;

[0014] The adversarial neural network uses the region features as input and generates a mask that points out the features of the discarded part;

[0015] Output a softmax classification and a soft-L1 regression.

[0016] Furthermore, the identification of illegal stall operations includes:

[0017] The lateral position of the person is determined by the left and right sides of the booth target beyond the shoulders or elbows of the person;

[0018] Determine the vertical position of a person on the booth target by using the key points of the person's eyes, nose or mouth;

[0019] If both the horizontal and vertical positions of the person meet the conditions, it is judged as a stall operating illegally.

[0020] Furthermore, the positions of the previous and next frames of the video are compared to determine the illegal business stalls that have been operating for a long time.

[0021] Furthermore, the positions of the previous and next frames of the video are compared to determine the illegal business stalls that have been operating for a long time, including:

[0022] Obtain the booth position target and change range of consecutive frames;

[0023] If the booth location distance change is lower than a preset threshold in multiple consecutive frames, the booth is judged as a long-term resident booth.

[0024] Furthermore, the parameters of the booth recognition model and the human key point detection model are encrypted using a homomorphic encryption algorithm.

[0025] Furthermore, the federated averaging algorithm is used to aggregate the global model.

[0026] In a second aspect, the present invention further provides a system for identifying illegal stall operations based on key point detection of a human body, comprising:

[0027] The data acquisition module is configured to: obtain booth video information:

[0028] The stall illegal operation identification module is configured to: obtain the stall illegal operation identification result according to the acquired stall video information, and the preset stall identification model and human key point detection model;

[0029] Among them, the stall recognition model is trained based on the A-FAST-RCNN algorithm and is used to identify the target edges of the stall; the human key point detection model is used to determine the lateral position of a person based on the identified edges, and to determine the longitudinal position of a person through the human key points and the stall target; and to identify stalls that operate illegally based on the lateral position and longitudinal position.

[0030] In a third aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for identifying illegal stall operations based on human body key point detection as described in the first aspect.

[0031] In a fourth aspect, the present invention further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method for identifying illegal stall operations based on human body key point detection as described in the first aspect are implemented.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] The present invention adopts a human key point detection model and a stall target detection model based on A-FAST-RCNN to identify illegal operations. The illegal operation stalls are judged by combining stall detection and human key point detection. The detection of stalls is more accurate and the problem of missed detection when the target is obscured can be effectively reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The drawings in the specification that constitute a part of this embodiment are used to provide a further understanding of this embodiment. The schematic embodiments of this embodiment and their descriptions are used to explain this embodiment and do not constitute improper limitations on this embodiment.

[0035] Figure 1 This is a flow chart of Example 1 of the present invention. DETAILED DESCRIPTION

[0036] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0037] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0038] Embodiment 1:

[0039] like Figure 1 As shown, this embodiment provides a method for identifying illegal stall operations based on human key point detection, and the specific steps are as follows:

[0040] S1, each street server uses images from surveillance videos to create a dataset for A-FAST-RCNN model training;

[0041] S2, the street server downloads the initialization model sent by the central server;

[0042] S3, the street server trains the local stall recognition model and the human key point detection model;

[0043] S4, each street server uses homomorphic encryption algorithm to encrypt the model parameters and upload them to the central server;

[0044] S5, the central server uses the federated average algorithm to aggregate the global model;

[0045] S6, the central server sends the aggregated global model to each street server;

[0046] S7, the street server uses the global model to detect the target;

[0047] S8. Use the stall recognition model and the human key point detection model (Keypoint R-CNN algorithm of the Detectron2 framework) to determine the illegal stalls;

[0048] S9. In this embodiment, the positions of the previous and next frames of the video are compared to determine the illegal business stalls that have been operating for a long time. In other embodiments, it is also possible to determine whether the stall target is operating for a long time based on other target detection algorithms and the similarity between adjacent frame images.

[0049] The A-Fast-RCNN algorithm introduces adversarial learning into the problem of target detection. It builds an adversarial network to generate samples that are difficult for the original network to detect (i.e., data with occlusion), thereby making the network more robust to occlusion problems. The adversarial network that generates occluded features is ASDN, which uses the convolutional features of each target proposal after the ROI pooling layer in Fast R-CNN as the input of the adversarial network. Given the features of a target, ASDN attempts to generate a mask in which some parts of the feature are dropped out, causing the detector to be unable to recognize the object. In the forward propagation process, ASDN is first used to generate a feature mask after the ROI pooling layer, and then the importance sampling method is used to generate a binary mask. The mask is used to clear the values ​​of the corresponding parts of the feature, and the modified features continue to forward propagate to calculate the loss. This process generates difficult features for training the detector;

[0050] This embodiment is based on the A-Fast-RCNN algorithm and the human key point detection algorithm Keypoint R-CNN to detect illegal business stalls within the electronic fence area on the street, and uses a federated learning framework for multi-server joint training to improve the model training effect and protect residents' data privacy.

[0051] In step S3 of this embodiment, the stall recognition model training is performed on each local data set based on A-FAST-RCNN, which mainly includes:

[0052] S3.1, input the image into the convolutional neural network for feature extraction;

[0053] S3.2, through a RoI neural network using a RoI-pooling layer and some output fully connected layers;

[0054] S3.3, the adversarial neural network uses the regional features as input and generates a mask to indicate which part of the features are discarded, making it more difficult for the detector to detect the object;

[0055] S3.4. Output a softmax classification and a soft-L1 regression.

[0056] In step S8 of this embodiment, the stall recognition model and the human body key point detection model are used simultaneously to judge the illegal operation stalls, which mainly includes:

[0057] S8.1. Determine the lateral position of a person by the left and right sides of the booth target beyond the shoulders or elbows of the person;

[0058] S8.2. Determine the vertical position of a person on the booth target by using the key points of the person's eyes, nose, or mouth;

[0059] S8.3. If the horizontal and vertical positions of the personnel meet the requirements, the stall is judged to be operating in violation of regulations.

[0060] In step S9 of this embodiment, comparing the positions of the previous and next frames of the video to determine the illegal business stalls that have been resident for a long time mainly includes:

[0061] S9.1, obtaining the booth position target and change range of consecutive frames;

[0062] S9.2. If the distance change of the booth position is lower than the threshold value in multiple consecutive frames, it is judged as a long-term resident booth.

[0063] Since the one-stage target detection algorithm has low recognition accuracy, and the mainstream two-stage algorithm Faster R-CNN has poor detection effect on small targets / occluded targets, in this embodiment, based on the A-FAST-RCNN algorithm, ASDN is used to generate occluded samples to improve the robustness of target detection; since the types of stalls are varied, in this embodiment, a stall detection model and a human key point model are used to make a more accurate comprehensive judgment on stall operations; uploading the video / image data of each camera to the cloud for centralized deep learning has problems such as large data volume and insecure data transmission. Therefore, in this embodiment, a federated learning framework is applied to perform multi-node joint training without uploading data, thereby ensuring data privacy.

[0064] Embodiment 2:

[0065] This embodiment provides a system for identifying illegal stall operations based on human key point detection, including:

[0066] The data acquisition module is configured to: obtain booth video information:

[0067] The stall illegal operation identification module is configured to: obtain the stall illegal operation identification result according to the acquired stall video information, and the preset stall identification model and human key point detection model;

[0068] Among them, the stall recognition model is trained based on the A-FAST-RCNN algorithm and is used to identify the target edges of the stall; the human key point detection model is used to determine the lateral position of a person based on the identified edges, and to determine the longitudinal position of a person through the human key points and the stall target; and to identify stalls that operate illegally based on the lateral position and longitudinal position.

[0069] The working method of the system is the same as the method for identifying illegal stall operations based on human key point detection in Example 1, and will not be repeated here.

[0070] Embodiment 3:

[0071] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the method for identifying illegal stall operations based on human body key point detection described in Example 1 are implemented.

[0072] Embodiment 4:

[0073] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the method for identifying illegal stall operations based on human body key point detection described in Example 1 are implemented.

[0074] The above description is only a preferred embodiment of the present embodiment and is not intended to limit the present embodiment. For those skilled in the art, the present embodiment may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present embodiment shall be included in the protection scope of the present embodiment.

Claims

1. A method for identifying illegal stall operations based on human body key point detection, characterized in that: include: Get booth video information: Based on the acquired stall video information, as well as the preset stall recognition model and human key point detection model, the illegal operation identification results of the stall are obtained; The stall recognition model is trained based on the A-FAST-RCNN algorithm and is used to identify the target edges of stalls. The human key point detection model is used to determine the horizontal position of a person based on the identified edges, and the vertical position of a person based on the human key points and stall targets. Illegal stalls are identified based on the horizontal and vertical positions. Identification of illegal stall operations, including: Determine the lateral position of a person by ensuring that the left and right sides of the booth target are outside the person's shoulders or elbows; Determine the vertical position of a person on the booth target by using the key points of the human eyes, nose or mouth; If both the horizontal and vertical positions of the person meet the conditions, it is judged as a stall operating in violation of regulations; Compare the positions of the previous and next frames of the video to identify illegal stalls that have been operating for a long time, including: Obtain the booth position target and change range of consecutive frames; If the distance change of the booth position is lower than the preset threshold in multiple consecutive frames, the booth is judged as a long-term resident booth.

2. The method for identifying illegal stall operations based on human body key point detection according to claim 1, characterized in that: The training of the stall recognition model and the human key point detection model includes: Get video information of training booths; Input the training booth video information into the convolutional neural network for feature extraction; Through a RoI neural network using a RoI-pooling layer and multiple output fully connected layers; The adversarial neural network uses the region features as input and generates a mask that indicates the features of the discarded part; Output a softmax classification and a soft-L1 regression.

3. The method for identifying illegal stall operations based on human body key point detection according to claim 1, characterized in that: The parameters of the stall recognition model and the human key point detection model are encrypted using the homomorphic encryption algorithm.

4. The method for identifying illegal stall operations based on human body key point detection according to claim 1, characterized in that: The federated averaging algorithm is used to aggregate the global model.

5. The system for identifying illegal stall operations based on human key point detection is characterized by: include: The data acquisition module is configured to: obtain booth video information: The illegal stall operation identification module is configured to obtain illegal stall operation identification results based on the acquired stall video information, as well as a preset stall recognition model and a human key point detection model; The stall recognition model is trained based on the A-FAST-RCNN algorithm and is used to identify the target edges of stalls. The human key point detection model is used to determine the horizontal position of a person based on the identified edges, and the vertical position of a person based on the human key points and stall targets. Illegal stalls are identified based on the horizontal and vertical positions. Identification of illegal stall operations, including: Determine the lateral position of a person by ensuring that the left and right sides of the booth target are outside the person's shoulders or elbows; Determine the vertical position of a person on the booth target by using the key points of the human eyes, nose or mouth; If both the horizontal and vertical positions of the person meet the conditions, it is judged that the stall is operating in violation of regulations; Compare the positions of the previous and next frames of the video to identify illegal stalls that have been operating for a long time, including: Obtain the booth position target and change range of consecutive frames; If the distance change of the booth position is lower than the preset threshold in multiple consecutive frames, the booth is judged as a long-term resident booth.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method for identifying illegal stall operations based on human key point detection as described in any one of claims 1 to 4 are implemented.

Images