Super-store object detection method based on deep learning

Abstract

The invention discloses an overstore object detection method based on deep learning, and the method comprises the steps: obtaining the image information of a store and the overstore standard of a current store through a camera, and carrying out the overstore object detection of an image through a deep learning algorithm, namely, extracting a feature map from an input image through employing ResNet101 as a basic grid, and carrying out the detection of the overstore object through employing ResNet101 as a basic grid; the method comprises the following steps of: obtaining feature maps of multiple scales and a position map of a door line in a predicted picture, splicing the feature maps of the multiple scales and the position map of the door line in the predicted picture on a channel, inputting a region generation network (RPN), and judging and identifying a store exceeding operation behavior of a storefront after obtaining the position of an object and a corresponding object category; and finally, object identification is carried out according to the judgment result of the exceeding store operation behavior of the store, alarming is carried out according to the object identification result, exceeding store object detection can be carried out only by using the position information of the door line coordinates, dependence on the information of the preset position is avoided, and the method can freely adapt to the required scene.

Description

Technical Field

[0001] The present invention belongs to the technical field of super-store object detection, and specifically relates to a super-store object detection method based on deep learning. Background Art

[0002] Super-store operation is the behavior of operators occupying public places outside the permitted business premises to conduct business. The problem of super-store operation of stores is one of the important issues of urban management. Usually, it is managed by human patrol and inspection, which requires a certain amount of manpower. The disadvantage is that it cannot be monitored continuously and the effect is not good. Nowadays, surveillance cameras have reached a considerable degree of popularity. Using surveillance images and computer technology to automatically identify stores that operate super-stores can save manpower, have a wide range of application scenarios, and have considerable practical value.

[0003] In addition to the hardware method, the existing technology uses dynamic object detection methods to perform background modeling to detect the phenomenon of super-store operation. When the video viewing angle is fixed, that is, the store location captured in each frame remains unchanged, the dynamic object detection method is used to determine the objects outside the store, and then the objects outside the store are classified using the out-of-store object classification model; when the video viewing angle is not fixed, the object detection model is used to determine the object classification and object location in the frame image, and then the out-of-store object is determined based on the object location. It is obvious that the super-store operation method detected by this method is very complicated and the workload is large.

[0004] There is also a method and device for analyzing super-store operations based on image deep learning algorithms. Video data is obtained by setting a camera with more than or equal to one preset position, and the video data is detected by the image deep learning algorithm to identify the super-store operation and issue a violation alarm. The violation alarm situation obtained by detecting at least one preset position is combined and counted, thereby obtaining super-store operation information. However, the image recognition method based on deep learning depends on the setting of the preset position, that is, the camera should be set in the preset position area, or at least know the angle position and other information of the known preset position, and detect and classify the objects in the captured image, and judge the super-store operation through these real physical information. It has great limitations, poor generalization ability in changing scenes, and requires retraining or maintaining the preset position setting to maintain the recognition quality. The operability in real scenes is limited. Therefore, we need to propose a super-store object detection method based on deep learning to solve the above problems, so that it can not rely on the information of the preset position and can freely adapt to the required scene. Summary of the invention

[0005] The purpose of the present invention is to provide a super-store object detection method based on deep learning, so that it can be independent of preset position information and can freely adapt to the required scene to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A method for detecting objects in a super-store based on deep learning comprises the following steps:

[0008] S1, using a camera to obtain the image information of the store and the super-store standard of the current store, that is, the location information of the door line;

[0009] S2. Perform super-store object detection on the image through deep learning algorithm, specifically:

[0010] S2.1. First, use ResNet101 as the basic grid to extract feature maps from the input image, obtain feature maps of multiple scales and predict the position map of the goal line in the image;

[0011] S2.2, then concatenate the feature maps of multiple scales and the position map of the door line in the predicted image on the channel and input them into the regional generation network RPN;

[0012] S2.3, ResNet101 and feature maps of multiple scales and network RPN are used to generate a large number of candidate boxes, and a fixed-size candidate box feature map is obtained through the RoI pooling layer;

[0013] S2.4, perform a full connection operation on the fixed-size candidate frame feature map through a fully connected layer to obtain the location of the object and the category of the object in the image, and after obtaining the location of the object and the corresponding object category, judge and identify the super-store operation behavior of the store;

[0014] S3. Perform object recognition based on the store's extra-store business behavior judgment result, and implement an alarm based on the result of the object recognition.

[0015] Preferably, in step S1, the camera is arranged above the current store, and the door line information in the current scene is obtained by using the camera monitoring video information as a data source, and the approximate position of the door line is marked with a straight line.

[0016] Preferably, the position of the door line is marked by obtaining the horizontal and vertical coordinates of the two ends of the line segment. In the deep learning algorithm training and the process of recognition using the trained algorithm, it is necessary to draw the door line on the image according to the coordinates, thereby assisting the recognition of super stores.

[0017] Preferably, in step S2, when performing super-store object detection on an image, a target detection model is first established based on the FasterR-CNN network, and ResNet101 is used as the basic network to extract feature maps from the input image, wherein each convolutional layer is followed by a ReLU layer and a BatchNormalization layer.

[0018] Preferably, in step S2.1, ResNet101 is divided into two branches:

[0019] The first branch: input the feature map into a fully convolutional network FCN to predict the position of the door line in the image. FCN includes 4 3x3 convolutional layers;

[0020] The second branch: input the obtained feature map into the feature pyramid network FPN to obtain feature maps of multiple scales to improve the detection effect of objects of different sizes.

[0021] Preferably, in step S2.2, when generating the network RPN, first generate pre-selected box anchors of different sizes and ratios for each pixel on the feature map, and the specific number of each pre-selected box is controlled by two parameters, scales and aspectratios. Scale refers to the pixel size of the longest side of each shape frame, and aspectratios refers to the aspect ratio of the pre-selected box. Scales = [8, 16, 32] and aspectratios = [0.5, 1, 2] are generated at each position, with a total of 3x3 = 9 pre-selected boxes.

[0022] Preferably, the network RPN includes three convolutional layers: first, the feature map passes through a 3x3 convolutional layer to make the extracted features more robust, and then two 1x1 convolutional layers are performed respectively, one of which is responsible for predicting the foreground and background judgment, and the number of channels of the output feature map is 2x9=18, indicating the scores of the 9 pre-selected boxes at each position belonging to the foreground and background respectively; the other convolutional layer is responsible for predicting the offset value of the pre-selected box and the real border (x, y, w, h), where (x, y) represents the coordinates of the center point of the border, and (w, h) represents the width and height. The number of channels of the convolutional layer output feature map is 4x9=36, representing the offset value of each pre-selected box (x, w, w, h), and the pre-selected box will adjust the size and position according to the predicted offset value to form an accurate candidate box.

[0023] Preferably, in step S2.3, the fixed-size candidate box feature maps can be uniformly input into the subsequent fully connected operation for target recognition and positioning.

[0024] Preferably, in step 2.4, during the full connection operation, the L1 loss is used to perform bounding box regression on the candidate box again to obtain the location of the object, and Softmax is used to classify the candidate box to obtain the object category in the image. After obtaining the object location and the corresponding object category, the store's super-store operating behavior can be judged and warned.

[0025] Preferably, in step S3, during object recognition, a door line is first drawn on the image according to the coordinates. When an alarm is implemented, an alarm standard is designed according to the actual application scenario. When the identified object is a super-store, first determine whether the confidence of the object output by the network exceeds the threshold value of 0.5. If it is less than 0.5, filter the object, and then determine the size of the object. If the area of ​​the object is less than 0.05 of the image, it is far away from the camera and is ignored.

[0026] The deep learning-based super-store object detection method proposed in this invention has the following advantages compared with the prior art:

[0027] 1. The present invention first obtains the image information of the store and the super-store standard of the current store through a camera, and then performs super-store object detection on the image through a deep learning algorithm. Finally, the object is identified according to the judgment result of the super-store business behavior of the store, and an alarm is implemented according to the object identification result. Super-store object detection can be performed only using the position information of the door line coordinates, which does not rely on the information of the preset position, is easy to still have a good effect after the scene changes, and can freely adapt to the required scene.

[0028] 2. The present invention uses a single camera to obtain the image information of the store and the super-store standard of the current store, which reduces the demand for special hardware or the number of machines and saves manpower and material resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a flowchart of the present invention;

[0030] Figure 2 Schematic diagram of the target detection model of the present invention;

[0031] Figure 3 It is a schematic diagram of the network RPN structure of the present invention. DETAILED DESCRIPTION

[0032] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. The specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0033] The present invention provides Figure 1 A method for detecting objects in a super store based on deep learning is shown, comprising the following steps:

[0034] S1, using a camera to obtain the image information of the store and the super-store standard of the current store, that is, the location information of the door line;

[0035] The camera is arranged above the current store, and uses the camera monitoring video information as a data source to obtain the door line information in the current scene, and uses a straight line to mark the approximate position of the door line. Due to the camera angle, if the door line information is perfectly marked in the image, curves will generally be involved, which increases the operating cost. By using a straight line to mark the approximate position of the door line, the convenience of door line marking is improved.

[0036] The position of the door line is marked by obtaining the horizontal and vertical coordinates of the two ends of the line segment. In the deep learning algorithm training and the process of recognition using the trained algorithm, it is necessary to draw the door line on the image according to the coordinates, thereby assisting the recognition of super stores.

[0037] S2. Perform super-store object detection on the image through deep learning algorithm. When performing super-store object detection on the image, first establish a target detection model based on the FasterR-CNN network, such as Figure 2 As shown in the figure, ResNet101 is used as the basic network to extract feature maps from the input image, as shown in the following table:

[0038]

[0039]

[0040] Among them, each convolutional layer is followed by a ReLU layer and a BatchNormalization layer; specifically:

[0041] S2.1. First, use ResNet101 as the basic grid to extract feature maps from the input image, obtain feature maps of multiple scales and predict the position map of the goal line in the image;

[0042] ResNet101 is then divided into two branches:

[0043] The first branch: input the feature map into a fully connected network (FCN) to predict the position of the door line in the image. The FCN includes four 3x3 convolutional layers.

[0044] The second branch: input the obtained feature map into the Feature Pyramid Network (FPN) to obtain feature maps of multiple scales to improve the detection effect of objects of different sizes.

[0045] S2.2, then concatenate the feature maps of multiple scales and the position map of the door line in the predicted image on the channel and input them into the region proposal network (RPN);

[0046] When generating the network RPN, we first generate pre-selected box anchors of different sizes and ratios for each pixel on the feature map. The specific number of each pre-selected box is controlled by two parameters: scales and aspectratios. Scale refers to the pixel size of the longest side of each shape frame, and aspectratios refers to the aspect ratio of the pre-selected box. At each position, we generate 3x3=9 pre-selected boxes with scales=[8,16,32] and aspectratios=[0.5,1,2].

[0047] like Figure 3 As shown in the figure, the network RPN includes three convolutional layers: first, the feature map passes through a 3x3 convolutional layer to make the extracted features more robust, and then two 1x1 convolutional layers are performed respectively, one of which is responsible for predicting the foreground and background judgment, and the number of channels of the output feature map is 2x9=18, indicating the scores of the 9 pre-selected boxes at each position belonging to the foreground and background respectively; the other convolutional layer is responsible for predicting the offset value of the pre-selected box and the real box (x, y, w, h), where (x, y) represents the coordinates of the center point of the box, and (w, h) represents the width and height. The number of channels of the convolutional layer output feature map is 4x9=36, representing the offset value of each pre-selected box (x, w, w, h). The pre-selected box will adjust its size and position according to the predicted offset value to form an accurate candidate box.

[0048] S2.3, ResNet101 and feature maps of multiple scales and network RPN are used to generate a large number of candidate boxes, and a fixed-size candidate box feature map is obtained through the RoI pooling layer;

[0049] The fixed-size candidate frame feature maps can be uniformly input into the following fully connected operation for target recognition and positioning.

[0050] S2.4, perform a full connection operation on the fixed-size candidate frame feature map through a fully connected layer to obtain the location of the object and the category of the object in the image, and after obtaining the location of the object and the corresponding object category, judge and identify the super-store operation behavior of the store;

[0051] During the full connection operation, the L1 loss is used to perform bounding box regression on the candidate box again to obtain the location of the object, and the Softmax is used to classify the candidate box to obtain the object category in the image. After obtaining the location of the object and the corresponding object category, the store's super-store operation behavior can be judged and warned. Among them, L1 loss is a commonly used regression loss function, which is used to measure the absolute difference between the predicted value and the true value. In bounding box regression, L1 loss is often used to calculate the position error of the candidate box. The goal of bounding box regression is to adjust the position of the initial candidate box by the predicted value to more accurately locate the object. In order to measure the difference between the predicted value and the true value, the L1 loss function can be used, which calculates the absolute difference between the predicted value and the true value. Assuming that the predicted bounding box position is (x, y, w, h) and the true bounding box position is (x', y', w', h'), the L1 loss can be expressed as: L1 loss = |x-x'|+|y-y'|+|w-w'|+|h-h'|, where |*| represents the absolute value operation. The smaller the L1 loss, the closer the predicted position is to the true position. By using L1 loss as the objective function for training, the position of the candidate box can be adjusted by minimizing the L1 loss to better match the actual position of the object.

[0052] S3. Perform object recognition based on the judgment results of the store's super-store business behavior, and implement alarms based on the results of object recognition; at the object recognition intersection, draw the door line on the image according to the coordinates. When implementing the alarm, design the alarm standard according to the actual application scenario. When the identified object is a super-store, first determine whether the confidence of the network output of the object exceeds the threshold of 0.5. The threshold of 0.5 is used as an empirical choice in some scenarios. It can balance the accuracy and recall rate to a certain extent, and provides a relatively reasonable judgment standard. If it is less than 0.5, filter the object, and then judge the size of the object. If the area of ​​the object is less than 0.05 of the image, it is far away from the camera in this case and is ignored. This can improve accuracy in actual applications and reduce the occurrence of false alarms.

[0053] In summary, the image information of the store and the super-store standard of the current store are obtained through a camera, and then the super-store object detection is performed on the image through the deep learning algorithm. Finally, the object recognition is performed according to the super-store business behavior judgment result of the store, and an alarm is implemented according to the object recognition result. Super-store object detection can be performed only using the position information of the door line coordinates. It does not rely on the information of the preset position, and it is easy to still have a good effect after the scene changes. It can freely adapt to the required scene. Moreover, the image information of the store and the super-store standard of the current store are obtained by a single camera, which reduces the demand for special hardware or the number of machines and saves manpower and material resources.

[0054] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for detecting objects in a hyperstore based on deep learning, characterized in that: The steps include: S1, using a camera to obtain the image information of the store and the super-store standard of the current store, that is, the location information of the door line; S2. Perform super-store object detection on the image through deep learning algorithm, specifically: S2.

1. First, use ResNet101 as the basic grid to extract feature maps from the input image, obtain feature maps of multiple scales and predict the position map of the goal line in the image; S2.2, then concatenate the feature maps of multiple scales and the position map of the door line in the predicted image on the channel and input them into the regional generation network RPN; S2.3, ResNet101 and feature maps of multiple scales and network RPN are used to generate a large number of candidate boxes, and a fixed-size candidate box feature map is obtained through the RoI pooling layer; S2.4, perform a full connection operation on the fixed-size candidate frame feature map through a fully connected layer to obtain the location of the object and the category of the object in the image, and after obtaining the location of the object and the corresponding object category, judge and identify the super-store operation behavior of the store; S3. Perform object recognition based on the store's extra-store business behavior judgment result, and implement an alarm based on the result of the object recognition.

2. The method for detecting objects in a super-store based on deep learning according to claim 1, characterized in that: In step S1, the camera is arranged above the current store, and the door line information in the current scene is obtained by using the camera monitoring video information as the data source, and the approximate position of the door line is marked with a straight line.

3. The method for detecting objects in a super-store based on deep learning according to claim 2, characterized in that: The position of the door line is marked by obtaining the horizontal and vertical coordinates of the two ends of the line segment. In the deep learning algorithm training and the process of recognition using the trained algorithm, it is necessary to draw the door line on the image according to the coordinates, thereby assisting the recognition of super stores.

4. The method for detecting objects in a super-store based on deep learning according to claim 3, characterized in that: In step S2, when performing super-store object detection on the image, a target detection model is first established based on the FasterR-CNN network, and ResNet101 is used as the basic network to extract feature maps from the input image, where each convolutional layer is followed by a ReLU layer and a BatchNormalization layer.

5. The method for detecting objects in a super-store based on deep learning according to claim 4, characterized in that: In step S2.1, ResNet101 is divided into two branches: The first branch: input the feature map into a fully convolutional network FCN to predict the position of the door line in the image. The FCN includes 4 3x3 convolutional layers; The second branch: input the obtained feature map into the feature pyramid network FPN to obtain feature maps of multiple scales to improve the detection effect of objects of different sizes.

6. The method for detecting objects in a super-store based on deep learning according to claim 5, characterized in that: In step S2.2, when generating the network RPN, first generate pre-selected box anchors of different sizes and ratios for each pixel on the feature map. The specific number of each pre-selected box is controlled by two parameters: scales and aspectratios. Scale refers to the pixel size of the longest side of each shape frame, and aspectratios refers to the aspect ratio of the pre-selected box. At each position, scales = [8, 16, 32] and aspectratios = [0.5, 1, 2] are generated, with a total of 3x3 = 9 pre-selected boxes.

7. The method for detecting objects in a super-store based on deep learning according to claim 6, characterized in that: The network RPN includes three convolutional layers: first, the feature map passes through a 3x3 convolutional layer to make the extracted features more robust, and then two 1x1 convolutional layers are performed respectively, one of which is responsible for predicting the foreground and background judgment, and the number of channels of the output feature map is 2x9=18, indicating the scores of the 9 pre-selected boxes at each position belonging to the foreground and background respectively; the other convolutional layer is responsible for predicting the offset value of the pre-selected box and the real box (x, y, w, h), where (x, y) represents the coordinates of the center point of the box, and (w, h) represents the width and height. The number of channels of the convolutional layer output feature map is 4x9=36, representing the offset value of each pre-selected box (x, w, w, h). The pre-selected box will adjust its size and position according to the predicted offset value to form an accurate candidate box.

8. The method for detecting objects in a super-store based on deep learning according to claim 7, characterized in that: In step S2.3, the fixed-size candidate box feature map can be uniformly entered into the back to perform target recognition and positioning using a fully connected operation.

9. The method for detecting objects in a super-store based on deep learning according to claim 8, characterized in that: In step 2.4, during the full connection operation, the L1 loss is used to perform bounding box regression on the candidate box again to obtain the location of the object, and Softmax is used to classify the candidate box to obtain the object category in the image. After obtaining the object location and the corresponding object category, the store's super-store operating behavior can be judged and warned.

10. The method for detecting objects in a super-store based on deep learning according to claim 9, characterized in that: In step S3, during object recognition, the door line is first drawn on the image according to the coordinates. When the alarm is implemented, the alarm standard is designed according to the actual application scenario. When the identified object is a super store, first determine whether the confidence of the object output by the network exceeds the threshold value of 0.

5. If it is less than 0.5, filter the object, and then determine the size of the object. If the area of ​​the object is less than 0.05 of the image, it is far away from the camera and is ignored.

Images