Small target detection method and system

Abstract

The invention belongs to the technical field of small target detection in a complex environment, and relates to a small target detection method and system. The method comprises the following steps: S1, performing conversion and framing screening on a collected monitoring video to obtain a data set; s2, performing multi-scale feature extraction of a small target object on the data set; s3, improving a transformation neural network structure based on deformable convolution and a spatial domain attention mechanism, and performing image feature extraction on the data set processed in the step S2; s4, training the data set processed in the step S3 to obtain a weight model; and S5, loading the weight model, accessing a video source, detecting the video source by using the weight model, and outputting a result. The system comprises a conversion and framing screening module, a multi-scale feature extraction module, an image feature extraction module, a training module and a detection module. Through multi-scale feature extraction and small target recognition, a complete algorithm and system for detecting small targets in a video are provided, and a small target area in the video is stably and accurately detected.

Description

Technical Field

[0001] The present invention belongs to the technical field of small target detection in complex environments, and in particular relates to a small target detection method and system. Background Art

[0002] With the development of computer vision and deep learning, target detection technology based on deep learning has made great progress and is widely used in face recognition, video surveillance, smart transportation, industrial detection and other fields. Small target object recognition and detection has always been an important issue affecting target detection performance and has long been a difficulty and research hotspot in computer vision.

[0003] The video surveillance on the site of the petroleum and petrochemical industry is installed at a high position with a wide field of view; the identification of small targets includes key areas such as well sites, stations, pipelines, oil depots, gas stations, as well as donkey heads, packings, and early oil leakage areas. In special operations, the identification of small targets includes goggles, gas detectors, safety belts, etc.

[0004] The traditional video surveillance system in the petroleum and petrochemical industry records workers' illegal operations and equipment operation by installing a large number of monitoring devices on site. When problems occur, the only way to deal with them is to retrieve relevant monitoring afterwards. Companies using traditional monitoring systems need to arrange a large number of employees to view monitoring images around the clock, which is time-consuming and prone to errors. In this context, machines replacing manual monitoring has become a focus of the petroleum and petrochemical industry, so intelligent video surveillance systems have emerged. Summary of the invention

[0005] In view of the above problems, the present invention proposes a small target detection method and system, which can initially realize the recognition of small target objects in the petroleum and petrochemical industry scenes, and assist safety management personnel in identifying common violations on site.

[0006] The technical solution adopted by the present invention is as follows: a small target detection method, the method comprising the following steps: S1, converting and frame-screening the collected monitoring video to obtain a data set; S2, extracting multi-scale features of small target objects from the data set; S3, improving the transformation neural network structure based on deformable convolution and spatial domain attention mechanism, and extracting image features from the data set processed by step S2; S4, training the data set processed by step S3 to obtain a weight model; S5, loading the weight model, connecting to the video source, using the weight model to detect the video source and outputting the result.

[0007] Furthermore, in step S1, the data set is used for training to obtain a weight file for identifying target categories of small target objects; and then a labeling tool is used to manually label the target categories in the data set and save them in an xml file.

[0008] Furthermore, before step S2, low-light image enhancement processing is performed on the data set processed in step S1.

[0009] Furthermore, the low-light image enhancement processing is performed on the data set processed in step S1, including: color balancing, histogram equalization and Gaussian blur adjustment are performed on the original image of the data set in sequence; then it is input into the Retinex model for sharpening, color enhancement and image brightness enhancement in sequence, and finally a processed low-light enhanced image is obtained.

[0010] Further, in step S2, multi-scale feature extraction of small target objects is performed on the dataset processed by low-light image enhancement, including: obtaining five groups of feature maps of different sizes [c1, c2, c3, c4, c5] from the dataset processed by low-light image enhancement through a bidirectional fusion backbone network structure; then obtaining four groups of feature maps [p4, p3, p2, p1] through a top-down path; adding a bottom-up path aggregation network after the top-down path, and obtaining a total of four feature map fusion modules [N2, N3, N4, N5] through the bottom-up path aggregation network; and then through a shallower N i And deeper p i+1 After fusion, the next layer N is obtained through the mixed domain attention mechanism. i+1 , and finally passes through a feature adaptive pooling layer to output the extracted multi-scale features.

[0011] Furthermore, step S3 includes: improving the transformation neural network structure based on deformable convolution and spatial domain attention mechanism, extracting complete irregular image features from the multi-scale features of the small target object in step S2 through 3×3 deformable convolution; inputting the complete irregular image features into the spatial domain self-attention mechanism, performing corresponding spatial transformation on the spatial domain information in the image, extracting local information, and obtaining the final feature map of each image.

[0012] Furthermore, the deformable convolution formula is: Among them, x is the input feature map, w is the convolution kernel, R is the size set of the receptive field, P0 is the pixel at each position, P n is the size of a receptive field in R, ΔP n is each offset of P0 within the convolution kernel range.

[0013] Furthermore, step S4 includes: training the data set processed by step S3 using structural reparameterization, using a multi-branch structure during training, and fusing the multi-branch structure into a single-path structure during inference, and finally obtaining a weight model.

[0014] Furthermore, the data set processed in step S3 is trained by structural reparameterization. A multi-branch structure is used during training, and the multi-branch structure is fused into a single-path structure during inference. The weight model finally obtained includes:

[0015] The data set processed in step S3 is trained using structural reparameterization. During training, the D×C×K×K convolution in the transformed neural network structure is decomposed into three branches of D×C×K×K, D×C×1×K and D×C×K×1 convolution. At the same time, a batch normalization layer is added after each branch, and then the model is trained. During inference, the D×C×K×K convolution is followed by the ReLu activation function without going through the branch structure. Finally, the weighted model is obtained.

[0016] Furthermore, the monitoring video includes on-site monitoring video taken by a camera and on-site construction video taken by a SLR device.

[0017] Furthermore, the video source is connected by a server, a client, and a database to realize information interaction and is used to detect small targets on site.

[0018] In addition, the present invention also proposes a small target detection system, which includes a conversion and frame screening module, a multi-scale feature extraction module, an image feature extraction module, a training module and a detection module; wherein the conversion and frame screening module is used to convert and frame screen the collected surveillance video to obtain a data set; the multi-scale feature extraction module is used to extract multi-scale features of small target objects from the data set; the image feature extraction module is used to improve the transformation neural network structure based on deformable convolution and spatial domain attention mechanism, and extract image features from the data set after multi-scale feature extraction; the training module is used to train the data set after image feature extraction to obtain a weight model; the detection module is used to load the weight model, access the video source, use the weight model to detect the video source and output the result.

[0019] Furthermore, the system also includes a low-light image enhancement processing module, which is used to perform color balance, histogram equalization and Gaussian blur adjustment on the original image of the data set processed in step S1 in sequence; then input it into the Retinex model for sharpening, color enhancement processing and image brightness enhancement in sequence, and finally obtain a processed low-light enhanced image.

[0020] Furthermore, the image feature extraction module includes an irregular image feature extraction unit and a spatial transformation unit; wherein the regular image feature extraction unit is used to improve the transformation neural network structure based on deformable convolution and spatial domain attention mechanism, and extract complete irregular image features from multi-scale features of small target objects through 3×3 deformable convolution; the spatial transformation unit is used to input the complete irregular image features into the spatial domain self-attention mechanism, perform corresponding spatial transformation on the spatial domain information in the image, extract local information, and obtain the final feature map of each image.

[0021] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention proposes a low-illumination image enhancement algorithm based on RetinexNet, which can effectively denoise and enhance images; through multi-scale feature extraction and small target recognition, a complete algorithm and system for small target detection in videos are proposed, which can stably and accurately detect small target areas in videos; a combination of a bidirectional fusion network and a spatial domain self-attention mechanism is used to fuse multi-scale features, and the receptive field of a deep network is beneficial to the idea of ​​extracting target semantic information, and adding scales for multi-scale fusion makes full use of shallow information to improve the resolution of small targets.

[0022] Other features and advantages of the present invention will be described in the following description, and part of them will become obvious from the description, or be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the description and the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 It is a flowchart of a small target detection method;

[0025] Figure 2 It is a schematic diagram of the bidirectional fusion backbone network structure;

[0026] Figure 3 Schematic diagram of the structure of the improved transformation neural network based on deformable convolution and spatial domain attention mechanism;

[0027] Figure 4 Schematic diagram of a small target detection system. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] The present invention proposes a small target detection method, such as Figure 1 As shown, the following steps are included:

[0030] S1, converting and filtering the collected surveillance video to obtain a data set;

[0031] S2, extract multi-scale features of small target objects in the data set;

[0032] S3, based on deformable convolution and spatial domain attention mechanism, the transformation neural network structure is improved to extract image features from the data set processed in step S2;

[0033] S4, training the data set processed in step S3 to obtain a weight model;

[0034] S5. Load the weight model, access the video source, use the weight model to detect the video source and output the result.

[0035] Specifically, in the embodiment of the present invention, the data set is a fixed-resolution data set obtained by conversion and frame screening, which can be an oilfield data set. The oilfield data set can include two target categories, namely, smoke (cigarette) target and phone (mobile phone) target. In step S1, the oilfield data set is used for training to obtain a weight file for identifying the two target categories of smoke and phone; the labeling tool LabelImg is then used to manually label the target category and location information in the oilfield data set, and save it in an xml file. It should be noted that small target objects are not limited to the two target categories of smoke and phone, but can also be gas detectors, safety belts, protective masks, etc. The monitoring video in step S1 includes on-site monitoring videos taken by cameras and on-site construction videos taken using SLR equipment.

[0036] Before step S2, the data set processed by step S1 needs to be subjected to low-light image enhancement processing; the low-light image enhancement processing includes the following steps: first, the original image of the data set processed by step S1 is subjected to color balance, histogram equalization and Gaussian blur adjustment to reduce noise and improve contrast; then, the image is input into the Retinex model for sharpening, color enhancement and image brightness enhancement to obtain an enhanced image, thereby further improving image quality.

[0037] In the target detection task, there are significant size differences between targets of different categories and even targets of the same category, which makes it difficult to effectively detect targets of different sizes using only a single-scale feature map. In order to utilize multi-scale information, the method currently used more often is to use feature pyramids or feature fusion to comprehensively utilize feature maps of different scales. In the feature pyramid, the size differences of feature maps at different levels are large, and the feature maps of adjacent levels are not continuous in scale. When multi-scale feature maps are fused, due to the discontinuity of scale information, when the deep-level feature map is upsampled to the same resolution as the low-level feature map and then the corresponding position elements are added, the misalignment of feature maps at different levels is ignored, which may destroy the spatial information of the low-level feature map. In order to maintain the continuity of the scale information of feature maps of adjacent levels while also having obvious scale differences, the present invention uses PANet (path aggregation network) as a bidirectional fusion backbone network structure to extract features. In an embodiment of the present invention, the bidirectional fusion backbone network structure is as follows Figure 2 As shown in the figure, the low-light image enhancement dataset is processed by bidirectional fusion backbone network structure to obtain five groups of feature maps of different sizes [c1, c2, c3, c4, c5]; then four groups of feature maps [p4, p3, p2, p1] are obtained through the top-down path. A bottom-up path is added after the top-down path, through which PANet obtains a total of 4 FeatureMap fusion modules [N2, N3, N4, N5]. It is through a shallower N i And deeper p i+1 After fusion, the next layer N is obtained through the mixed domain attention mechanism i+1 Finally, after a feature adaptive pooling layer, the extracted multi-scale features are output. In the fusion process, the mixed domain attention mechanism is added to enable the low-light enhanced dataset to fuse features between different layers to ensure the integrity and diversity of the features.

[0038] In step S3, the transformation neural network structure (such as Figure 3As shown in the figure, the input image features are connected to the average pooling layer and the 1x1 convolution after four 2x2 convolutional layers and the feature extraction module. The feature extraction module uses a jump connection to connect the result of the deformable convolution layer to the deformable pooling layer and the spatial transformation (local network, sampling network and differentiable image sampling), and finally outputs the result through a deformable convolution. Specifically, step S3 includes the following steps: based on the deformable convolution and spatial domain attention mechanism, the transformation neural network structure is improved, and the multi-scale features of the small target object in step S2 are extracted through a 3×3 deformable convolution. The complete irregular image features are input into the spatial domain self-attention mechanism, the spatial domain information in the image is subjected to corresponding spatial transformation, and the local information is extracted to obtain the final feature map of each image.

[0039] Due to the transformation in the spatial domain, it has functions such as image rotation and scaling transformation to extract important local information; at the same time, it reduces the number of network structure parameters and avoids unnecessary calculations. The deformable convolution formula is: Among them, x is the input feature map, w is the convolution kernel, R is the size set of the receptive field, P0 is the pixel at each position, P n is the size of a receptive field in R, ΔP n is the offset of P0 within the convolution kernel range.

[0040] Step S4 includes the following steps: the data set processed by step S3 is trained by structural reparameterization, a multi-branch structure is used during training, and the multi-branch structure is merged into a single-path structure during reasoning, and finally a weight model is obtained. Specifically, the data set processed by step S3 is trained by structural reparameterization, and during training, the convolution of D×C×K×K in the transformation neural network structure is decomposed into three branches of D×C×K×K, D×C×1×K and D×C×1×K, and a batch normalization layer is added after each branch, and then the model is trained. This method can improve the network performance of the convolution. During reasoning, the D×C×K×K convolution is used followed by the ReLu activation function, and the model runs efficiently without going through the branch structure; finally, a weight model is obtained.

[0041] In step S5, the video source is connected by the server, the client and the database to realize information exchange and is used to detect small targets on site.

[0042] In addition, the present invention also proposes a small target detection system, such as Figure 4As shown, the system includes a conversion and frame screening module, a multi-scale feature extraction module, an image feature extraction module, a training module and a detection module; wherein the conversion and frame screening module is used to convert and frame screen the collected surveillance video to obtain a data set; the multi-scale feature extraction module is used to extract multi-scale features of small target objects from the data set; the image feature extraction module is used to improve the transformation neural network structure based on deformable convolution and spatial domain attention mechanism, and extract image features from the data set after multi-scale feature extraction; the training module is used to train the data set after image feature extraction to obtain a weight model; the detection module is used to load the weight model, access the video source, use the weight model to detect the video source and output the result.

[0043] Furthermore, the system also includes a low-light image enhancement processing module, which is used to perform color balance, histogram equalization and Gaussian blur adjustment on the original image of the data set processed in step S1 in sequence; then input it into the Retinex model for sharpening, color enhancement processing and image brightness enhancement in sequence, and finally obtain a processed low-light enhanced image.

[0044] The multi-scale feature extraction module and the image feature extraction module are both used to extract image information of the data set. Specifically, the image feature extraction module includes an irregular image feature extraction unit and a spatial transformation unit; wherein the regular image feature extraction unit is used to improve the transformation neural network structure based on deformable convolution and spatial domain attention mechanism, and extract the complete irregular image features of the multi-scale features of small target objects through 3×3 deformable convolution; the spatial transformation unit is used to input the complete irregular image features into the spatial domain self-attention mechanism, perform corresponding spatial transformation on the spatial domain information in the image, extract local information, and obtain the final feature map of each image.

[0045] Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent substitutions for some of the technical features therein; and these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A small target detection method, characterized in that: The method comprises the following steps: S1, converting and filtering the collected surveillance video to obtain a data set; S2, extract multi-scale features of small target objects in the data set; S3, based on deformable convolution and spatial domain attention mechanism, the transformation neural network structure is improved to extract image features from the data set processed in step S2; S4, training the data set processed in step S3 to obtain a weight model; S5. Load the weight model, access the video source, use the weight model to detect the video source and output the result.

2. A small target detection method according to claim 1, characterized in that: In step S1, a weight file for identifying target categories of small target objects is obtained by training with the data set; and then the target categories in the data set are manually annotated using an annotation tool and saved in an xml file.

3. A small target detection method according to claim 1, characterized in that: Before step S2, low-light image enhancement processing is performed on the data set processed in step S1.

4. A small target detection method according to claim 3, characterized in that: Performing low-light image enhancement processing on the data set processed in step S1 includes: The original images of the dataset are subjected to color balance, histogram equalization and Gaussian blur adjustment in turn; then they are input into the Retinex model for sharpening, color enhancement and image brightness enhancement in turn, and finally the processed low-light enhanced image is obtained.

5. A small target detection method according to claim 3 or 4, characterized in that: In step S2, multi-scale feature extraction of small target objects is performed on the data set after low-light image enhancement processing, including: The low-light image enhancement dataset is processed through a bidirectional fusion backbone network structure to obtain five groups of feature maps of different sizes [c1, c2, c3, c4, c5]; Then four sets of feature maps [p4, p3, p2, p1] are obtained from the top-down path; A bottom-up path aggregation network is added after the top-down path, and four feature mapping fusion modules [N2, N3, N4, N5] are obtained through the bottom-up path aggregation network; Then through a shallower N i And deeper p i+1 After fusion, the next layer N is obtained through the mixed domain attention mechanism. i+1 , and finally passes through a feature adaptive pooling layer to output the extracted multi-scale features.

6. A small target detection method according to claim 1, characterized in that: Step S3 includes: Based on the deformable convolution and spatial domain attention mechanism, the transformation neural network structure is improved, and the complete irregular image features are extracted from the multi-scale features of the small target object in step S2 through 3×3 deformable convolution; The complete irregular image features are input into the spatial domain self-attention mechanism, the spatial domain information in the image is transformed accordingly, the local information is extracted, and the final feature map of each image is obtained.

7. A small target detection method according to claim 6, characterized in that: The deformable convolution formula is: Among them, x is the input feature map, w is the convolution kernel, R is the size set of the receptive field, P0 is the pixel at each position, P n is the size of a receptive field in R, ΔP n is each offset of P0 within the convolution kernel range.

8. A small target detection method according to claim 1, characterized in that: Step S4 includes: The data set processed in step S3 is trained by structural reparameterization. A multi-branch structure is used during training, and the multi-branch structure is fused into a single-path structure during inference to finally obtain a weighted model.

9. A small target detection method according to claim 1, characterized in that: The data set processed in step S3 is trained by structural reparameterization. A multi-branch structure is used during training, and the multi-branch structure is fused into a single-path structure during inference. The final weight model includes: The data set processed in step S3 is trained using structural reparameterization. During training, the D×C×K×K convolution in the transformed neural network structure is decomposed into three branches of D×C×K×K, D×C×1×K and D×C×K×1 convolution. At the same time, a batch normalization layer is added after each branch, and then the model is trained. During inference, the D×C×K×K convolution is followed by the ReLu activation function without going through the branch structure. Finally, the weighted model is obtained.

10. A small target detection method according to claim 1, characterized in that: The surveillance videos include on-site surveillance videos shot by cameras and on-site construction videos shot using SLR equipment.

11. A small target detection method according to claim 10, characterized in that: The video source is connected by the server, client and database to realize information exchange and is used to detect small targets on site.

12. A small target detection system, characterized in that: The system includes a conversion and frame screening module, a multi-scale feature extraction module, an image feature extraction module, a training module and a detection module; wherein, The conversion and frame filtering module is used to convert and frame filter the collected surveillance video to obtain a data set; The multi-scale feature extraction module is used to extract multi-scale features of small target objects in the data set; The image feature extraction module is used to improve the transformation neural network structure based on deformable convolution and spatial domain attention mechanism, and perform image feature extraction on the data set after multi-scale feature extraction; The training module is used to train the data set after image feature extraction to obtain a weight model; The detection module is used to load the weight model, access the video source, use the weight model to detect the video source and output the results.

13. A small target detection system according to claim 12, characterized in that: The system also includes a low-light image enhancement processing module, which is used to perform color balance, histogram equalization and Gaussian blur adjustment on the original image of the data set processed in step S1 in sequence; then input it into the Retinex model for sharpening, color enhancement and image brightness enhancement in sequence, and finally obtain a processed low-light enhanced image.

14. A small target detection system according to claim 12 or 13, characterized in that: The image feature extraction module includes an irregular image feature extraction unit and a space transformation unit; wherein, The regular image feature extraction unit is used to improve the transformation neural network structure based on deformable convolution and spatial domain attention mechanism, and extract complete irregular image features from multi-scale features of small target objects through 3×3 deformable convolution; The spatial transformation unit is used to input the complete irregular image features into the spatial domain self-attention mechanism, perform corresponding spatial transformation on the spatial domain information in the image, extract local information, and obtain the final feature map of each image.

Images