A method for real-time multi-target detection based on a dual-spectrum camera

Abstract

This invention discloses a real-time multi-target detection method based on a dual-spectrum camera. This method employs an improved SSD target detection algorithm. During the training phase of the detection algorithm, the detection process is visualized, presenting details of the detection at each layer of feature maps. Appropriate feature maps are selected for detection, and an SE module is added to the detection network. This module can enhance useful features based on their importance, thus addressing the issues of missed and false detections of small targets. The added SE module does not alter the original network structure, offers fast processing speed, minimal increase in computational cost, and does not affect real-time detection. It reduces the missed detection rate of small targets to a certain extent, improves detection accuracy, and performs better for both small and occluded targets.

Description

Technical Field

[0001] This invention relates to the field of target detection, specifically a method for real-time multi-target detection based on a dual-spectrum camera. Background Technology

[0002] With the rapid development of the surveillance industry, surveillance is playing an increasingly important role in many fields such as transportation, security, military, and healthcare. Object detection technology is also constantly improving and developing, attracting widespread attention across various sectors, and object detection algorithms are becoming increasingly prevalent. Currently, object detection algorithms are mainly divided into two types: one-stage and two-stage. Two-stage algorithms mainly include the Region with CNN features (R-CNN) algorithm series. These algorithms generally have high detection accuracy but are relatively slow, making them difficult to use for real-time detection in surveillance systems. One-stage algorithms include You Only Look Once (YOLO) and Single Shot Multibox Detector (SSD). These algorithms have fast detection speeds but slightly lower accuracy.

[0003] As a one-stage algorithm, SSD uses regression for classification and localization, making it faster than the two-stage Faster R-CNN. Furthermore, SSD employs the anchor concept of Faster R-CNN for prediction, resulting in higher detection accuracy than YOLO. However, SSD still suffers from missed or false detections when dealing with small targets. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a real-time multi-target detection method based on a dual-spectrum camera. By improving the SSD target detection model, the detection accuracy is enhanced, and missed or false detections are avoided.

[0005] To solve the aforementioned technical problem, the present invention adopts the following technical solution: a multi-target real-time detection method based on a dual-spectrum camera, wherein the dual-spectrum camera includes a visible light camera and an infrared camera, and includes the following steps:

[0006] S01) Upon receiving a target detection request, receive the video from the current target detection video source, which is from a dual-spectrum camera;

[0007] S02) Input the video from the current target detection video source into the SSD target detection module for detection, including the following specific steps:

[0008] S21) Divide the video source into multiple frames, preprocess the input images to remove noise, and unify the image size;

[0009] S22) The preprocessed image is fed into the pre-trained improved SSD network. The improved SSD network includes a base network and an additional network. The base network adopts the VGG-16 network and optimizes it. The fully connected layers FC6 and FC7 are converted into convolutional layers. In addition, multiple convolutional layers are added after the base network as an additional network. The additional network connects to the final feature map of the base network and obtains new feature maps through the convolution operation of the additional network.

[0010] S23) Select 6 layers of feature maps of different sizes from the SSD network in step S22) and input them into the SE module. The SE module learns to obtain the importance of each feature channel, redistributes the weights of the feature channels of the feature map, enhances the feature weights of effective channels, and suppresses the feature weights of ineffective channels.

[0011] S24) The weighted feature map is passed into the detection layer. The prior box mechanism is used in the detection layer to generate a prediction box in each feature map. For the generated prediction box, non-maximum suppression is used to remove prediction windows with an overlap greater than a set value. In the position where a lot of prediction boxes are generated, only one window with a maximum confidence is retained, and other non-maximum windows are suppressed to obtain the detection result.

[0012] S03) Determine whether to issue an alarm signal based on the detection results output by the SSD target detection module.

[0013] Furthermore, the SE module processes the feature map as follows:

[0014] S231) Compression operation: Each feature map is compressed along the spatial dimension using a global pooling operation, turning each feature channel into a real number;

[0015] S232), activation operation, after obtaining the real number of the previous operation, add an FC fully connected layer to predict the importance of each channel and generate weights for each feature channel;

[0016] S233) Weight calibration operation: The weights generated in the previous step are multiplied and added to the previous feature map to complete the weight calibration of the original feature map in the channel dimension.

[0017] Furthermore, when training the SSD object detection module, a deconvolutional network is added to visualize the detection algorithm. The deconvolutional network is located after the extended network. The feature maps output by each layer of the base network and the extended network are fed into the deconvolutional network. The deconvolutional network outputs a visualized image. The richness of semantic information in the visualized image is used as the criterion to select the output image of the convolutional layer for object detection.

[0018] Furthermore, the output images of convolutional layers Conv4_3, Conv7, Conv8_2, Conv9_2, Conv10_2, and Conv11_2 are selected for target detection.

[0019] Furthermore, the scale, aspect ratio, and position of the prior boxes are confirmed based on the visualized images. A linearly increasing method is used to confirm the scale of the prior boxes for each feature map. The aspect ratios of the prior boxes are selected as 1, 2, 3, 1 / 2, and 1 / 3, respectively. The center of each unit is selected as the center of the prior box, and the part of the prior box that exceeds the feature map size is directly truncated.

[0020] Further, in step S24), the feature map is set to have a size of m*m and has m*m units. Each unit generates 5 prior boxes of fixed size. The location regression layer outputs four location correction parameters. After the prior boxes are transformed according to the correction parameters, they coincide with the actual labeled area. Then, non-maximum suppression is used to remove prediction windows with high overlap.

[0021] Furthermore, the video source switching modes include timed switching, manual switching, and intelligent switching. In timed switching mode, the target detection video source is set for different time periods based on weather, time, and user experience. In manual switching mode, the target detection video source is switched between visible light camera video and infrared camera video at any time based on the actual situation on site. In intelligent switching mode, the target detection video source is determined to be either visible light camera video or infrared camera video based on the data fed back by the light sensor.

[0022] The beneficial effects of this invention are as follows: This invention adds an SE module to the original SSD network. The SE module learns the importance of each feature channel, redistributes the weights of the feature channels in the feature map, enhances the feature weights of effective channels, and suppresses the feature weights of ineffective channels. The added SE module does not change the original network structure, has a fast processing speed, a small increase in computational cost, and does not affect real-time detection. It reduces the false negative rate of small targets to a certain extent, improves detection accuracy, and performs better in detecting small targets and occluded targets. During training, a deconvolutional network is added to visualize the detection algorithm. Based on the visualized image, convolutional layer images with more high-level semantic information are selected for target detection. Furthermore, the scale, aspect ratio, and position of the prior bounding boxes can be confirmed based on the visualized image. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the network described in Example 1. Detailed Implementation

[0024] The present invention will be further described below with reference to specific embodiments.

[0025] Example 1

[0026] The technical solutions in the embodiments of the present invention will be clearly and completely described below. The embodiments described in this invention are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0028] This embodiment discloses a multi-target real-time detection method based on a dual-spectrum camera. The dual-spectrum camera in this method includes a visible light camera and an infrared camera, enabling real-time monitoring throughout the day. The dual-spectrum camera can switch the video source for target detection at set times, manually, or intelligently to achieve better detection results and reduce missed and false detections.

[0029] This method processes real-time images from video sources to detect the presence of targets matching certain features. It is an improved SSD target detection algorithm. To further explore the algorithm's effectiveness and limitations, the detection process is visualized during training, showcasing the details of each feature map layer's detection. Appropriate feature maps are selected for detection, and analysis reveals that the importance of different detection channels varies. Useful features can be enhanced based on importance to address the issues of missed and false detections of small targets. Therefore, an SE module is added to the detection network. The detection framework is as follows: Figure 1 As shown, the improved SSD target detection algorithm can achieve a detection speed of 63 frames per second, meeting the requirements for real-time detection.

[0030] The specific steps of this method are as follows:

[0031] S1: Upon receiving a target detection request, receive the video from the current target detection video source.

[0032] S2: Transfer the video from the current video source to the improved SSD object detection module.

[0033] S3: Based on the detection results transmitted by the detection module, determine whether to issue an alarm signal.

[0034] Further:

[0035] S11: In this method, the video source switching modes are divided into timed switching, manual switching, and intelligent switching. In timed switching mode, users can set the target detection video source for different time periods based on weather, time, and usage experience. For example, the video source can be set to the visible light camera during the day and the video source can be set to the thermal imaging camera at night. In manual switching mode, the target detection video source can be switched to visible light or thermal imaging at any time according to the actual situation on site. In intelligent switching mode, based on the data fed back by the light sensor, when the light intensity is low, the video from the thermal imaging camera contains more and clearer information, so the video source is switched to the thermal imaging camera; when the light intensity is high, the video from the visible light camera contains more and clearer information, so the video source is switched to the visible light camera.

[0036] Step S2 is as follows:

[0037] S21: Perform necessary preprocessing on the input image, remove noise from the image, and unify the image size.

[0038] S22: Input the pre-processed image into the pre-trained improved SSD network.

[0039] S23: The improved SSD network consists of a base network and supplementary networks. By comparing the classification accuracy and complexity of commonly used convolutional networks, this system selects a portion of the VGG-16 network as the base network and optimizes it. The fully connected layers FC6 and FC7 are converted into convolutional layers, and multiple convolutional layers are added after the base network as supplementary networks. The supplementary networks connect to the final feature map of the base network, and new feature maps are obtained through the convolutional operations of the supplementary networks. Compared with the base network, the feature maps output by the supplementary networks have gradually reduced resolution and contain more high-level global semantic information, which is more conducive to the detection of large-sized objects. The feature maps output by the base network contain more local detail information and are more suitable for the detection of small-sized objects.

[0040] S24: To improve detection accuracy, six feature maps of different scales are selected from the network in the previous step. In this system, the feature maps output by the convolutional layers Conv4_3, Conv7, Conv8_2, Conv9_2, Conv10_2, and Conv11_2 are selected. These feature maps contain different semantic features and positional sensitivities, which can be used for subsequent detection and classification.

[0041] S25: If the feature map obtained in the previous step is directly used for detection and classification, it is found that small targets are prone to missed detections and false detections. Therefore, before performing target detection, the feature map obtained in the previous step is passed to the Squeeze-and-Excitation (SE) module. This module can learn the importance of each feature channel, redistribute the feature channel weights of the feature map, enhance the feature weights of effective channels, and suppress the feature weights of ineffective channels.

[0042] S26: The weighted feature maps are fed into the detection layer. Borrowing the Anchor concept from the Faster R-CNN algorithm, a Prior Box mechanism is used in the detection layer to generate predicted bounding boxes in each feature map. Assuming the feature map size is m*m with m*m units, each unit generates 5 fixed-size Prior Boxes. A location regression layer outputs four location correction parameters. After transformation based on these parameters, the Prior Boxes coincide with the actual labeled regions.

[0043] S27: In the previous step, a large number of prediction boxes were generated. Non-maximum suppression (NMS) was used to remove prediction windows with high overlap. Only one window with maximum confidence was retained in the location where many prediction boxes were generated, and other non-maximum windows were suppressed.

[0044] The SE module in step S25 above does not change the original network structure, improves the SSD network's ability to detect small targets without missing or false detections, and has a fast processing speed, without affecting real-time detection. The specific steps of the SE module in processing the feature map are as follows:

[0045] S251: Squeeze operation, which compresses each feature map along the spatial dimension using a global pooling operation, turning each feature channel into a real number.

[0046] S252: Excitation operation. After obtaining the real number from the previous operation, a fully connected (FC) layer is added to predict the importance of each channel and generate weights for each feature channel.

[0047] S253: Weight scaling operation (Scale), which adds the channel weights generated in the previous step to the previous feature map through multiplication, thus completing the weight scaling of the original feature map in the channel dimension.

[0048] Additionally, the SSD network model needs to be trained. During training, feature maps required for detection and classification need to be selected, and the five prior boxes required in step S26 need to be determined. However, since the internal operating mechanism cannot be seen during detection, a deconvolutional network is used to visualize the detection algorithm. This technique does not require changing the original network structure. The feature maps output from each layer are fed into the deconvolutional network, which will output a visualized image. The deconvolution uses a transposed convolutional kernel (the parameters are the same, but the horizontal and vertical directions of the kernel are flipped). Based on the visualized image, it was found that many feature maps lacked sufficient high-level abstraction features and were unsuitable for further detection tasks. After comparison, the feature maps output by convolutional layers Conv4_3, Conv7, Conv8_2, Conv9_2, Conv10_2, and Conv11_2 possessed more high-level semantic information and could be used for object detection.

[0049] In addition, image visualization facilitates the confirmation of the scale, aspect ratio, and location of the prior boxes. Regarding scale, it was found that as the feature map scale decreases, the area of ​​the target mapped onto the original image becomes larger; therefore, a linearly increasing method was used to confirm the scale of the prior boxes for each feature map. For aspect ratio, to detect more categories, aspect ratios of 1, 2, 3, 1 / 2, and 1 / 3 were selected. For location, the center of each cell was used as the center of the prior box, and any portion of the prior box exceeding the feature map size was directly truncated.

[0050] The improved SSD algorithm proposed in this invention adds an SE module without altering the original network structure, and offers faster processing speed with minimal increase in computational cost, without affecting real-time detection. It reduces the false negative rate for small targets to some extent, improves detection accuracy, and performs better for both small and occluded targets.

[0051] The above description is merely the basic principle and preferred embodiment of the present invention. Improvements and substitutions made by those skilled in the art based on the present invention are within the scope of protection of the present invention.

Claims

1. A method for real-time multi-target detection based on a dual-spectrum camera, wherein the dual-spectrum camera includes a visible light camera and an infrared camera, characterized in that: Includes the following steps: S01) Upon receiving a target detection request, receive the video from the video source of the current target detection, wherein the video source is derived from a dual-spectrum camera; S02) Input the video from the current target detection video source into the SSD target detection module for detection, including the following specific steps: S21) Divide the video source into multiple frames, preprocess the input images to remove noise, and unify the image size; S22) The preprocessed image is fed into the pre-trained improved SSD network. The improved SSD network includes a base network and an additional network. The base network adopts the VGG-16 network and optimizes it. The fully connected layers FC6 and FC7 are converted into convolutional layers. In addition, multiple convolutional layers are added after the base network as an additional network. The additional network connects to the final feature map of the base network and obtains new feature maps through the convolution operation of the additional network. S23) Select 6 layers of feature maps of different sizes from the SSD network in step S22) and input them into the SE module. The SE module learns to obtain the importance of each feature channel, redistributes the weights of the feature channels of the feature map, enhances the feature weights of effective channels, and suppresses the feature weights of ineffective channels. S24) The weighted feature map is passed into the detection layer. The prior box mechanism is used in the detection layer to generate a prediction box in each feature map. For the generated prediction box, non-maximum suppression is used to remove prediction windows with an overlap greater than a set value. In the position where a lot of prediction boxes are generated, only one window with a maximum confidence is retained, and other non-maximum windows are suppressed to obtain the detection result. S03) Determine whether to issue an alarm signal based on the detection results output by the SSD target detection module.

2. The multi-target real-time detection method based on a dual-spectrum camera according to claim 1, characterized in that: The SE module processes the feature map as follows: S231) Compression operation: Each feature map is compressed along the spatial dimension using a global pooling operation, turning each feature channel into a real number; S232), activation operation, after obtaining the real number of the previous operation, add an FC fully connected layer to predict the importance of each channel and generate weights for each feature channel; S233) Weight calibration operation: The weights generated in the previous step are multiplied and added to the previous feature map to complete the weight calibration of the original feature map in the channel dimension.

3. The multi-target real-time detection method based on a dual-spectrum camera according to claim 1, characterized in that: When training the SSD object detection module, a deconvolutional network is added to visualize the detection algorithm. The deconvolutional network is located after the additional network. The feature maps output by each layer of the base network and the additional network are fed into the deconvolutional network. The deconvolutional network outputs a visualization image. The richness of semantic information in the visualization image is used as the criterion to select the output image of the convolutional layer for object detection.

4. The multi-target real-time detection method based on a dual-spectrum camera according to claim 3, characterized in that: The output images of convolutional layers Conv4_3, Conv7, Conv8_2, Conv9_2, Conv10_2, and Conv11_2 are selected for object detection.

5. The multi-target real-time detection method based on a dual-spectrum camera according to claim 1, characterized in that: The scale, aspect ratio, and position of the prior boxes are confirmed based on the visualized images. A linearly increasing method is used to confirm the scale of the prior boxes for each feature map. The aspect ratios of the prior boxes are selected as 1, 2, 3, 1 / 2, and 1 / 3, respectively. The center of each cell is used as the center of the prior box, and any part of the prior box that exceeds the feature map size is directly truncated.

6. The multi-target real-time detection method based on a dual-spectrum camera according to claim 1, characterized in that: In step S24), let the size of the feature map be m. m, has m The system consists of m cells, each generating 5 fixed-size prior boxes. A location regression layer outputs four location correction parameters. After the prior boxes are transformed according to the correction parameters, they coincide with the actual labeled regions. Then, non-maximum suppression is used to remove prediction windows with high overlap.

7. The multi-target real-time detection method based on a dual-spectrum camera according to claim 1, characterized in that: The video source switching modes include timed switching, manual switching, and intelligent switching. In timed switching mode, the target detection video source is set for different time periods based on weather, time, and user experience. In manual switching mode, the target detection video source is switched between visible light camera video and infrared camera video at any time based on the actual situation on site. In intelligent switching mode, the target detection video source is determined to be either visible light camera video or infrared camera video based on the data fed back by the light sensor.

Images