Target detection method for severe weather based on multi-scale feature extraction and model quantification

Abstract

The invention relates to a severe weather target detection method based on multi-scale feature extraction and model quantification. The method comprises the following steps: 1) preprocessing an image; 2) fusing image information; 3) extracting image information; and 4) carrying out image feature prediction to obtain segmentation anchor points. The method can effectively process images in low-illumination and high-noise environments, accurately recognize target objects hidden in dark places, can accurately recognize targets in night images, improves the detection effect of the night images in dense fog weather, has weak influence on the model performance by image resolution differences, and improves the detection accuracy of the night images. And the model can keep a consistent detection effect under different image qualities.

Description

Technical Field

[0001] The present invention belongs to the technical field of bad weather target detection tasks, and is used to detect targets such as people and vehicles in images under various weather conditions such as foggy days and rainy days. Specifically, it relates to a bad weather target detection method based on multi-scale feature extraction and model quantization. Background Art

[0002] Object detection is a core research direction in the field of computer vision and is widely used in various practical scenarios, such as medical image analysis and industrial inspection. In the object detection task, the computer identifies and labels different types of objects by analyzing each pixel in the image, thereby locating and classifying specific objects. Usually, the area in the image can be divided into background and foreground. The background includes roads, sky, walls, etc., while the foreground is the target object, such as people, animals, vehicles, etc.

[0003] Traditional image segmentation and object detection methods rely on manually selecting low-level features in images, such as color, texture, and spatial structure, to predict and segment the boundaries of objects. Common methods include segmentation based on graph theory, segmentation based on clustering, and segmentation based on decision trees. However, with the rapid growth of computing power and data volume, deep learning technology has been widely used in various computer vision tasks. Deep neural networks, especially convolutional neural networks (CNNs), can automatically extract features from a large amount of labeled data and effectively perform object detection and segmentation tasks through learning. Deep learning has shown excellent performance in image segmentation and object detection tasks, especially under clear image conditions. However, in practical applications, severe weather such as fog and snow have a serious impact on image quality, resulting in blurred boundaries and difficulty in recognition of objects. This not only increases the complexity of the detection algorithm, but also affects the accuracy and robustness of the model.

[0004] In order to cope with the impact of bad weather on algorithm accuracy, research in recent years has begun to focus on how to improve the performance of target detection under bad weather conditions. In particular, for fully supervised deep learning models, although they rely on a large amount of accurately labeled data, in practical applications, data labeling is costly, time-consuming, and difficult to obtain on a large scale, especially in complex and changing environments such as border monitoring. Therefore, studying how to accurately extract high-dimensional semantic edge information in bad weather and use this information to improve the robustness of target detection has become an important research topic at present.

[0005] Currently, target detection algorithms are mainly divided into two types: (1) Two-stage target detection algorithms, such as Fast R-CNN, Faster R-CNN, Mask R-CNN, etc. The characteristic of the above algorithms is that the target detection process is divided into two stages. First, the candidate target information is extracted using the RPN network, and then the candidate object is located and classified through the detection network; (2) One-stage target detection algorithms, such as SSD, YOLO, YOLO 9000, YOLO V3 algorithm, etc. The early representatives of the one-stage network, SSD and YOLO, were proposed mainly to meet the real-time requirements of the project, and to improve the detection accuracy of the algorithm network as much as possible under the requirement of real-time. The one-stage network is also divided into the skeleton network, the neck network and the head network, but the recognition and positioning are both performed in the same branch of the head network. Compared with the two-stage detection algorithm of the same period, the one-stage detection algorithm has lower detection accuracy and faster speed. Due to its strong real-time performance, it is widely used in actual engineering.

[0006] Abroad, due to the large number of small targets in the MS COCO dataset, it has developed together with the accompanying target detection algorithm. At the same time, there are also some studies, especially for the detection features of small targets, which have been specially studied and improved, and the detection accuracy and efficiency of small targets have been significantly improved, such as the classic FPN network, SNIP network, etc. The article "Augmentation for small object detection" performs data augmentation for small targets, effectively expands the data set of small targets, and achieves the effect of improving detection accuracy. For example, ALFNet, Perceptual Generative Adversarial Networks, etc. have proposed some methods and strategies for data augmentation for small targets. The article "Automated detection of COVID-19 cases using deep neural networks with X-ray images" proposes a method of adding filtering to the YOLO algorithm model to improve the detection accuracy of the target, and realizes the introduction of different filters on all 17 convolutional layers. VIOLAP proposes a new fast detection algorithm that can quickly process images and has high detection accuracy. This fast detection algorithm has three major contributions. The first method is to introduce a new image representation called "integral image", which can quickly extract the features required by the detector. The second major contribution is the learning method based on the AdaBoost algorithm, which can extract fewer important visual features from a large amount of image data, thereby achieving very effective classification of the target. The third major contribution is the application of the "cascade" method to the classifier, which can eliminate the influence of the background on the detection and focus the operation on the object category area. Compared with previous methods, the cascade can be regarded as a special focusing mechanism that ensures that there are no related objects in the discarded area. In terms of face recognition, the detection accuracy of this system reached the detection accuracy of the most advanced face recognition technology system at that time. In application, the detector can run at a speed of 15 frames per second, and does not require the use of skin color detection and image difference methods.

[0007] However, the above methods are often obtained based on high-quality image training. In actual application scenarios, image resolution varies greatly, and the edges of targets in low-resolution images are unclear. At the same time, due to the complexity of application scenarios, a single model is difficult to solve the target detection problem. In addition, there are differences in computing power resources in actual application scenarios. In some areas, it is difficult to build computing power resources and model deployment is difficult. Summary of the invention

[0008] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a severe weather target detection method based on multi-scale feature extraction and model quantization. The multi-scale target detection method based on pre-sampling is used to accurately extract image features. In addition, in order to apply the model to application scenarios with different computing power conditions, the model storage occupancy is reduced through mixed precision and pre-compilation schemes, thereby improving the model execution efficiency.

[0009] The present invention solves the technical problem by the following technical solutions:

[0010] A method for detecting targets in severe weather based on multi-scale feature extraction and model quantization, the method comprising the following steps:

[0011] S1. Reduce the image size by bicubic linear interpolation, then introduce the horizontal Sobel operator, vertical Sobel operator and composite Sobel operator to perform differential estimation on the image edge and expand the edge information in the image. The weight matrix of the horizontal Sobel operator and the vertical Sobel operator is:

[0012]

[0013] S2, get the convolution kernel S x , S y Then, use G x =S x A and G y =S y A performs horizontal and vertical edge detection, and uses an approximate fusion method to fuse the horizontal and vertical edges. The fusion function used is:

[0014] |G|=|G x |+|G y |

[0015] S3, data reception, queuing and scheduling are realized through the routing branch detection algorithm, and the pre-processed images are allocated according to the priority order by using the resource scheduling module to schedule computing resources; in the corresponding computing resources, the multi-scale feature extraction module is used to extract key information from feature maps of different sizes to enhance the model detection accuracy;

[0016] S4. According to the extracted image features, the free anchor prediction module is used to predict. When predicting, it does not rely on fixed-size anchors, but directly predicts the center and size of the detection target. After obtaining the image features, the segmentation result extraction module is used to extract the results to obtain the final segmentation anchor. The final segmentation anchor is in the form of (b x ,b y ,b h ,b w,c,p), directly predict the position and size of the anchor point, and predict the confidence of each category.

[0017] The positive effects that the present invention can produce are:

[0018] 1. The model Monitor-Net of the present invention can effectively process images in low-light and high-noise environments through a multi-scale feature extraction network and the introduction of edge information, accurately identify target objects hidden in the dark, and accurately identify targets in night images. The detection effect in foggy weather and night images is improved, and the image resolution difference has little effect on the model performance. The model can maintain consistent detection effects under different image qualities.

[0019] 2. The model Monitor-Net of the present invention has strong versatility and can be deployed in different experimental environments. The execution speed difference under different hardware conditions is small. On high-performance servers or embedded devices with limited resources, Monitor-Net can execute tasks at a relatively consistent speed and maintain a high detection efficiency. At the same time, the detection rate of the model in different experimental environments has a small fluctuation range, which further proves its stability and reliability. In different application scenarios, whether in dynamic or static environments, Monitor-Net can maintain a high consistency and provide excellent detection performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a schematic diagram of the model architecture of the present invention;

[0021] Figure 2 It is a schematic diagram of a free anchor point prediction module of the present invention;

[0022] Figure 3 It is a schematic diagram of the prediction result fusion module of the present invention;

[0023] Figure 4 It is a schematic diagram of the resource scheduling module of the present invention. DETAILED DESCRIPTION

[0024] The present invention is further described in detail below through specific examples. The following examples are only illustrative and not restrictive, and the protection scope of the present invention cannot be limited thereto.

[0025] A method for detecting targets in bad weather based on multi-scale feature extraction and model quantization. Its innovations are: a Monitor-Net model architecture is proposed. The Monitor-Net model architecture is improved based on the characteristics of limited definition of the detected target and high contrast in outdoor environments. In order to solve the problem of limited computing power in the use environment, a multi-path detection algorithm is introduced and a detection routing algorithm is used for control;

[0026] The steps of the method are:

[0027] S1. Reduce the image size by bicubic linear interpolation, then introduce the horizontal Sobel operator, vertical Sobel operator and composite Sobel operator to perform differential estimation on the image edge and expand the edge information in the image. The weight matrix of the horizontal Sobel operator and the vertical Sobel operator is:

[0028]

[0029] S2, get the convolution kernel S x , S y Then, use G x =S x A and G y =S y A performs horizontal and vertical edge detection, and uses an approximate fusion method to fuse the horizontal and vertical edges. The fusion function used is:

[0030] |G|=|G x |+|G y |

[0031] S3, data reception, queuing and scheduling are realized through the routing branch detection algorithm, and the pre-processed images are allocated according to the priority order by using the resource scheduling module to schedule computing resources; in the corresponding computing resources, the multi-scale feature extraction module is used to extract key information from feature maps of different sizes to enhance the model detection accuracy;

[0032] S4. According to the extracted image features, the free anchor prediction module is used to predict. When predicting, it does not rely on fixed-size anchors, but directly predicts the center and size of the detection target. After obtaining the image features, the segmentation result extraction module is used to extract the results to obtain the final segmentation anchor. The final segmentation anchor is in the form of (b x ,b y ,b h ,b w ,c,p), directly predict the position and size of the anchor point, and predict the confidence of each category.

[0033] Monitor-Net contains feature extraction networks of various scales for detection at different speeds and accuracies. The overall framework is as follows: Figure 1 In this framework, the feature extraction network is mainly responsible for prediction and regression tasks, so this module undertakes the main task of detection. Monitor-Net contains three scales of detection networks, and the backbone architecture is shown in Table 1 below:

[0034] Table 1

[0035]

[0036]

[0037] During the pre-training process, classification data sets are used for classification training to enhance the network's feature extraction capabilities. After multi-scale features, regression prediction is performed on the feature map to obtain three confidence matrices, with matrix dimensions of [x, y, h, w, c], such as Figure 2 As shown, c is the confidence value. The regression matrices obtained by the fast detection framework, balanced detection framework, and fine detection framework are specified as m1, m2, and m3 respectively. The confidences obtained by the three methods are fused using a soft fusion method. Since the results obtained by the fine detection framework are usually more accurate, the fusion is more inclined to adopt the results of the precise framework detection, such as Figure 3 The specific fusion method is:

[0038] m=λ 2 ·m1+λ·m2+(1-λ-λ 2 )·m3

[0039] During the experiment, the value of λ was 0.3, and most decisions came from the fine detection framework. In actual use, considering the limitations of the device, only the fast detection model can be deployed to increase the detection speed and reduce the computing power requirements. After testing, the fast detection model can complete the target detection task at a speed of 5FPS in a high-performance single-chip microcomputer (Respberrypi 4B+8G). The training process takes the balanced detection framework as an example, using SGD as the optimizer, and the exponential decay method to optimize the learning rate. The data randomly selects 15,000 mixed data from the total data set for detection, 80% of which are used as training sets and 20% as test sets for testing. The loss function iteration during the training process is as follows, and convergence can be completed in 1200 rounds. The test accuracy reached 73.4%, and the detection efficiency was 117FPS (computing card: 2080Ti).

[0040] Considering the network and computing power limitations in the deployment area. It will bring a series of problems and challenges, mainly including the following aspects. The first is the communication delay problem. Network limitations may lead to increased communication delay. In some areas, the network bandwidth is limited and the communication speed is slow, which will have a negative impact on the real-time and responsiveness of the model. Especially for tasks with high real-time requirements, such as real-time image processing or video analysis, communication delays may cause performance degradation. Secondly, the model size and computing power requirements. The limitations of computing power may limit the scale and complexity of deployable models. In some deployment areas, hardware resources may be limited and cannot support large-scale and complex deep learning models. This may require streamlining the model or using lightweight models to adapt to hardware limitations. Finally, there is the real-time requirement. The lack of computing power of some nodes may make it impossible to meet the real-time requirements. For example, real-time image processing on edge devices may require higher computing resources, and limited computing power may make real-time performance impossible to guarantee.

[0041] At the same time, there are also some difficulties in deployment and maintenance. In areas with limited network and computing power, deploying and maintaining deep learning models may become more complicated. Hardware problems in remote areas and unstable network connections may lead to system instability and unpredictable problems.

[0042] In order to solve the above problems, the project uses the resource scheduling module to maintain the computing power nodes as pluggable modules to better utilize the existing computing power.

[0043] Use queue management to maintain all CPU / GPU computing resources. Queue management is an important part of the resource management system, which is responsible for receiving, queuing and scheduling tasks submitted by users.

[0044] When a suspicious target is found or the target recognition network produces a detection result with low confidence, more computing power is required for refined detection. First, formulate a detectable priority sequence table, that is, the minimum required resources needed for multi-layer detection. For example, all computing resources can be used for rapid target detection, balanced target detection can only be performed by containers containing GPUs, and refined target detection can only be performed by the computing power of high-performance GPUs. For each task, a feasible rating table is formulated. Then use load balancing to select the first executable container to undertake the computing task. Load Balancing Priority is a scheduling strategy whose goal is to achieve balanced distribution of tasks in the GPU resource management system to make full use of all available GPU and CPU devices and reduce system load unevenness. The primary goal is to maintain load balance among the GPU devices in the system to avoid overloading some GPU devices while other devices are idle. For a large number of detection tasks, multiple computing resources can be used in a balanced manner, such as Figure 4 shown.

[0045] The load balancing priority strategy is of great significance in improving system resource utilization, reducing energy consumption, and improving overall performance. When implementing it, it is necessary to comprehensively consider the actual system environment to achieve the best effect.

[0046] The deployment environment in the actual environment cannot guarantee computing power, so the quantization method is used to optimize the model. Quantization optimization is conducive to deploying the model more reliably on a variety of devices.

[0047] Model quantization can effectively reduce the computational burden, parameter size, and memory usage by converting floating-point calculations into low-bit fixed-point calculations. This technique usually results in a significant decrease in accuracy. Therefore, the project uses statistical methods to compensate for the accuracy loss caused by quantization after quantization.

[0048] In the quantization process, we first prepare the trained model and load and fine-tune it. After the fine-tuning is successful, the project uses the OpenVINO tool to build the quantized model. The OpenVINO toolkit is an open source toolkit that can reduce latency, increase throughput, and accelerate the AI ​​reasoning process while maintaining accuracy, reducing the model footprint, and optimizing hardware usage.

[0049] Compared with other model compression tools such as "onnx", OpenVINO can better support dynamic precision and integer quantization. It also supports a variety of development boards, such as embedded devices such as stm32, providing a working basis for subsequent embedded optimization. Next, use the packaging tool to package the model into a binary file. At the same time, integer optimization needs to consider the statistical information of the data set to optimize the precision loss. In order to further improve the detection accuracy and execution efficiency of the model, statistics are used to post-process the model, and a pre-compilation method is introduced to further accelerate the model.

[0050] After obtaining the quantitative data set, during the deployment process, it is necessary to use nncf to introduce statistical post-processing to optimize the accuracy of the quantitative model. At the same time, the model is packaged and used using the compilation method and an external calling interface is provided to facilitate subsequent deployment.

[0051] In order to promote the application of the Monitor-Net model trained by the present invention to environments with different computing power conditions, the present invention adopts model quantization technology to reduce the memory usage of the model and further improve the computing speed, so that it still has real-time detection capabilities in application scenarios with scarce computing power conditions.

[0052] The experimental environment used is Ubuntu 22.04, the kernel environment is Linux 5.15.0-72-generic, the graphics card uses Nvidia RTX 2080Ti and Tesla V30 for the experiment, and the processor model is Intel (R) Xeon (R) CPU E5-2670v2 @ 2.50GHz. The experimental environment is Python 3.11, and the scientific computing library Pytorch environment is 1.12.0. The subsequent plan is to use domestic devices such as Loongson to carry out the model and try to optimize it by means such as pre-compilation. Focus on improving the model to enhance its detection ability. According to the task implementation plan, the detection rate and execution efficiency of the model are continuously improved through downsampling processing, edge detection assistance, integer quantization and test period tuning.

[0053] The present invention has been experimented on a private data set. As shown in Table 2, the present invention has a good detection effect, and the detection rate has reached more than 83% under different experimental environments.

[0054] Table 2 Monitor-Net detection results under different experimental environments

[0055] Hash Type Model Type Execution speed Detection rate i5-12500 CPU Monitor-Net 25FPS 85.3％ E5-2670 v2 CPU Monitor-Net 28FPS 83.2％ 2080Ti graphics card acceleration Monitor-Net 120FPS 85.6％ E5-2670 v2 CPU Monitor-Ne+ Quantization 56FPS 84.3％ E5-2670 v2 CPU Monitor-Ne+quantization+nncf 52FPS 85.1％

[0056] Although the embodiments and drawings of the present invention are disclosed for illustrative purposes, those skilled in the art will appreciate that various substitutions, changes and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the scope of the present invention is not limited to the contents disclosed in the embodiments and drawings.

Claims

1. A method for detecting targets in severe weather based on multi-scale feature extraction and model quantization, characterized in that: The steps of the method are: S1. Reduce the image size by bicubic linear interpolation, then introduce the horizontal Sobel operator, vertical Sobel operator and composite Sobel operator to perform differential estimation on the image edge and expand the edge information in the image. The weight matrix of the horizontal Sobel operator and the vertical Sobel operator is: S2, get the convolution kernel S x , S y Then, use G x =S x A and G y =S y A performs horizontal and vertical edge detection, and uses an approximate fusion method to fuse the horizontal and vertical edges. The fusion function used is: |G|=|G x |+|G y | S3, data reception, queuing and scheduling are realized through the routing branch detection algorithm, and the pre-processed images are allocated according to the priority order by using the resource scheduling module to schedule computing resources; in the corresponding computing resources, the multi-scale feature extraction module is used to extract key information from feature maps of different sizes to enhance the model detection accuracy; S4. According to the extracted image features, the free anchor prediction module is used to predict. When predicting, it does not rely on fixed-size anchors, but directly predicts the center and size of the detection target. After obtaining the image features, the segmentation result extraction module is used to extract the results to obtain the final segmentation anchor. The final segmentation anchor is in the form of (b x ,b y ,b h ,b w ,c,p), directly predict the position and size of the anchor point, and predict the confidence of each category.

Images