A multi-target detection method in open-air industrial scenes

Abstract

The present invention discloses a multi-target detection method in an open-air industrial scene, which comprises the following steps: obtaining a multi-target detection image in the open-air industrial scene; inputting the image to be detected into a multi-target detection model to obtain a target detection result of the image to be detected output by the multi-target detection model; wherein the multi-target detection model comprises a residual network, a feature pyramid, an attention module and a detection head; the detection head comprises a feature alignment module and a rotation feature detection module; a fog enhancement mechanism for model training is added at the model input end; the attention module is arranged between the feature extraction network and the detection head, wherein a spatial attention module is applied to a shallow feature layer with a smaller number of channels, and a combination structure of a spatial attention module and a pixel attention module is applied to a deep feature layer with a larger number of feature channels; KL divergence is set at the detection head as a consistency loss, and the KL divergence is calculated by a classification feature map and a feature regression map output by the feature alignment module and the rotation feature detection module.

Description

Technical Field

[0001] The present invention belongs to the technical field of visual neural networks, and in particular relates to a multi-target detection method in open-air industrial scenarios. Background Art

[0002] On-site supervision in the industrial sector primarily relies on human effort to detect, identify, and analyze target behavior. However, this reliance on manual labor not only consumes human resources but can also lead to erroneous judgments due to subjective factors. Compared to common object detection tasks in natural scene images, in the task of detecting vehicles in focal field scenes, different types of vehicles are located in close proximity. Using a rectangular frame results in a high degree of overlap between vehicles. Therefore, a rotating object detection algorithm is required to detect vehicles in the focal field.

[0003] In the rotation target detection task, S 2 The ANet (an article published on arxiv in 2020 by Jiaming Han et al., “Align Deep Features for Oriented Object Detection”) detection model was proposed to solve the misalignment between convolutional features and anchor boxes. The detection head of this model mainly consists of two modules, the feature alignment module (FAM) and the rotation detection module (ODM). The feature alignment module generates high-quality anchor boxes through the anchor box refinement network ARN. The rotation feature detection module contains a structural active rotation filter ARF that encodes direction information. Its main function is to generate rotation-invariant features and rotation-sensitive features. Finally, the two features are input into the regression subnetwork and the classification subnetwork to obtain the final prediction results.

[0004] Directly S 2 ANet still faces challenges when applied to rotating target detection in open-air industrial scenarios. The complex field of view and high dust levels result in low contrast in the captured images, making the vehicle targets less visible. Furthermore, the size of the operating vehicle targets is relatively small relative to the overall field of view. Therefore, it is necessary to design an object detection algorithm that is robust to low contrast and incorporates an attention mechanism to improve detection performance for small objects. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the present invention provides a multi-target detection method for open-air industrial scenarios. The main purpose of the present invention is to solve the problem of construction vehicle target positioning in focal field scenarios, namely the problem of rotating target detection in specific scenarios, using a deep learning method.

[0006] The present invention achieves this object through the following technical solutions:

[0007] A multi-target detection method in an open-air industrial scene includes the following steps:

[0008] Acquire multi-target detection images in open-air industrial scenes;

[0009] Inputting the image to be detected into a multi-target detection model to obtain a target detection result of the image to be detected output by the multi-target detection model;

[0010] The multi-target detection model includes a residual network, a feature pyramid, an attention module and a detection head; the detection head includes a feature alignment module and a rotation feature detection module; a fog enhancement mechanism for model training is added to the model input end; the attention module is between the feature layer and the detection head, wherein the spatial attention module is applied to the shallow feature layer with a smaller number of channels, and the combined structure of the spatial attention module and the pixel attention module is applied to the deep feature layer with a larger number of feature channels; the KL divergence is set as the consistency loss at the detection head, and the KL divergence is calculated by the classification feature map and feature regression map output by the feature alignment module and the rotation feature detection module.

[0011] Furthermore, the multi-target detection model is obtained by the following steps:

[0012] S1: Construct an open-air industrial scene target detection dataset, preprocess the data and divide it into training set, validation set and test set;

[0013] S2: Generate an image copy from the image and perform fog enhancement on the copy;

[0014] S3: Splice the enhanced image with the original image in the batchsize dimension, and perform conventional data augmentation on the spliced ​​result;

[0015] S4: After inputting the feature extraction network, the feature map passes through the attention module and the detection head to obtain the classification feature map and the regression feature map;

[0016] S5: Extract the classification feature map and regression feature map of the original image and the fog-enhanced image in the batchsize dimension, and calculate the KL divergence of the classification and regression feature maps of the original image and the classification and regression feature maps of the fog-enhanced image;

[0017] S6: The calculated KL divergence is used as the consistency loss and added to the inherent loss of the model. The gradient is back-propagated to update the parameters, optimize the position of the detection box, and guide the feature extraction network to learn fine-grained features.

[0018] S7: Iterate steps S2-S6 through the training set until the model converges and the training phase ends;

[0019] S8: Debug and test the model using the validation set and test set.

[0020] Furthermore, the fog enhancement method is to calculate an enhancement mask for the original image based on the calculation script provided by YOLOv8, and then add the mask to the original image.

[0021] Furthermore, the brightness adjustment parameter of the enhancement mask setting is calculated to be 0.4, the pixel threshold is 0.025, the center position of the image is set as the fog center, and the enhancement mask value of the image from the fog center outward is calculated.

[0022] Furthermore, after the feature extraction network, four feature maps of different scales are obtained. The feature sizes from shallow to deep are: F1∈R 64×96×256 , F2∈R 128×48×128 、F3∈R 256×24×64 、F4∈R 512×12×32 .

[0023] Furthermore, the feature maps are classified according to the number of channels, and F1 and F2 are respectively passed through the spatial attention module as shallow feature maps of the network, and F3 and F4 are respectively passed through the combined attention module as deep feature maps of the network. The combined attention module includes spatial attention and pixel attention.

[0024] Furthermore, the combined attention module is composed of spatial attention and pixel attention connected sequentially.

[0025] Furthermore, the fog enhancement consistency loss in steps S5 and S6 uses KL divergence. Due to the magnitude difference between the classification consistency loss and the regression consistency loss, a balance coefficient is set for the consistency loss of the regression features to balance the losses of each part and improve the model's ability to regress the bounding box.

[0026] Furthermore, the balance coefficient is set to 20.

[0027] Furthermore, the fog enhancement is only performed on the image during network training and does not participate in the model inference process during the verification and testing phases.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] 1. The present invention proposes a new fog enhancement mechanism in the network input stage, sets the KL divergence as the consistency loss at the detection head position, and effectively guides the feature extraction network to capture features in the input image by minimizing the difference in the distribution of the inference feature maps of the original image and the enhanced image, thereby improving the robustness of the model detection.

[0030] 2. The present invention introduces a combined attention module at the network structure level, which can enable the model to assign more weight to the foreground area and improve the model's detection effect on foreground targets.

[0031] 3. The fog enhancement consistency loss introduced in this invention not only optimizes the feature extraction network parameters, but also further optimizes the network parameters of the added combined attention module.

[0032] 4. The fog enhancement consistency designed by the present invention improves the indicators without increasing the computational complexity of model reasoning, thereby improving the practical applicability of the model.

[0033] Aiming at the high requirements for foreground features in rotating target detection tasks, the present invention designs a multi-target detection method in open-air industrial scenarios. Fog enhancement is added during the training process, and the difference in the inference feature map between the original image and the fog-enhanced map is minimized, thereby improving the robustness of the model to disturbances and further improving the detection performance. The present invention adds a combined attention module to the model structure, uses different attention modules for feature layers of different depths, and assigns weight parameters to the features extracted by the feature extraction network as much as possible, thereby improving the model's attention to the foreground area. After adding the fog enhancement consistency loss, the original network parameters of the model and the added attention module parameters can be further optimized to improve the accuracy of the model in detecting foreground targets. Among them, fog enhancement does not increase the testing cost, and the fusion with the combined attention module effectively improves the detection model's prediction effect on foreground targets and enhances the model's robustness to interference such as bad weather or dust. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are merely exemplary, and those skilled in the art can, without inventive effort, derive other implementation drawings based on the provided drawings.

[0035] Figure 1 This is a flowchart of a multi-target detection model training process according to an embodiment of the present invention;

[0036] Figure 2 is a comparative diagram of fog enhancement generation in an embodiment of the present invention;

[0037] Figure 3 Schematic diagram of the multi-target detection model structure in an embodiment of the present invention;

[0038] Figure 4 This is a comparison chart of the effects of the multi-target detection model and the benchmark detection model in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art. It should be noted that, unless there is a conflict, the embodiments of the present invention and the features within the embodiments may be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0040] The present invention discloses a multi-target detection method in an open-air industrial scene, comprising the following contents:

[0041] Acquire multi-target detection images in open-air industrial scenes;

[0042] The image to be detected is input into the multi-target detection model to obtain the target detection result of the image to be detected output by the multi-target detection model.

[0043] Multi-target detection model based on S 2 The ANet detection network model mainly consists of a residual network (ResNet), a feature pyramid (FPN), an attention module, and a detection head; the detection head includes a feature alignment module and a rotation feature detection module. A fog enhancement mechanism is added to the model input, and the KL divergence is set as the consistency loss at the detection head. The KL divergence is calculated from the classification feature map and feature regression map output by the feature alignment module and the rotation feature detection module for the original image and the enhanced image. The attention module is located between the feature layer and the detection head. The spatial attention module is applied to shallow feature layers with a small number of channels, and a combination of the spatial attention module and the pixel attention module is applied to deep feature layers with a large number of feature channels.

[0044] The overall inference process of the multi-object detection model is as follows: the scene image and its fog-enhanced image are fed into the classic deep network ResNet and feature pyramid FPN architecture as input to generate feature maps at four scales. These feature maps are then partitioned by depth, with the two with the smallest number of channels being fed into the spatial attention module as the shallow feature layer, and the two with the largest number of channels being fed into the combined spatial attention and pixel attention architecture as the deep feature layer. The attention module then feeds the feature maps at all four scales into the detection head, extracting refined features and obtaining anchor box biases to generate rotated predictions. The feature alignment and rotation feature extraction modules in the detection head calculate a consistency loss. The KL divergence is calculated for the feature extraction results of the original and enhanced image feature maps and incorporated into the network's intrinsic loss for backpropagation and parameter optimization. Finally, all feature maps are fed into the classification and regression subnetworks to obtain object classification and regression predictions.

[0045] The process of obtaining the multi-target detection model is as follows Figure 1 、 Figure 3 The specific steps are as follows:

[0046] S1: Construct an open-air industrial scene target detection dataset, preprocess the data and divide it into training set, validation set and test set.

[0047] The images in the dataset are all taken from real open-air industrial scenes. They were captured using cameras placed at equal distances within the scene and then processed using a stitching algorithm and distortion correction. Furthermore, due to the short sampling intervals, the small scope of the engineering work area on a single day, the fixed number of operating vehicles, and the minimal variations in vehicle operation patterns, the images collected on a single day are highly similar. Therefore, when partitioning the dataset, all data for a single day was placed in only one dataset among the training, validation, and test sets. This means that data from the same day will not be included in different datasets.

[0048] S2: Generate a copy of the original image and perform fog enhancement on the copy.

[0049] Specifically: save a copy of the original image and perform fog enhancement on the copy image. The main principle of the fog enhancement method is to calculate the enhancement mask for the original image based on the calculation script provided by YOLOv8, and then add the mask to the original image. Preferably, the brightness adjustment parameter for calculating the enhancement mask setting is 0.4, the pixel threshold is 0.025, the center position of the image is set to the fog center and the enhancement mask value of the image from the fog center to the outside is calculated. The simulated fog concentration gradually decreases from the fog center to the edge of the image. The comparison before and after fog enhancement is as follows: Figure 2 shown.

[0050] The fog enhancement strategy closely simulates situations where foreground objects are blurred or the foreground object has low contrast with the background, such as those caused by inclement weather or factory dust. By simulating real-world scenarios on the input image, the model is guided to produce the same fit between the enhanced image and the original image, thereby improving the robustness of the model's predictions.

[0051] S3: The fog-enhanced image is spliced ​​with the original image in the batchsize dimension, and the spliced ​​result is subjected to conventional data enhancement.

[0052] In the stitching result, the copy image is enhanced with fog and the inherent data of the model, while the original image is only enhanced with the inherent data of the model.

[0053] S4: After the enhanced result of S3 is input into the feature extraction network, the feature map passes through the attention module and the detection head to obtain the classification feature map and the regression feature map.

[0054] Specifically: After the feature extraction network, four feature maps of different scales are obtained. The feature sizes from shallow to deep are: F1∈R 64×96×256 , F2∈R 128×48×128 、F3∈R 256×24×64 、F4∈R 512×12×32 Classification is performed based on the number of channels in the feature map: F1 and F2 are used as shallow feature maps of the network and passed through the spatial attention module, while F3 and F4 are used as deep feature maps of the network and passed through the combined attention module. The combined attention module consists of a spatial attention mechanism and a pixel attention mechanism. The output of the attention module is input into the detection head of the model to obtain classification feature maps and regression feature maps.

[0055] The attention module uses a spatial attention mechanism for shallow feature layers and a combination of spatial and pixel attention mechanisms for deeper feature layers. The spatial attention mechanism enables the model to focus on regions of the feature map that are more beneficial to the prediction results. The pixel attention mechanism primarily calculates an enhanced mask for the locations of foreground objects in foggy images, giving greater attention to pixels within the feature map. Adding a pixel attention mechanism to the detection network assigns weights to pixel regions in the feature map, allowing the model to allocate more weight parameters to foreground pixels. This is a fine-grained weight allocation module. The combined attention module consists of a sequential splicing of the spatial and pixel attention modules. Specifically, the feature map passes through the two attention modules sequentially, giving greater attention to the target regions. Because features at different depths have different densities and representation levels, different attention modules are used to fully exploit these features, allowing the model to further allocate position weights and improve the model's focus on foreground regions.

[0056] S5: Take out the classification feature map and regression feature map of the original image and the fog-enhanced image in the batchsize dimension respectively, and calculate the KL divergence of the classification and regression feature maps of the original image and the classification and regression feature maps of the fog-enhanced image.

[0057] S6: Use KL divergence as the consistency loss and splice it into the model's inherent loss. Backpropagate the gradient, update the parameters, optimize the detection box position, and guide the feature extraction network to learn fine-grained features.

[0058] Before splicing, there is an imbalance in magnitude between the regression part of the consistency loss and the classification part of the consistency loss and the inherent loss in the model. The imbalance in magnitude will make the model focus more on larger loss values, while the consistency regression loss with smaller loss value will have a poor effect on parameter optimization. Therefore, a balance coefficient is set in the regression part of the consistency loss so that the consistency regression loss can be slightly higher than the consistency classification loss and reach a balance in magnitude with the inherent loss of the model. Specifically, the consistency regression loss is multiplied by the balance coefficient, which is set to 20 in the present invention. After splicing, the gradient is backpropagated, the parameters are updated, the detection box position is optimized, the robustness of the detection network to interference from severe weather is improved, and the model detection performance is thereby improved.

[0059] The KL divergence is used for the fog enhancement consistency loss in steps S5 and S6. Due to the magnitude difference between the classification consistency loss and the regression consistency loss, a balance coefficient is set for the consistency loss of the regression features to balance the losses of each part and improve the model's ability to regress the bounding box.

[0060] S7: Iterate steps S2-S6 through the training set until the model converges and the training phase ends.

[0061] S8: Debug and test the model using the validation set and test set.

[0062] During the testing phase, fog enhancement consistency loss is used as a data augmentation method and does not participate in the model inference process. The main function of fog enhancement is to generate perturbed image results, which are used to calculate the difference between the results and the original image results. Minimizing this difference value can guide the learning of the feature extraction network and enable the feature extraction network to have a certain degree of perturbation robustness. After model training, the testing phase does not involve the backpropagation process, so the fog enhancement mechanism is no longer required, reducing the amount of computation required. At the same time, because the testing phase mainly targets the original image data, the indicators can better reflect the detection improvement effect on the original scene image. This not only improves the model's robustness to low clarity and contrast caused by bad weather or dust, but also does not increase the computational workload, time, and hardware cost of model verification and testing.

[0063] During the model training process, a combination of pixel attention and spatial attention modules is used to guide the feature extraction network to focus on learning the foreground target features of the input image, thereby improving the model's ability to detect blurred and small targets.

[0064] The detection effect of the multi-target detection model in this embodiment on the test set is visualized and compared with the benchmark network detection results. The detection effect is as follows: Figure 4 shown. Figure 4 (a) and Figure 4 (b) is S 2Detection results of the ANet benchmark network model. The green dotted boxes mark targets that the model missed due to low resolution or contrast, to assist observation. Figure 4 (c) and Figure 4 (d) is the detection result of the multi-target detection model of the present invention. The blue dotted box marks the target that the multi-target detection model missed for the benchmark network.

[0065] The present invention has been described in detail above through the embodiments, but the contents described are only exemplary embodiments of the present invention and cannot be considered to limit the scope of implementation of the present invention. The scope of protection of the present invention is defined by the claims. Any use of the technical solution described in the present invention, or any person skilled in the art who, inspired by the technical solution of the present invention, designs a similar technical solution within the essence and scope of protection of the present invention to achieve the above-mentioned technical effects, or any equivalent changes and improvements made to the scope of application, shall still fall within the scope of protection covered by the patent of the present invention.

Claims

1. A multi-target detection method in an open-air industrial scene, characterized in that: The following steps are involved: Acquire multi-target detection images in open-air industrial scenes; Inputting the multi-target detection image into a multi-target detection model to obtain a target detection result of the multi-target detection image output by the multi-target detection model; The multi-target detection model includes a residual network, a feature pyramid, an attention module, and a detection head; the detection head includes a feature alignment module and a rotation feature detection module; a fog enhancement mechanism for model training is added to the model input; the attention module is between the feature layer and the detection head, wherein a spatial attention module is applied to the shallow feature layer, and a combination of a spatial attention module and a pixel attention module is applied to the deep feature layer; the KL divergence is set as the consistency loss at the detection head, and the KL divergence is calculated based on the feature extraction results of the original image and the enhanced image feature map in the classification feature map and feature regression map output by the feature alignment module and the rotation feature detection module respectively; The multi-target detection model is obtained by the following steps: S1: Construct an open-air industrial scene target detection dataset, preprocess the data and divide it into training set, validation set and test set; S2: Generate an image copy from the image and perform fog enhancement on the copy; S3: The enhanced image is spliced ​​with the original image in the batchsize dimension, and the spliced ​​result is subjected to conventional data augmentation. In the spliced ​​result, the duplicate image is subjected to fog enhancement and model-specific data augmentation, while the original image is subjected to model-specific data augmentation only. S4: After the data enhancement result obtained in S3 is input into the feature extraction network, the feature map passes through the attention module and the detection head to obtain the classification feature map and the regression feature map; S5: The feature alignment module and the rotation feature detection module extract the classification feature map and regression feature map of the original image and the fog-enhanced image in the batchsize dimension respectively, and calculate the KL divergence for the feature extraction results of the original image and the fog-enhanced image in the classification feature map and the original image and the fog-enhanced image in the regression feature map respectively; S6: The calculated KL divergence is used as the consistency loss and added to the inherent loss of the model. The gradient is back-propagated to update the parameters, optimize the position of the detection box, and guide the feature extraction network to learn fine-grained features. S7: Iterate steps S2-S6 through the training set until the model converges and the training phase ends; S8: Debug and test the model using the validation set and test set; The fog enhancement method is to calculate an enhancement mask for the original image based on the calculation script provided by YOLOv8, and then add the mask to the original image.

2. The multi-target detection method according to claim 1, wherein: The brightness adjustment parameter for calculating the enhancement mask setting is 0.4, the pixel threshold is 0.025, the center position of the image is set as the fog center, and the enhancement mask value of the image from the fog center outward is calculated.

3. The multi-target detection method according to claim 1, wherein: After the feature extraction network, four feature maps of different scales are obtained. The feature sizes from shallow to deep are: F1∈ R 64×96×256 、F2∈R 128×48×128 、F3∈R 256×24×64 、F4∈R 512×12×32 。 4. The multi-target detection method according to claim 3, wherein: Classify according to the number of channels of the feature map, and pass F1 and F2 as shallow feature maps of the network through the spatial attention module respectively, and pass F3 and F4 as deep feature maps of the network through the combined attention module respectively. The combined attention module includes spatial attention and pixel attention.

5. The multi-target detection method according to claim 4, characterized in that: The combined attention module consists of spatial attention and pixel attention connected sequentially.

6. The multi-target detection method according to claim 1, wherein: The KL divergence is used for the fog enhancement consistency loss in steps S5 and S6. Due to the magnitude difference between the classification consistency loss and the regression consistency loss, a balance coefficient is set for the consistency loss of the regression features to balance the losses of each part and improve the model's ability to regress the bounding box.

7. The multi-target detection method according to claim 6, characterized in that: The balance coefficient is set to 20.

8. The multi-target detection method according to claim 1, wherein: The fog enhancement is only used to enhance the image during network training and does not participate in the model inference process during the verification and testing phases.

Images