Sand and dust weather target detection method, system, equipment and medium

Abstract

The invention discloses a sand and dust weather target detection method, system and device and a medium. The method comprises the following steps: constructing a road scene image data set in the sand and dust weather; constructing an image restoration module based on an AOD-Net image defogging network, and improving the image restoration module based on a dust scattering model; constructing a target detection module based on a YOLOv8 model, adding a small target detection layer into the target detection module, and introducing a loss function InnerMPDIOU; and a sand and dust weather target detection model is constructed based on the improved image restoration module and the improved target detection module, the sand and dust weather target detection model is trained based on the road scene image data set, and image target detection under the wind and sand weather is executed according to the trained sand and dust weather target detection model. According to the technical scheme, images can be processed in real time and accurate target detection can be carried out in sand and dust weather, and the actual application requirement of an automatic driving system is met.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection, and in particular relates to a method, system, equipment and medium for detecting targets in dusty weather. Background Art

[0002] With the development of autonomous driving technology, the ability of vehicles to drive in various weather conditions has become crucial. In autonomous driving systems, input data from sensors such as cameras, radars, and lidars need to be processed with high precision to ensure that the vehicle can correctly perceive the surrounding environment and make correct driving decisions. However, severe weather conditions, such as sandstorms, pose a huge challenge to the working performance of these sensors.

[0003] The particulate matter in the sandstorm is large, which will seriously interfere with the image quality collected by the camera, making the image blurry and affecting the subsequent image processing and target detection tasks. In this case, the autonomous driving system may not be able to accurately identify key targets such as road signs, pedestrians, and vehicles, thereby increasing the risk of accidents. The existing road target detection algorithms for complex environmental conditions of sandstorms have problems such as low detection accuracy, poor robustness, and limitations of data sets. It is urgent to design an effective dust removal image processing method to improve the reliability and safety of the autonomous driving system in sandstorm weather. Summary of the invention

[0004] The purpose of the present invention is to provide a method, system, device and medium for detecting targets in sandstorm weather to solve the problems existing in the above-mentioned prior art.

[0005] To achieve the above object, the present invention provides a method for detecting targets in sandstorm weather, comprising:

[0006] Construct a dataset of road scene images in windy and sandy weather;

[0007] An image restoration module is constructed based on the AOD-Net image dehazing network, and the image restoration module is improved based on a dust scattering model to obtain an improved image restoration module;

[0008] Building a target detection module based on the YOLOv8 model, adding a small target detection layer to the target detection module and introducing a loss function InnerMPDIoU to obtain an improved target detection module;

[0009] A dust weather target detection model is constructed based on the improved image restoration module and the improved target detection module, the dust weather target detection model is trained based on the road scene image dataset, and image target detection in windy and sandy weather is performed according to the trained dust weather target detection model.

[0010] Optionally, the process of constructing the road scene image dataset includes:

[0011] Obtain open source synthetic scene sand data and road images taken in sandy weather;

[0012] The open-source synthetic scene wind and sand data and the road image are preprocessed based on data enhancement technology and manual labeling method, and a road scene image dataset is constructed based on the preprocessed image data.

[0013] Optionally, the image restoration module is improved based on a dust scattering model, and the specific calculation formula includes:

[0014] I(x)=(J(x)-A+1)t(x)+A

[0015]

[0016] J(x)=K(x)I(x)-K(x)+b

[0017]

[0018] Where I(x) is a haze-free image, A is the atmospheric light value, t(x) is a set of transfer functions, K(x) is the parameter obtained by integrating the atmospheric light value and the transfer function, J(x) is a dust-free image, and b is a constant bias with a default value of 1.

[0019] Optionally, a small target detection layer is added to the target detection module and a loss function InnerMPDIoU is introduced, specifically including:

[0020] A small target detection layer is added on the basis of the three original detection layers of the YOLOv8 model; the Inner-IoU loss function is integrated into the MPDIoU loss function to obtain the InnerMPDIoU loss function, and the InnerMPDIoU loss function is used as the loss function of the target detection module to obtain an improved target detection module.

[0021] Optionally, the process of training the sandstorm weather target detection model includes:

[0022] Dividing the road scene image dataset into a training set, a test set, and a validation set based on a preset division rule;

[0023] The images in the training set are input into the improved image restoration module for dust removal, and the images after dust removal are input into the improved target detection module for target detection, and the classification confidence and position coordinates of the target are output; the network parameters of the two modules are simultaneously updated through the back propagation algorithm; the dust weather target detection model after parameter update is evaluated based on the validation set, and the dust weather target detection model is jointly optimized based on the evaluation results, and the iterative training and optimization process is repeated until the preset number of iterations is reached, and the trained dust weather target detection model is output.

[0024] A sandstorm weather target detection system, comprising:

[0025] Data acquisition module, used to construct a road scene image dataset under windy and sandy weather;

[0026] A model construction training module is used to construct an image restoration module based on the AOD-Net image defogging network, and improve the image restoration module based on the dust scattering model to obtain an improved image restoration module; a target detection module is constructed based on the YOLOv8 model, a small target detection layer is added to the target detection module, and a loss function InnerMPDIoU is introduced to obtain an improved target detection module; a sandstorm weather target detection model is constructed based on the improved image restoration module and the improved target detection module, and the sandstorm weather target detection model is trained based on the road scene image dataset;

[0027] The target detection module is used to perform image target detection in sandstorm weather based on the trained sandstorm weather target detection model.

[0028] An electronic device comprises a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the method for detecting targets in sandstorm weather.

[0029] A computer-readable storage medium stores a computer program, which implements the method for detecting targets in sandstorm weather when executed by a processor.

[0030] The technical effects of the present invention are:

[0031] Based on the dust scattering model, the present invention improves AOD-Net, which can more effectively remove large particles in sandstorm images and improve image clarity; the present invention optimizes the YOLOv8 model to achieve higher target detection accuracy and speed on images after dust removal, thereby enhancing the reliability of the autonomous driving system in dusty weather. The method proposed in the present invention jointly optimizes the dust removal module and the target detection module to form a complete system, which can process images in real time and perform accurate target detection in dusty weather, meeting the practical application requirements of the autonomous driving system. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0033] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0034] Figure 1 is a detection flow chart in an embodiment of the present invention;

[0035] Figure 2 is a structural diagram of a target detection model in an embodiment of the present invention;

[0036] Figure 3 This is a structural diagram of the improved YOLOv8 model in an embodiment of the present invention. DETAILED DESCRIPTION

[0037] Various exemplary embodiments of the present invention will now be described in detail. This detailed description should not be considered as limiting the present invention, but should be understood as a more detailed description of certain aspects, features, and embodiments of the present invention.

[0038] It should be understood that the terms described in the present invention are only for describing special embodiments and are not intended to limit the present invention. In addition, for the numerical range in the present invention, it should be understood that each intermediate value between the upper and lower limits of the scope is also specifically disclosed. Each smaller range between the intermediate value in any stated value or stated range and any other stated value or intermediate value in the described range is also included in the present invention. The upper and lower limits of these smaller ranges can be independently included or excluded in the scope.

[0039] Unless otherwise indicated, all technical and scientific terms used herein have the same meanings as those of ordinary skill in the art generally understood by the present invention. Although the present invention describes only preferred methods, any method similar or equivalent to that described herein may also be used in the implementation or testing of the present invention. All documents mentioned in this specification are incorporated by reference to disclose and describe the methods related to the documents. In the event of a conflict with any incorporated document, the content of this specification shall prevail.

[0040] It will be apparent to those skilled in the art that various modifications and variations may be made to the specific embodiments of the present invention description without departing from the scope or spirit of the present invention. Other embodiments derived from the present invention description will be apparent to those skilled in the art. The present application description and examples are exemplary only.

[0041] The words “include,” “including,” “have,” “contain,” etc. used in this article are open-ended terms, meaning including but not limited to.

[0042] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0043] Embodiment 1

[0044] like Figure 1 - Figure 3 As shown, a dust weather target detection method is provided in this embodiment, including: constructing a road scene image dataset under windy and sandy weather; constructing an image restoration module based on an AOD-Net image defogging network, and improving the image restoration module based on a dust scattering model to obtain an improved image restoration module; constructing a target detection module based on a YOLOv8 model, and adding a small target detection layer to the target detection module; introducing a new loss function InnerMPDIoU to obtain an improved target detection module; constructing a dust weather target detection model based on the improved image restoration module and the improved target detection module, training the dust weather target detection model based on the road scene image dataset, and performing image target detection under windy and sandy weather according to the trained dust weather target detection model.

[0045] Aiming at the problems of low detection accuracy and poor robustness in road target detection under complex environmental conditions of sandstorm, an improved YOLOv8 target detection model for sandstorm weather is proposed. The model consists of an image restoration module and a detection module, which are jointly optimized during the training process. To ensure the accuracy and real-time performance of the model in complex weather scenes, in the dust removal module, an improved AOD-Net network is used: Dedust-AOD-Net performs image restoration processing on the input image to reduce the interference of sand and dust on the target to be detected in the image; in the detection module, an improved YOLOv8 model is used to output the classification confidence and position coordinates of the target. In order to improve the detection performance of the network, this embodiment adds an additional small target detection layer on the basis of YOLOv8, retains more shallow detail information of the feature map, and improves the model's detection ability for small targets. In order to improve the adaptability of the improved model to sand and dust image detection, a new InnerMPDIoU loss function is used, and finally a real-world sandstorm dataset is built for joint training, detection and optimization of sand and dust removal and detection models.

[0046] This embodiment improves AOD-Net based on the dust scattering model, which can more effectively remove large particles in sandstorm images and improve image clarity. This embodiment optimizes the YOLOv8 model so that it can achieve higher target detection accuracy and speed on images after dust removal, thereby enhancing the reliability of the autonomous driving system in dusty weather. This embodiment collects and produces a high-quality image dataset under various dusty weather conditions, covering different lighting and visibility conditions, providing rich samples for model training and evaluation, and significantly improving the performance of the model in real environments. The method proposed in this embodiment organically combines the dust removal module and the target detection module to form a complete system, which can process images in real time and perform accurate target detection in dusty weather, meeting the actual application needs of the autonomous driving system.

[0047] The specific implementation process of this embodiment includes:

[0048] Dust removal module design: AOD-Net is an end-to-end dehazing model that integrates the two parameters of the atmospheric light value A and the transfer function t(x) in the atmospheric scattering model into a parameter K(x). A multi-scale feature fusion network consisting of a small number of convolutional layers is used to estimate K(x). Then, the obtained K(x) is substituted into formula (1) to obtain the corresponding haze-free image. K(x) is shown in formula (4).

[0049] I(x)=J(x)t(x)+A(1-t(x)) (1)

[0050] t(x)=e -βd(x) (2)

[0051] J(x)=K(x)I(x)-K(x)+b (3)

[0052]

[0053] Where I(x) is a haze-free image, A is the atmospheric light value, t(x) is a set of transfer functions, K(x) is the parameter obtained by integrating the atmospheric light value and the transfer function, J(x) is a dust-free image, and b is a constant bias with a default value of 1.

[0054] Based on the dust scattering model, the dust removal of sandstorm images is achieved by improving the existing AOD-Net structure. The mathematical expression of the dust scattering model is as follows:

[0055] I(x)=(J(x)-A+1)t(x)+A (5)

[0056]

[0057] J(x)=K(x)I(x)-K(x)+b (7)

[0059]

[0060] The improved AODNet consists of two parts: the K estiamting module and the image generation module. The K estimating module consists of a small convolutional neural network, which contains five convolutional layers. The output feature maps of different convolutional layers are combined through feature fusion operations. The K estimating module adaptively learns the parameter K(x) from the input image; the image generation module substitutes the K(x) predicted by the K estimating module into formula (7) to obtain the dust-free image corresponding to the input image. At the same time, the ReLU activation function in the original network structure is changed to the Swish activation function, and the improved AOD-Net is named Dedust-AOD-NET.

[0061] In this embodiment, AOD-Net is redesigned according to the dust scattering model, and the network originally used for removing haze is transformed into a network suitable for removing dust. This redesign enables the network to process images in dusty environments more effectively, significantly improving image clarity and quality.

[0062] Dataset creation: In order to improve the performance of the model in real environments, we collected and created a high-quality image dataset containing a variety of dust weather conditions. This dataset covers conditions of different lighting, visibility, and dust concentrations, providing rich samples for model training and evaluation.

[0063] The dataset production process includes:

[0064] 1. Construction of a road target dataset under sandstorm weather: Since the current open source road target dataset under sandstorm weather has a small number of samples, the project team will integrate the open source synthetic scene sandstorm dataset with the self-built real-world sandstorm dataset to enrich the number of dataset samples.

[0065] (1) Use data enhancement technology to expand the open source dataset. Extract scene images under windy and sandy weather from the open source road target dataset, use data enhancement technology to process the extracted images to expand the sample size of the dataset, and unify the image annotation format.

[0066] (2) Collect images of road scenes in windy and dusty weather. The project team will collect images of road scenes in dusty weather. These images come from weather forecast websites, surveillance cameras, and road traffic scene pictures taken by the team. Use data enhancement techniques such as random rotation, translation, and scaling to process the images and increase the richness of the dataset. Ensure that the dataset contains images from different times and locations to increase the diversity and representativeness of the dataset.

[0067] (3) Label the collected images. Use professional labeling tools such as LabelImg to manually label the collected images. The labeling content includes the location and category information of targets such as roads, vehicles, and pedestrians. Figure 3 As shown, attention should be paid to the accuracy and consistency of marking.

[0068] (4) Divide the dataset. Combine the open source dataset with the self-collected image data to create a road target dataset under windy and sandy weather, and divide the dataset into training set, validation set, and test set. Usually, the ratio of 7:2:1 is used for model training, parameter adjustment, and evaluation. When dividing the dataset, ensure that the number of samples of each category in the training set, validation set, and test set is relatively balanced.

[0069] This embodiment expands the training samples by constructing a real-world dataset containing a variety of sandstorm environments. This dataset includes sandstorm images of various intensities and environments, providing rich training data for the model, thereby significantly improving the training effect and generalization ability of the model.

[0070] YOLO target detection network improvements:

[0071] (1) Improvement of loss function: IoU (Intersection over Union) is an important component used to measure the overlap between the predicted box and the true box. IoU refers to the intersection over union ratio between the predicted box B and the true box A, which can more intuitively reflect the positioning accuracy of the bounding box. The calculation formula is shown in formula (9).

[0072]

[0073] However, the existing IoU loss function has insufficient generalization ability on different datasets and detectors. Using Inner-IoU Loss, we select auxiliary bounding boxes of different sizes by controlling the scale factor to adapt to different datasets and detectors. In order to further improve the generalization ability of the loss function for different datasets, Inner-IoU is integrated into MPDIoU to form a new loss function InnerMPDIoU. MPDIoU includes all relevant factors considered in the existing loss function, namely overlapping or non-overlapping areas, center point distance, width and height deviation, and simplifies the calculation process.

[0074]

[0075]

[0076] inter=(min(b r gt ,b r )-max(b l gt ,b l ))*(min(b b gt ,b b )-max(b t gt ,b t )) (14)

[0077] union=(w gt *h gt )*(ratio) 2 +(w*h)*(ratio) 2 -inter (15)

[0078]

[0079] d1 2 =(x1 prd -x1 gt ) 2 +(y1 prd -y1 gt ) 2 (17)

[0080] d2 2 =(x2 prd -x2 gt ) 2 +(y2 prd -y2 gt ) 2 (18)

[0081]

[0082] L MDPIoU =1-MDPIoU (20)

[0083] L Inner-MDPIou =L MDPIoU +IoU-IoU inner (twenty one)

[0084] In (10)-(21), the real box and the predicted box are represented by B gt and B. The center point of the real box and the inside of the real box is represented by (x c gt ,y c gt ) indicates that (x c ,y c ) represents the prediction box and the center point of the prediction box. gt ,x1 prd Indicates the coordinates of the upper left corner of the real box and the predicted box, x2 gt ,x2 prd Represents the coordinates of the lower right corner of the real box and the predicted box, W and H represent the width and height of the input image, and the width and height of the real box are represented by w gt and h gt The width and height of the predicted box are represented by w and h. The variable "ratio" corresponds to the scaling factor, which usually takes a value in the range [0.5, 1.5].

[0085] This embodiment changes the loss function to InnerMDPIoU to make it more suitable for the self-built sandstorm dataset. The InnerMDPIoU loss function can better capture the shape and position of the target and improve the detection accuracy of the model in complex environments.

[0086] (2) Adding a small target detection layer: The YOLOV8 backbone network completes the construction of feature maps of different scales through a series of downsampling. As the downsampling operation gradually deepens, the detail information of the feature map gradually weakens, while the semantic information gradually increases. Shallow feature maps have rich texture and edge features, so shallow networks are more suitable for extracting detail information of small-scale targets. In order to solve the problem of poor small target detection due to oversampling of the feature extraction network, this paper adds an additional small target detection layer on the basis of the three detection layers of YOLOv8, and forms a new detection head with the other three detection heads. On the basis of the original network, a 60×160 scale small target detection layer is added (including supplementary fusion feature layers and the introduction of additional detection heads) to enable accurate identification of distant targets. First, the 80×80 scale feature layer and the upsampled feature layer of the PANet structure are stacked upwards, and after the C2f module, Conv convolution and upsampling processing, a deep semantic feature layer containing the feature information of small targets is obtained; then the feature layer is stacked with the shallow position feature layer of the fourth layer of the backbone network to improve and supplement the expression ability of the 160×160 scale fusion feature layer for the semantic features and position information of small targets; finally, the fusion feature layer with the number of channels adjusted by the C2f module is sent to the additional detection head for decoding detection. The supplementation of the fusion feature layer can deepen the bidirectional transmission path of the Neck feature fusion layer, so that the feature information of distant targets can continue to be transmitted to other scale feature layers along the downsampling path through the PANet structure, thereby strengthening the feature fusion ability of the network and improving the detection accuracy of small-sized targets; the introduction of the additional detection head can expand the detection range of road targets. The improvement of detection accuracy and range enables the detection network to identify distant targets earlier.

[0087] This embodiment significantly improves the detection capability of the YOLOv8 model for small targets by adding an additional small target detection layer. This improvement is particularly important in sandstorm environments, because in such environments, small targets are more difficult to be accurately identified by traditional models. The improved model is named Dudest-YOLOv8. The technical process of this embodiment is as follows: Figure 1 The overall structure of the model is shown in Figure 2 As shown, the improved YOLOv8 network structure is as follows Figure 3 shown.

[0088] A sandstorm weather target detection system, comprising:

[0089] Data acquisition module, used to construct a road scene image dataset under windy and sandy weather;

[0090] A model construction training module is used to construct an image restoration module based on the AOD-Net image defogging network, and improve the image restoration module based on the dust scattering model to obtain an improved image restoration module; a target detection module is constructed based on the YOLOv8 model, a small target detection layer is added to the target detection module, and a new loss function InnerMPDIoU is introduced to obtain an improved target detection module; a sandstorm weather target detection model is constructed based on the improved image restoration module and the improved target detection module, and the sandstorm weather target detection model is trained based on the road scene image dataset;

[0091] The target detection module is used to perform image target detection in sandstorm weather based on the trained sandstorm weather target detection model.

[0092] An electronic device comprises a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the method for detecting targets in sandstorm weather.

[0093] A computer-readable storage medium stores a computer program, which implements the method for detecting targets in sandstorm weather when executed by a processor.

[0094] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for detecting targets in sandstorm weather, characterized in that: include: Construct a dataset of road scene images in windy and sandy weather; An image restoration module is constructed based on the AOD-Net image dehazing network, and the image restoration module is improved based on a dust scattering model to obtain an improved image restoration module; Building a target detection module based on the YOLOv8 model, adding a small target detection layer to the target detection module and introducing a loss function InnerMPDIoU to obtain an improved target detection module; A dust weather target detection model is constructed based on the improved image restoration module and the improved target detection module, the dust weather target detection model is trained based on the road scene image dataset, and image target detection in windy and sandy weather is performed according to the trained dust weather target detection model.

2. A sandstorm target detection method according to claim 1, characterized in that: The process of constructing the road scene image dataset includes: Obtain open source synthetic scene sand data and road images taken in sandy weather; The open-source synthetic scene wind and sand data and the road image are preprocessed based on data enhancement technology and manual labeling method, and a road scene image dataset is constructed based on the preprocessed image data.

3. The method for detecting targets in sandstorm weather according to claim 1, characterized in that: The image restoration module is improved based on the dust scattering model, and the specific calculation formula includes: I(x)=(J(x)-A+1)t(x)+A J(x)=K(x)I(x)-K(x)+b Where I(x) is a haze-free image, A is the atmospheric light value, t(x) is a set of transfer functions, K(x) is the parameter obtained by integrating the atmospheric light value and the transfer function, J(x) is a dust-free image, and b is a constant bias with a default value of 1.

4. The method for detecting targets in sandstorm weather according to claim 1, characterized in that: A small target detection layer is added to the target detection module and a loss function InnerMPDIoU is introduced, specifically including: A small target detection layer is added on the basis of the three original detection layers of the YOLOv8 model; the Inner-IoU loss function is integrated into the MPDIoU loss function to obtain the InnerMPDIoU loss function, and the InnerMPDIoU loss function is used as the loss function of the target detection module to obtain an improved target detection module.

5. The method for detecting targets in sandstorm weather according to claim 1, characterized in that: The process of training the sandstorm target detection model includes: Dividing the road scene image dataset into a training set, a test set, and a validation set based on a preset division rule; The images in the training set are input into the improved image restoration module for dust removal, and the images after dust removal are input into the improved target detection module for target detection, and the classification confidence and position coordinates of the target are output; the network parameters of the two modules are simultaneously updated through the back propagation algorithm; the dust weather target detection model after parameter update is evaluated based on the validation set, and the dust weather target detection model is jointly optimized based on the evaluation results, and the iterative training and optimization process is repeated until the preset number of iterations is reached, and the trained dust weather target detection model is output.

6. A sandstorm target detection system, characterized in that: include: Data acquisition module, used to construct a road scene image dataset under windy and sandy weather; A model construction training module is used to construct an image restoration module based on the AOD-Net image defogging network, and improve the image restoration module based on the dust scattering model to obtain an improved image restoration module; a target detection module is constructed based on the YOLOv8 model, a small target detection layer is added to the target detection module, and a loss function InnerMPDIoU is introduced to obtain an improved target detection module; a sandstorm weather target detection model is constructed based on the improved image restoration module and the improved target detection module, and the sandstorm weather target detection model is trained based on the road scene image dataset; The target detection module is used to perform image target detection in sandstorm weather based on the trained sandstorm weather target detection model.

7. An electronic device, characterized in that: It includes a memory and a processor, the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform a sandstorm weather target detection method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that: It stores a computer program, which, when executed by a processor, implements a sandstorm weather target detection method as described in any one of claims 1 to 5.

Images