A method and apparatus for wildfire image augmentation
By preprocessing and target detection of wildfire images, and calculating the optimal seed point to fuse sunlight and cloud/fog images, the false detection problem in wildfire detection was solved, and the detection accuracy and model precision were improved.
Patent Information
- Application Number
- CN202410617311.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-17
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2044-05-17
Smart Images

Figure CN118537638B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to a method and apparatus for augmenting wildfire images. Background Technology
[0002] Trees, thatch, and other flammable and explosive materials under power transmission lines in power transmission corridors can easily cause wildfires, which can significantly reduce the insulation of the air gaps around the power transmission lines, leading to power line tripping.
[0003] Existing research on wildfire target detection algorithms for power transmission corridors generally relies on wildfire image samples collected from actual fields. After labeling flames and smoke, the wildfire image dataset is used as a training set to train the target detection model. However, due to the limited number of wildfire image samples, when the detection model is trained for on-site wildfire detection, it is prone to misdetecting targets such as sunlight and clouds as flames and smoke, resulting in low accuracy in wildfire detection. Summary of the Invention
[0004] This invention provides a method and apparatus for wildfire image augmentation to solve the problem that existing wildfire detection methods often misdetect targets such as sunlight and clouds as flames and smoke, resulting in low detection accuracy.
[0005] To address the aforementioned technical problems, embodiments of the present invention provide a wildfire image augmentation method, comprising:
[0006] Collect a dataset of wildfire images along the power transmission corridor, and perform preprocessing operations on the wildfire image dataset, including image filtering and image segmentation.
[0007] The wildfire images in the preprocessed wildfire image dataset are detected using a target detection algorithm to obtain the coordinate information of the first and second detection boxes of the wildfire images; the first detection box is the target detection box of the skyline of the wildfire image, and the second detection box is the target detection box of the flames and smoke.
[0008] Based on the coordinate information of the first and second detection boxes of the wildfire image, calculate the optimal sunlight seed point and the optimal smoke seed point of the wildfire image;
[0009] The expanded wildfire image is obtained by fusing the sunlight image and the cloud and fog image respectively based on the optimal sunlight seed point and the optimal cloud and fog seed point in the wildfire image.
[0010] This invention detects and identifies skylines, flames, and smoke in wildfire images, and determines the optimal sunlight seed point and optimal cloud seed point in the wildfire images based on the skyline, flames, and smoke. The sunlight image and the cloud image are then fused based on the optimal sunlight seed point and optimal cloud seed point, thereby expanding the background of the acquired wildfire images and increasing the number of false detection elements in the background. This avoids the irrationality caused by arbitrary fusion, makes the data samples more diverse, and improves the accuracy of the model trained based on the sample data.
[0011] Furthermore, the collection of wildfire image datasets along the power transmission corridor involves preprocessing the wildfire image datasets, specifically as follows:
[0012] The wildfire image dataset is filtered according to preset image selection rules, and the wildfire images are compressed according to preset size to generate a first wildfire image dataset; the image selection rules include resolution, time, image elements, and weather;
[0013] The first wildfire image in the first wildfire image dataset is segmented according to the semantic segmentation model to determine the skyline of the first wildfire image and its corresponding location information, thus completing the preprocessing operation.
[0014] Furthermore, before detecting wildfire images in the wildfire image dataset according to the target detection algorithm and obtaining the coordinate information of the first and second detection boxes of the wildfire images, the method further includes:
[0015] To establish a coordinate system for the preprocessed wildfire image based on the reference point, and to determine the image coordinate information of all segmentation points of the skyline based on the location information of the skyline;
[0016] The image coordinate information of all the segmentation points of the skyline is filled into a one-dimensional matrix according to the size of the horizontal coordinate to generate the position matrix of the skyline. The vertical coordinate of the unsuccessfully segmented points in the position matrix of the skyline is set to a null value.
[0017] Furthermore, the step of detecting wildfire images in the preprocessed wildfire image dataset using a target detection algorithm to obtain the coordinate information of the first and second detection boxes of the wildfire images specifically involves:
[0018] The wildfire images in the preprocessed wildfire image dataset are detected using a target detection algorithm to obtain the position information of the first and second detection boxes of the wildfire images.
[0019] The coordinate information of the first detection frame and the second detection frame is determined according to the coordinate system. The coordinate information includes the center coordinates of the detection frame and the size information of the detection frame.
[0020] Furthermore, the step of calculating the optimal sunlight seed point and optimal smoke seed point of the wildfire image based on the coordinate information of the first and second detection boxes of the wildfire image specifically involves:
[0021] Obtain the pixel coordinates of the wildfire image and generate a pixel matrix;
[0022] Several elements in the pixel matrix are randomly selected as seed points according to a preset threshold number, and the position coordinates of each seed point are obtained.
[0023] The optimal sunlight seed point and optimal smoke seed point of the wildfire image are calculated based on the position matrix of the skyline and the coordinate information of the second detection box.
[0024] Furthermore, the step of calculating the optimal sunlight seed point and optimal smoke seed point of the wildfire image based on the position matrix of the skyline and the coordinate information of the second detection box specifically involves:
[0025] Iterate through the position coordinates of each seed point, and remove the seed points whose x-coordinate is the same as the first x-coordinate. The first x-coordinate is the x-coordinate of the element whose y-coordinate is null in the position matrix of the skyline.
[0026] The seed points are filtered based on the distance between each seed point and the center point of the second detection box, the distance of the second detection box to the diagonal, and the radius of the image to be generated, to obtain the first seed point set;
[0027] Traverse the first seed point set and filter out the second seed points that have the same x-coordinate as the elements in the position matrix of the skyline;
[0028] The optimal sunlight seed point and the optimal smoke seed point are determined based on the distance between the second seed point and the vertical coordinate of the first element and the distance between the second seed point and the image edge. The first element is the element in the position matrix of the skyline whose horizontal coordinate is the same as the horizontal coordinate of the second seed point.
[0029] Furthermore, the step of determining the optimal sunlight seed point and the optimal smoke seed point based on the distance between the second seed point and the ordinate of the first element and the distance between the second seed point and the image edge specifically involves:
[0030] Based on the difference between the ordinate of the second seed point and the first element, the second seed point is divided into a sunlight seed point set and a cloud and fog seed point set. The difference between the ordinate of the seed point in the sunlight seed point set and the first element is greater than zero, while the difference between the ordinate of the seed point in the cloud and fog seed point set and the first element is less than zero.
[0031] Calculate the square of the first difference between the ordinate of each sun seed point in the sun seed point set and its corresponding first element, and calculate the square of the first distance between each sun seed point and the top of the wildfire image. The point with the smallest absolute value of the difference between the square of the first difference and the square of the first distance is determined as the optimal sun seed point.
[0032] Calculate the squared second difference between the ordinate of each cloud seed point in the cloud seed point set and its corresponding first element, and calculate the squared second distance between each cloud seed point and the bottom of the wildfire image. The point with the smallest absolute value of the difference between the squared second difference and the squared second distance is determined as the optimal cloud seed point.
[0033] Furthermore, the step of fusing the sunlight image and the cloud / fog image based on the optimal sunlight seed point and the optimal cloud / fog seed point in the wildfire image to obtain the augmented wildfire image specifically involves:
[0034] Acquire sunlight images and cloud / fog images, and perform background removal and compression processing on the sunlight images and cloud / fog images;
[0035] The center points of the processed sunlight image and cloud image are placed on the optimal sunlight seed point and the optimal cloud seed point, respectively, and then fused to obtain the expanded wildfire image.
[0036] Secondly, the present invention provides a wildfire image augmentation device, comprising: a preprocessing module, a detection module, a seed point determination module, and a fusion module;
[0037] The preprocessing module is used to collect wildfire image datasets of the power transmission corridor and perform preprocessing operations on the wildfire image datasets, including image filtering and image segmentation.
[0038] The detection module is used to detect wildfire images in the preprocessed wildfire image dataset according to the target detection algorithm, and obtain the coordinate information of the first detection box and the second detection box of the wildfire image; the first detection box is the target detection box of the skyline of the wildfire image, and the second detection box is the target detection box of flames and smoke.
[0039] The seed point determination module is used to calculate the optimal sunlight seed point and the optimal smoke seed point of the wildfire image based on the coordinate information of the first and second detection boxes of the wildfire image.
[0040] The fusion module is used to fuse the sunlight image and the cloud image according to the optimal sunlight seed point and the optimal cloud seed point in the wildfire image, respectively, to obtain the expanded wildfire image.
[0041] Thirdly, the present invention also provides a computer device, including: a processor, a communication interface and a memory, wherein the processor, the communication interface and the memory are interconnected, wherein the memory stores executable program code, and the processor is used to call the executable program code to execute the wildfire image augmentation method. Attached Figure Description
[0042] Figure 1 A schematic flowchart of a wildfire image augmentation method provided in an embodiment of the present invention;
[0043] Figure 2 This is another flowchart illustrating the wildfire image augmentation method provided in an embodiment of the present invention.
[0044] Figure 3 This is a schematic diagram of target detection in a wildfire image augmentation method provided in an embodiment of the present invention;
[0045] Figure 4 A sunlight image is provided as an embodiment of the wildfire image augmentation method of the present invention.
[0046] Figure 5 A cloud and fog image provided by the wildfire image augmentation method in this embodiment of the invention;
[0047] Figure 6 An augmented wildfire image is provided as an embodiment of the wildfire image augmentation method according to the present invention. Detailed Implementation
[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0049] Example 1
[0050] Please refer to Figure 1 , Figure 1 A flowchart illustrating a wildfire image augmentation method provided in this embodiment of the invention includes steps 101 to 104, as detailed below:
[0051] Step 101: Collect a dataset of wildfire images along the power transmission corridor, and perform preprocessing operations on the wildfire image dataset, including image filtering and image segmentation.
[0052] In this embodiment, a dataset of wildfire images of the power transmission corridor is collected using photographic or radar equipment, and wildfire images that do not meet the requirements are removed from the dataset according to preset image selection rules.
[0053] In this embodiment, the process of collecting wildfire image datasets along the power transmission corridor and performing preprocessing operations on the wildfire image datasets specifically includes:
[0054] The wildfire image dataset is filtered according to preset image selection rules, and the wildfire images are compressed according to preset size to generate a first wildfire image dataset; the image selection rules include resolution, time, image elements, and weather;
[0055] The first wildfire image in the first wildfire image dataset is segmented according to the semantic segmentation model to determine the skyline of the first wildfire image and its corresponding location information, thus completing the preprocessing operation.
[0056] In this embodiment, the preset image selection rules stipulate that images with a resolution lower than a preset resolution threshold, nighttime images, cloudy images, images without sky, and images with sunlight or fog in the wildfire image dataset should be removed.
[0057] In this embodiment, images with a resolution of less than 640×640, images that are taken at night or on cloudy days, images that do not show the sky, and images that show sunlight or fog are obtained to obtain images that can be fused for image augmentation, thus addressing the irrationality of the image fusion.
[0058] In this embodiment, the wildfire image is compressed according to a preset size. Specifically, the larger of the length and width of the wildfire image is selected and compressed to a preset resolution threshold. The compression ratio is calculated based on the original size and the resolution threshold. The other side of the wildfire image is compressed proportionally according to the compression ratio, and the remaining part is filled with the background color.
[0059] As a specific example of an embodiment of the present invention, the remaining wildfire image size is compressed to 640×640. The compression method adopts proportional scaling, compressing the larger value to 640, compressing the other value proportionally, and filling the other parts with the background color.
[0060] In this embodiment, the trained DeepLabV3 semantic segmentation model is used to segment the sky and ground of the selected images, obtaining the sky and ground segmentation lines and their location information. Specifically, a semantic segmentation model is constructed based on the original DeepLabV3 semantic segmentation model, and the labeling software Labellme is used to label the sky and ground segmentation lines (i.e., the skyline) of 200 randomly selected images. After the labeling is completed, the semantic segmentation model is directly used as the training set for training. After training, all images in the first wildfire image dataset are segmented to obtain the segmentation lines and their location information.
[0061] Step 102: Detect wildfire images in the preprocessed wildfire image dataset using the target detection algorithm, and obtain the coordinate information of the first detection box and the second detection box of the wildfire image; the first detection box is the target detection box of the skyline of the wildfire image, and the second detection box is the target detection box of flames and smoke;
[0062] In this embodiment, before detecting wildfire images in the wildfire image dataset according to the target detection algorithm and obtaining the coordinate information of the first and second detection boxes of the wildfire images, the method further includes:
[0063] To establish a coordinate system for the preprocessed wildfire image based on the reference point, and to determine the image coordinate information of all segmentation points of the skyline based on the location information of the skyline;
[0064] The image coordinate information of all the segmentation points of the skyline is filled into a one-dimensional matrix according to the size of the horizontal coordinate to generate the position matrix of the skyline. The vertical coordinate of the unsuccessfully segmented points in the position matrix of the skyline is set to a null value.
[0065] In this embodiment, the lower left corner of the wildfire image is selected as the reference point, i.e., the origin of the coordinate axis (0, 0). The horizontal axis of the image is the x-axis, and the vertical axis is the y-axis, to establish a coordinate system. Then, a one-dimensional empty matrix A is generated. The empty matrix is a 1-row N-column empty matrix, where N is a preset resolution threshold size. The resolution threshold is set to 640 by default, i.e., the empty matrix is 1 row and 640 columns.
[0066] In this embodiment, the step of detecting wildfire images in the preprocessed wildfire image dataset using a target detection algorithm and obtaining the coordinate information of the first and second detection boxes of the wildfire images specifically involves:
[0067] The wildfire images in the preprocessed wildfire image dataset are detected using a target detection algorithm to obtain the position information of the first and second detection boxes of the wildfire images.
[0068] The coordinate information of the first detection frame and the second detection frame is determined according to the coordinate system. The coordinate information includes the center coordinates of the detection frame and the size information of the detection frame.
[0069] In this embodiment, a trained YOLOv8 object detection model is used to detect flames and smoke in a pre-processed wildfire image dataset, obtaining the location information of a first detection box and a second detection box. The first detection box is the target detection box for the skyline of the wildfire image, and the second detection box is the target detection box for both flames and smoke. The trained YOLOv8 object detection model first uses the labeling software Labellimg to label 1000 randomly selected images for the two targets of flames and smoke. After completion, this is used as the training set and added to the YOLOv8 object detection model for training. After training, all images in the pre-processed wildfire image dataset are detected to obtain target detection boxes and their location information.
[0070] In this embodiment, the obtained segmentation line position information is used to acquire the image coordinate information (x, y, y) of each segmentation point of the skyline. sk y sk The coordinate information is filled into an empty matrix A in ascending order of x-coordinate, with the y-coordinate of unsuccessfully segmented points set to a null value of 0. The center point coordinates (x, y) of the target detection box are determined using the target detection box information. pr y pr The detection frame size information includes the width and height information w of the target detection frame. pr h pr .
[0071] Step 103: Calculate the optimal sunlight seed point and optimal smoke seed point of the wildfire image based on the coordinate information of the first and second detection boxes of the wildfire image;
[0072] In this embodiment, the step of calculating the optimal sunlight seed point and optimal smoke seed point of the wildfire image based on the coordinate information of the first and second detection boxes of the wildfire image specifically involves:
[0073] Obtain the pixel coordinates of the wildfire image and generate a pixel matrix;
[0074] Several elements in the pixel matrix are randomly selected as seed points according to a preset threshold number, and the position coordinates of each seed point are obtained.
[0075] The optimal sunlight seed point and optimal smoke seed point of the wildfire image are calculated based on the position matrix of the skyline and the coordinate information of the second detection box.
[0076] In this embodiment, based on the acquired wildfire image, the position matrix of the skyline, and the coordinate information of the second detection box, the optimal sunlight seed point and the optimal cloud seed point are determined. Specifically, the pixel coordinates of the wildfire image are read one by one to generate a 640×640 matrix B, and each element of the matrix is represented in the original pixel coordinate form. 1000 elements in the matrix are randomly selected as seed points using a random but non-repeating sampling method, and the position information of each seed point is simultaneously confirmed as coordinates (x, y, y). se y se Then, based on the position matrix of the skyline and the coordinate information of the second detection box, the optimal sunlight seed point and the optimal smoke seed point of the wildfire image are calculated.
[0077] In this embodiment, the step of calculating the optimal sunlight seed point and optimal smoke seed point of the wildfire image based on the position matrix of the skyline and the coordinate information of the second detection box specifically involves:
[0078] Iterate through the position coordinates of each seed point, and remove the seed points whose x-coordinate is the same as the first x-coordinate. The first x-coordinate is the x-coordinate of the element whose y-coordinate is null in the position matrix of the skyline.
[0079] The seed points are filtered based on the distance between each seed point and the center point of the second detection box, the distance of the second detection box to the diagonal, and the radius of the image to be generated, to obtain the first seed point set;
[0080] Traverse the first seed point set and filter out the second seed points that have the same x-coordinate as the elements in the position matrix of the skyline;
[0081] The optimal sunlight seed point and the optimal smoke seed point are determined based on the distance between the second seed point and the vertical coordinate of the first element and the distance between the second seed point and the image edge. The first element is the element in the position matrix of the skyline whose horizontal coordinate is the same as the horizontal coordinate of the second seed point.
[0082] In this embodiment, determining the optimal sunlight seed point and the optimal smoke seed point based on the distance between the second seed point and the ordinate of the first element and the distance between the second seed point and the image edge specifically involves:
[0083] Based on the difference between the ordinate of the second seed point and the first element, the second seed point is divided into a sunlight seed point set and a cloud and fog seed point set. The difference between the ordinate of the seed point in the sunlight seed point set and the first element is greater than zero, while the difference between the ordinate of the seed point in the cloud and fog seed point set and the first element is less than zero.
[0084] Calculate the square of the first difference between the ordinate of each sun seed point in the sun seed point set and its corresponding first element, and calculate the square of the first distance between each sun seed point and the top of the wildfire image. The point with the smallest absolute value of the difference between the square of the first difference and the square of the first distance is determined as the optimal sun seed point.
[0085] Calculate the squared second difference between the ordinate of each cloud seed point in the cloud seed point set and its corresponding first element, and calculate the squared second distance between each cloud seed point and the bottom of the wildfire image. The point with the smallest absolute value of the difference between the squared second difference and the squared second distance is determined as the optimal cloud seed point.
[0086] In this embodiment, the position information of each generated seed point is traversed, and each seed point position information is compared with each element in the position matrix A of the skyline, excluding the horizontal coordinate x. se The x-coordinate of the point dividing the skyline is 0. sk The same seed point; then determine the coordinates (x, y) of the seed point from the center point of the second detection box. pr y pr Is the distance greater than half the diagonal distance of the second detection box? pr The sum of the points and the radius r of the image to be generated is used to obtain the first seed point set. If the sum is greater than the radius r, the points are retained; otherwise, they are removed.
[0087] In this embodiment, the seed point is located at the center point coordinates (x, y) of the second detection frame. pr y pr Distance, specifically:
[0088]
[0089] In this embodiment, half the diagonal distance r of the second detection frame pr The sum of the radius r of the image to be generated is as follows:
[0090]
[0091] Among them, (x se y se (x) represents the coordinates of the seed point. pr y pr ) represents the coordinates of the center point of the box, w pr h pr These represent the width and height of the bounding box, respectively, and r represents the radius of the image to be generated. pr This represents half the distance of the diagonal of the box.
[0092] In this embodiment, the first seed point set is traversed, and second seed points with the same horizontal coordinate as the elements in the position matrix of the skyline are selected. Then, the optimal sunlight seed point and the optimal smoke seed point are determined based on the distance between the second seed point and the vertical coordinate of the first element and the distance between the second seed point and the image edge.
[0093] In this embodiment, determining the optimal sunlight seed point and the optimal smoke seed point based on the distance between the second seed point and the ordinate of the first element and the distance between the second seed point and the image edge specifically involves:
[0094] Based on the difference between the ordinate of the second seed point and the first element, the second seed point is divided into a sunlight seed point set and a cloud and fog seed point set. The difference between the ordinate of the seed point in the sunlight seed point set and the first element is greater than zero, while the difference between the ordinate of the seed point in the cloud and fog seed point set and the first element is less than zero.
[0095] Calculate the square of the first difference between the ordinate of each sun seed point in the sun seed point set and its corresponding first element, and calculate the square of the first distance between each sun seed point and the top of the wildfire image. The point with the smallest absolute value of the difference between the square of the first difference and the square of the first distance is determined as the optimal sun seed point.
[0096] Calculate the squared second difference between the ordinate of each cloud seed point in the cloud seed point set and its corresponding first element, and calculate the squared second distance between each cloud seed point and the bottom of the wildfire image. The point with the smallest absolute value of the difference between the squared second difference and the squared second distance is determined as the optimal cloud seed point.
[0097] In this embodiment, the first seed point set is traversed, and the x-coordinate of the seed point in the first seed point set is selected. se Same x-axis as the skyline sk point.
[0098] In this embodiment, the difference s between the ordinates of two points is taken. If s is greater than 0, it indicates that the seed point is located above the segmented skyline and is a sunlight seed point; the square of the first difference s between the ordinates is taken. 2 And the squared first distance s1 from the top of the image to the sunlight seed point 2 The seed point with the smallest absolute value d of the difference is the optimal sunlight seed point.
[0099] In this embodiment, the difference s between the ordinates of two points is taken. If s is less than 0, it indicates that the seed point is located below the segmented skyline and is a cloud / fog seed point. The square of the second difference s between the ordinates is then taken. 2 And the squared second distance s2 from the cloud seed point to the bottom of the image 2 The seed point with the smallest absolute value of the difference d is the optimal cloud seed point.
[0100] In this embodiment, the two points are the x-coordinates of the seed points in the first seed point set. se Same x-axis as the skyline sk point.
[0101] In this embodiment, the difference in the vertical coordinates is specifically as follows:
[0102] s=y se -y s k (3)
[0103] In this embodiment, the distance from the top of the image to the sunlight seed point is:
[0104] s1 = 640 - y se (4)
[0105] In this embodiment, the distance from the cloud seed point to the bottom of the image is:
[0106] s2=y se (5)
[0107] In this embodiment, the formula for determining the optimal seed point is as follows:
[0108] d best =min{|s 2 -s i=1,2 2 |} (6)
[0109] Where s represents the difference between the ordinate of the seed point and the corresponding skyline segmentation point, s1 represents the distance between the sunlight seed point and the top of the image, and s2 represents the distance between the cloud seed point and the bottom of the image.
[0110] Step 104: Based on the optimal sunlight seed point and the optimal cloud and fog seed point in the wildfire image, fuse the sunlight image and the cloud and fog image respectively to obtain the expanded wildfire image.
[0111] In this embodiment, the step of fusing the sunlight image and the cloud / fog image based on the optimal sunlight seed point and the optimal cloud / fog seed point in the wildfire image to obtain the augmented wildfire image specifically involves:
[0112] Acquire sunlight images and cloud / fog images, and perform background removal and compression processing on the sunlight images and cloud / fog images;
[0113] The center points of the processed sunlight image and cloud image are placed on the optimal sunlight seed point and the optimal cloud seed point, respectively, and then fused to obtain the expanded wildfire image.
[0114] In this embodiment, sunlight images and cloud / fog images are acquired, compressed according to default sizes, and then fused based on the optimal sunlight seed point and cloud / fog seed point in each of the obtained images.
[0115] In this embodiment, sunlight and cloud / fog images collected from the internet are segmented using an image segmentation algorithm to obtain sunlight and cloud / fog element images. All other backgrounds are removed. The 100 sunlight and cloud / fog element images with the best segmentation results are selected. The collected images are then compressed to 40×40 using a proportional scaling method. The larger value is compressed to 40, and the other value is compressed proportionally. The remaining parts are filled with the background color. The compressed sunlight and cloud / fog element images are then placed on the optimal sunlight seed point and optimal smoke seed point of each wildfire image in the wildfire image dataset, respectively, to obtain the expanded power transmission corridor wildfire image dataset.
[0116] In this embodiment, after image augmentation is completed, the effectiveness of image augmentation in reducing false detections of sunlight and cloud targets in the images needs to be verified using the original YOLOv8 target detection model. Specifically, the augmented power transmission corridor wildfire image dataset is divided into a training validation set and a test set in a ratio of 8:2. The training validation set is added to the YOLOv8 target detection model for training. After obtaining the optimal weight W, images prone to false detection are detected, and the false detection rate is calculated based on the detection results. The effectiveness of the augmented power transmission corridor wildfire image dataset is evaluated based on the false detection rate.
[0117] Please refer to Figure 2 , Figure 2 Another schematic flowchart of the wildfire image augmentation method provided in this embodiment of the invention includes S1 to S4, specifically as follows:
[0118] S1: Collect a dataset of images of wildfires along the power transmission corridor, and perform image filtering and preprocessing.
[0119] S2: Based on the image data information obtained after preprocessing, set coordinate axes for the image, confirm the dividing line between the sky and the ground in the image, as well as the position information of the target detection boxes for flames and smoke, and represent the position information of the dividing line and the detection boxes in the form of coordinates;
[0120] S3: Based on the acquired image data information, determine the optimal seed points for generating sunlight and clouds;
[0121] S4: Based on the optimal sunlight seed point and cloud seed point in each obtained image, fuse the sunlight image and cloud image respectively, and verify the effectiveness.
[0122] In this embodiment, a dataset of 2,622 wildfire images was collected and preprocessed. The preprocessing process specifically includes: removing images with a resolution less than 640×640, removing images that are nighttime or cloudy, removing images that do not show the sky, and removing images that show sunlight or fog. The remaining images are then compressed to 640×640. After manual preprocessing, 2,106 images remain.
[0123] Please refer to Figure 3 , Figure 3 This is a schematic diagram of target detection in a wildfire image augmentation method provided in an embodiment of the present invention.
[0124] In this embodiment, 200 images were selected and labeled using the labeling tool Labellme to create segmentation line labels. These labels were then used as the training set to train DeepLabV3. After obtaining the optimal weights, the remaining images were processed. 97.8% of the images, totaling 1956 images, successfully yielded skyline segmentation lines. The location information of each segmentation point on the skyline was obtained and recorded in a txt file with the same name as the image. Using the labeling tool Labellimg, 1000 randomly selected images from the wildfire image sample library were labeled in Pascal VOC dataset format. Flames and smoke were selected as the recognition objects and labeled as 'f'. After dividing the dataset into training and test sets, the YOLOv8 model was trained. The YOLOv8 model with the optimal weights was then used to predict flames and smoke targets in 1956 images, obtaining bounding boxes and their information. Predictions were successful for all images. However, some false positives were found during subsequent checks. After deleting some images, 1832 images remained. The falsely detected bounding boxes in these 1832 images were then removed, resulting in the final 1832 images. These 1832 images contain both segmented skylines and predicted bounding boxes. Figure 3 As shown in the figure, the yellow line is the dividing line and the red box is the target detection box.
[0125] In this embodiment, based on the image data information obtained after preprocessing, coordinate axes are set for the image, the dividing lines between the sky and the ground in the image, as well as the position information of the target detection boxes for flames and smoke, are confirmed, and the position information of the dividing lines and detection boxes is represented in the form of coordinates.
[0126] In this embodiment, the lower left corner of the image is set as the origin of the coordinate axis (0, 0), the horizontal axis is the x-axis, and the vertical axis is the y-axis. After obtaining the dividing line and the coordinates of each dividing point on the dividing line, the position of the dividing line in the image is represented in matrix form, where all matrix elements are point coordinates. The minimum value of each point coordinate is controlled to be a single digit by rounding. If there are some empty values, the vertical coordinate is set to 0. The position information of the detection box is also rounded to a single digit. The first column of information indicates the type of detection target, '0' indicates smoke and '1' indicates flame. The second and third columns of information indicate the horizontal and vertical coordinates of the center point of the detection box. The fourth and fifth columns of information indicate the width and height of the detection box, all in units of one pixel.
[0127] In this embodiment, the pixel coordinates of the image are read one by one to generate a 640×640 matrix B, and each element of the matrix is represented in the original pixel coordinate form. 1000 elements in the matrix are randomly selected as seed points using a random but non-repeating sampling method, and the position information of each seed point is simultaneously confirmed to be represented in coordinate form (x, y, y). se y se The next step is to select the optimal seed point. The specific process for selecting the optimal seed point includes: traversing the position information of each generated seed point, comparing the position information of each seed point with each element in matrix A, and excluding the x-coordinate. se The x-coordinate of the point dividing the skyline is 0. sk The same seed point; then determine the coordinates (x, y) of the seed point relative to the center point of the bounding box of the wildfire image's flames and smoke targets. pr y pr Is the distance greater than half the diagonal distance of the box? pr The sum of the seed point and the radius r of the image to be generated is used. If the sum is greater than the seed point, the seed point is retained; otherwise, it is removed. Finally, the optimal seed point is selected, and the remaining seed points after the elimination are traversed. Then, the x-coordinate of the seed point is selected. se Same x-axis as the skyline sk For each point, the difference s between the ordinates of the two points is taken. If s is greater than 0, it indicates that the seed point is above the segmented skyline and is a sunlight seed point. The square of the difference s is then taken. 2 And the squared distance s1 from the top of the image to the sunlight seed point 2 The seed point with the smallest absolute value d of the difference is the optimal sunlight seed point; if s is less than 0, it indicates that this seed point is below the segmented skyline and is a cloud / fog seed point, and the square of the difference in the ordinate is taken as s. 2 And the squared distance s2 from the cloud seed point to the bottom of the image. 2 The seed point with the smallest absolute value of the difference d is the optimal cloud seed point.
[0128] In this embodiment, 1000 coordinate elements within a 640×640 matrix were randomly and non-repeatingly extracted using the NumPy library in Python. After selecting the optimal seed point, the following situations occurred: most images generated two optimal seed points; some images had only one optimal seed point, and these images were retained; a small number of images had multiple optimal sunlight seed points and optimal cloud / fog seed points, and these images were manually deleted by selecting the optimal seed point; some images did not have optimal sunlight seed points and optimal cloud / fog seed points, and these images were also deleted; finally, a total of 1488 images remained that could be successfully fused into sunlight or cloud / fog images.
[0129] Please refer to Figure 4 , Figure 5 and Figure 6 , Figure 4 A sunlight image is provided as an embodiment of the wildfire image augmentation method according to the present invention. Figure 5 A cloud and fog image provided by the wildfire image augmentation method in this embodiment of the invention; Figure 6 An augmented wildfire image is provided as an embodiment of the wildfire image augmentation method according to the present invention.
[0130] In this embodiment, a large number of sunlight and cloud / fog images were collected from the internet. Color space segmentation and threshold adjustment methods were used to segment the sunlight and cloud / fog images separately, resulting in images containing only sunlight and cloud / fog with all other background elements removed. The 100 highest quality sunlight and cloud / fog images were then selected, such as... Figure 4 and Figure 5 As shown, the collected images are compressed to 40×40 using a proportional scaling method. The larger value (length and width) is compressed to 40, and the other value is compressed proportionally. The remaining parts are filled with the background color. Sunlight and cloud / fog images are fused for all original power transmission line corridor wildfire images. The center point of the compressed sunlight and cloud / fog images is placed at the optimal seed point position, as shown below. Figure 6As shown, an expanded dataset of 1488 images of wildfires along the power transmission corridor was obtained. These expanded images were then added to the original dataset, replacing the original 1488 images. The expanded dataset was divided into a training / validation set and a test set in an 8:2 ratio. The training / validation set was then used to train a YOLOv8 object detection model. During training, 200 training epochs were set, the batch size was 16, the maximum learning rate was 0.01, and the minimum learning rate was 0.0001. Validation evaluation was also implemented in each epoch. In one test, the optimal training weight W was obtained by comparing the loss values. The optimal training weight W was then input into the YOLOv8 object detection model. A total of 1202 images with high false positive rates were used for testing. These images contained no flames or smoke targets, and most of them were easily misdetected targets such as sunlight, clouds, and fog. The result showed a false positive rate of 10.74%. Compared with the result of training on the unexpanded power transmission line corridor wildfire image dataset, which had a false positive rate of 74.56%, the false positive rate was significantly reduced. This effectively verified that the image expansion method can reduce the false positive rate of easily misdetected targets in wildfires.
[0131] This invention also provides a wildfire image augmentation device, comprising: a preprocessing module, a detection module, a seed point determination module, and a fusion module;
[0132] The preprocessing module is used to collect wildfire image datasets of the power transmission corridor and perform preprocessing operations on the wildfire image datasets, including image filtering and image segmentation.
[0133] The detection module is used to detect wildfire images in the preprocessed wildfire image dataset according to the target detection algorithm, and obtain the coordinate information of the first detection box and the second detection box of the wildfire image; the first detection box is the target detection box of the skyline of the wildfire image, and the second detection box is the target detection box of flames and smoke.
[0134] The seed point determination module is used to calculate the optimal sunlight seed point and the optimal smoke seed point of the wildfire image based on the coordinate information of the first and second detection boxes of the wildfire image.
[0135] The fusion module is used to fuse the sunlight image and the cloud image according to the optimal sunlight seed point and the optimal cloud seed point in the wildfire image, respectively, to obtain the expanded wildfire image.
[0136] This invention also provides a computer device, including: a processor, a communication interface, and a memory, wherein the processor, the communication interface, and the memory are interconnected, wherein the memory stores executable program code, and the processor is used to call the executable program code to execute the wildfire image augmentation method.
[0137] In this embodiment, by detecting and identifying the skyline, flames, and smoke in wildfire images, and determining the optimal sunlight seed point and optimal cloud seed point in the wildfire images based on the skyline, flames, and smoke, the sunlight image and cloud image are fused based on the optimal sunlight seed point and optimal cloud seed point. This process expands the background of the acquired wildfire images, increases the number of false detection elements in the background, avoids the irrationality caused by arbitrary fusion, makes the data samples more diverse, and improves the accuracy of the model trained based on the sample data.
[0138] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.
Claims
1. A method for augmenting wildfire images, characterized in that, include: Collect a dataset of wildfire images along the power transmission corridor, and perform preprocessing operations on the wildfire image dataset, including image filtering and image segmentation. The wildfire images in the preprocessed wildfire image dataset are detected using a target detection algorithm to obtain the coordinate information of the first and second detection boxes of the wildfire images; The first detection box is the target detection box of the skyline of the wildfire image, and the second detection box is the target detection box of the flames and smoke; Based on the coordinate information of the first and second detection boxes of the wildfire image, the optimal sunlight seed point and the optimal smoke seed point of the wildfire image are calculated; specifically, the pixel coordinates of the wildfire image are obtained and a pixel matrix is generated. Several elements in the pixel matrix are randomly selected as seed points according to a preset threshold number, and the position coordinates of each seed point are obtained; the position coordinates of each seed point are traversed, and seed points with the same horizontal coordinate as the first horizontal coordinate are removed. The first horizontal coordinate is the horizontal coordinate of the element with a null vertical coordinate in the position matrix of the skyline; each seed point is filtered according to the distance between each seed point and the center point of the second detection box, the distance of the diagonal of the second detection box, and the radius of the image to be generated, to obtain the first seed point set; Traverse the first seed point set and filter out the second seed points whose x-coordinates are the same as those of the elements in the position matrix of the skyline; based on the difference between the y-coordinates of the second seed points and the first element, divide the second seed points into a sunlight seed point set and a cloud seed point set, wherein the difference between the y-coordinates of the seed points in the sunlight seed point set and the first element is greater than zero, and the difference between the y-coordinates of the seed points in the cloud seed point set and the first element is less than zero; the first element is the element in the position matrix of the skyline whose x-coordinate is the same as that of the second seed point. Calculate the square of the first difference between the ordinate of each sun seed point in the sun seed point set and its corresponding first element, and calculate the square of the first distance between each sun seed point and the top of the wildfire image. The point with the smallest absolute value of the difference between the square of the first difference and the square of the first distance is determined as the optimal sun seed point. Calculate the squared second difference between the ordinate of each cloud seed point in the cloud seed point set and its corresponding first element, and calculate the squared second distance between each cloud seed point and the bottom of the wildfire image. The point with the smallest absolute value of the difference between the squared second difference and the squared second distance is determined as the optimal cloud seed point. The enhanced wildfire image is obtained by fusing the sunlight image and the cloud and fog image respectively based on the optimal sunlight seed point and the optimal cloud and fog seed point in the wildfire image.
2. The wildfire image augmentation method as described in claim 1, characterized in that, The data set of wildfire images collected from the power transmission corridor is preprocessed as follows: The wildfire image dataset is filtered according to preset image selection rules, and the wildfire images are compressed according to preset size to generate a first wildfire image dataset; the image selection rules include resolution, time, image elements, and weather; The first wildfire image in the first wildfire image dataset is segmented according to the semantic segmentation model to determine the skyline of the first wildfire image and its corresponding location information, thus completing the preprocessing operation.
3. The wildfire image augmentation method as described in claim 2, characterized in that, Before detecting wildfire images in the wildfire image dataset according to the target detection algorithm and obtaining the coordinate information of the first and second detection boxes of the wildfire images, the method further includes: To establish a coordinate system for the preprocessed wildfire image based on the reference point, and to determine the image coordinate information of all segmentation points of the skyline based on the location information of the skyline; The image coordinate information of all the segmentation points of the skyline is filled into a one-dimensional matrix according to the size of the horizontal coordinate to generate the position matrix of the skyline. The vertical coordinate of the unsuccessfully segmented points in the position matrix of the skyline is set to a null value.
4. The wildfire image augmentation method as described in claim 3, characterized in that, The step involves detecting wildfire images in the preprocessed wildfire image dataset using a target detection algorithm, and obtaining the coordinate information of the first and second detection boxes of the wildfire images. Specifically: The wildfire images in the preprocessed wildfire image dataset are detected using a target detection algorithm to obtain the position information of the first and second detection boxes of the wildfire images. The coordinate information of the first detection frame and the second detection frame is determined according to the coordinate system. The coordinate information includes the center coordinates of the detection frame and the size information of the detection frame.
5. The wildfire image augmentation method as described in claim 4, characterized in that, The process of fusing the sunlight image and the cloud / fog image based on the optimal sunlight seed point and the optimal cloud / fog seed point in the wildfire image to obtain the augmented wildfire image specifically involves: Acquire sunlight images and cloud / fog images, and perform background removal and compression processing on the sunlight images and cloud / fog images; The center points of the processed sunlight image and cloud image are placed on the optimal sunlight seed point and the optimal cloud seed point, respectively, and then fused to obtain the expanded wildfire image.
6. A wildfire image augmentation device, characterized in that, include: The module includes a preprocessing module, a detection module, a seed point determination module, and a fusion module. The preprocessing module is used to collect wildfire image datasets of the power transmission corridor and perform preprocessing operations on the wildfire image datasets, including image filtering and image segmentation. The detection module is used to detect wildfire images in the preprocessed wildfire image dataset according to the target detection algorithm, and to obtain the coordinate information of the first detection box and the second detection box of the wildfire image; The first detection box is the target detection box of the skyline of the wildfire image, and the second detection box is the target detection box of the flames and smoke; The seed point determination module is used to calculate the optimal sunlight seed point and the optimal smoke seed point of the wildfire image based on the coordinate information of the first and second detection boxes of the wildfire image; specifically, it obtains the pixel coordinates of the wildfire image and generates a pixel matrix. Several elements in the pixel matrix are randomly selected as seed points according to a preset threshold number, and the position coordinates of each seed point are obtained; the position coordinates of each seed point are traversed, and seed points with the same horizontal coordinate as the first horizontal coordinate are removed. The first horizontal coordinate is the horizontal coordinate of the element with a null vertical coordinate in the position matrix of the skyline; each seed point is filtered according to the distance between each seed point and the center point of the second detection box, the distance of the diagonal of the second detection box, and the radius of the image to be generated, to obtain the first seed point set; Traverse the first seed point set and filter out the second seed points whose x-coordinates are the same as those of the elements in the position matrix of the skyline; based on the difference between the y-coordinates of the second seed points and the first element, divide the second seed points into a sunlight seed point set and a cloud seed point set, wherein the difference between the y-coordinates of the seed points in the sunlight seed point set and the first element is greater than zero, and the difference between the y-coordinates of the seed points in the cloud seed point set and the first element is less than zero; the first element is the element in the position matrix of the skyline whose x-coordinate is the same as that of the second seed point. Calculate the square of the first difference between the ordinate of each sun seed point in the sun seed point set and its corresponding first element, and calculate the square of the first distance between each sun seed point and the top of the wildfire image. The point with the smallest absolute value of the difference between the square of the first difference and the square of the first distance is determined as the optimal sun seed point. Calculate the squared second difference between the ordinate of each cloud seed point in the cloud seed point set and its corresponding first element, and calculate the squared second distance between each cloud seed point and the bottom of the wildfire image. The point with the smallest absolute value of the difference between the squared second difference and the squared second distance is determined as the optimal cloud seed point. The fusion module is used to fuse the sunlight image and the cloud image according to the optimal sunlight seed point and the optimal cloud seed point in the wildfire image, respectively, to obtain the expanded wildfire image.
7. A computer device, characterized in that, include: The system includes a processor, a communication interface, and a memory, which are interconnected. The memory stores executable program code, and the processor is used to call the executable program code to execute the wildfire image augmentation method as described in any one of claims 1 to 5.
Citation Information
Patent Citations
Forest fire smoke detection method based on image segmentation
CN110222644A
Forest fire smoke detection model training method and forest fire smoke identification method
CN116071612A
