A method and system for making a deep learning data set in a flood rescue scene

Abstract

The application discloses a kind of flood rescue scene under the production method and system of deep learning data set, the method in it first uses webpage element acquisition technology to collect picture data under flood rescue scene;Then the target detection model of pre-trained is used to detect the picture data collected, is divided into fallen person picture and other pictures, respectively saved to positive sample folder and flood scene folder;Then the fallen person picture contained in positive sample folder is preprocessed;Then the mean hash and structure similarity method is used to the fallen person picture after preprocessing is de-duplicated, again the fallen person picture left in positive sample folder is used as positive sample, the picture contained in flood scene folder is used as negative sample, and based on the way of multi-label information fusion is used to the positive sample and negative sample is labeled;Finally, the labeled sample is processed using image data augmentation, to obtain the final deep learning data set.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and in particular to a method and system for creating a deep learning dataset for flood relief scenarios. Background Technology

[0002] The three essential elements of deep learning technology, or its three pillars, are algorithms, data, and computing power. Among these three elements, researchers mostly focus on algorithms, and computing power is achieved by purchasing GPU hardware or renting cloud computing resources, while data is often an easily overlooked aspect.

[0003] A high-quality dataset allows the model trained by the algorithm to have excellent generalization performance and high recognition rate. For object detection and recognition tasks, a high-quality dataset can improve the model's recognition performance and scene adaptability without changing the algorithm.

[0004] In the context of flood relief, the inventors of this application discovered the following difficulties and technical problems in the prior art during the implementation of this invention:

[0005] First, there's the issue of data sources. Nowadays, many websites technically block web crawlers to protect data and privacy, rendering crawling tools unusable. Second, there's the issue of the number of data samples. Although there are many images of flood relief scenes, there are duplicate images or images that are not suitable for annotation, making dataset creation very difficult and inefficient. Summary of the Invention

[0006] This invention provides a method and system for creating deep learning datasets for flood relief scenarios, which solves or at least partially solves the technical problems of difficulty and low efficiency in dataset creation in the prior art.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0008] The first aspect provides a method for creating a deep learning dataset for flood relief scenarios, including:

[0009] Using webpage element extraction technology to collect image data in flood relief scenarios;

[0010] A pre-trained target detection model was used to detect the collected image data, which was divided into images of people falling into the water and other images, and saved to the positive sample folder and the flood scene folder respectively.

[0011] Preprocess the images of people who have fallen into the water contained in the positive sample folder;

[0012] The method of mean hashing and structural similarity is used to deduplicate preprocessed images of people who have fallen into the water. Specifically, the images are resized to 500*500 pixels, grayscale is applied, and the average grayscale value is calculated. Based on the relationship between the grayscale value of each pixel and the average grayscale value, a hash value for each pixel is obtained, and the image fingerprint is further derived. The Hamming distance between two images is calculated based on their fingerprints. If the Hamming distance is greater than a threshold, the two images are considered similar, and the image to be removed is determined based on their resolution. An improved structural similarity function is used to perform pairwise calculations on the remaining images. Based on the relationship between the structural similarity threshold and the calculated function value, as well as the resolution, the image to be deleted is determined. In the improved structural similarity function, the exponent of the brightness contrast function is set to 0.5, the exponent of the contrast contrast function is set to 1, and the exponent of the structural similarity function is set to 1.5.

[0013] Images of people who fell into the water and kept in the positive sample folder were used as positive samples, while images in the flood scene folder were used as negative samples. The positive and negative samples were labeled using a multi-label information fusion method.

[0014] The labeled samples are processed using image data augmentation to obtain the final deep learning dataset.

[0015] In one implementation, after using a pre-trained target detection model to detect the collected image data and classify it into images of people falling into the water and other images, the method further includes:

[0016] Check and confirm the images in the positive sample folder.

[0017] In one implementation, preprocessing is performed on the images of people who have fallen into the water contained in the positive sample folder, including:

[0018] Sort the images in the positive sample folder according to their resolution.

[0019] The system sequentially checks if there are black borders on the edges of the sorted images. If black borders are found, the corresponding images are moved to the folder where black borders need to be removed.

[0020] Remove black borders from the images in the folder containing the black border removal tool.

[0021] Character detection is performed on the images kept in the positive sample folder and the images after black border removal in the folder to be removed. If the number of characters in the image exceeds the character count threshold, the corresponding image is cut and moved to the folder to be removed.

[0022] Remove captions from the images in the folder containing the captions to be removed;

[0023] Cut the image after removing the subtitles to the positive sample folder.

[0024] In one implementation, the structural similarity threshold includes a first threshold and a second threshold, with the second threshold being greater than the first threshold. The calculated function value includes a first function value and a second function value. Based on the relationship between the structural similarity threshold and the calculated function value, and the resolution, it is determined which image to delete, including:

[0025] If the calculated value of the first function is greater than the first threshold, then the images with lower resolution are cropped to the positive sample-temporary folder;

[0026] An improved structural comparison function is used to calculate the images in the positive sample-temporary folder pairwise. The calculated second function value is then checked against a second threshold. If the second function value is greater than the threshold, both images corresponding to the second function value are deleted. Otherwise, the two images corresponding to the second function value are copied to the positive sample folder.

[0027] In one implementation, positive and negative samples are labeled using a multi-label information fusion method, including:

[0028] Difficult targets in images from both positive and negative samples are further categorized into two attributes: occlusion and blurring. Difficult targets are those with complex annotations.

[0029] Based on the same inventive concept, a second aspect of this invention provides a system for creating deep learning datasets for flood relief scenarios, comprising:

[0030] The image data acquisition module is used to collect image data in flood relief scenarios using web page element acquisition technology.

[0031] The image detection module is used to detect the collected image data using a pre-trained object detection model. The images are divided into images of people falling into the water and other images, which are saved to the positive sample folder and the flood scene folder, respectively.

[0032] The preprocessing module is used to preprocess the images of people who have fallen into the water contained in the positive sample folder;

[0033] The method of mean hashing and structural similarity is used to deduplicate preprocessed images of people who have fallen into the water. Specifically, the images are resized to 500*500 pixels, grayscale is applied, and the average grayscale value is calculated. Based on the relationship between the grayscale value of each pixel and the average grayscale value, a hash value for each pixel is obtained, and the image fingerprint is further derived. The Hamming distance between two images is calculated based on their fingerprints. If the Hamming distance is greater than a threshold, the two images are considered similar, and the image to be removed is determined based on their resolution. An improved structural similarity function is used to perform pairwise calculations on the remaining images. Based on the relationship between the structural similarity threshold and the calculated function value, as well as the resolution, the image to be deleted is determined. In the improved structural similarity function, the exponent of the brightness contrast function is set to 0.5, the exponent of the contrast contrast function is set to 1, and the exponent of the structural similarity function is set to 1.5.

[0034] Images of people who fell into the water and kept in the positive sample folder were used as positive samples, while images in the flood scene folder were used as negative samples. The positive and negative samples were labeled using a multi-label information fusion method.

[0035] The labeled samples are processed using image data augmentation to obtain the final deep learning dataset.

[0036] Based on the same inventive concept, a third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0037] Compared with the prior art, the advantages and beneficial technical effects of the present invention are as follows:

[0038] The method of this invention uses webpage element acquisition technology to collect image data in flood rescue scenarios, which avoids the limitations of using web crawlers to obtain data. A pre-trained object detection model is used to perform preliminary detection on the collected image data. Then, the images of people who have fallen into the water contained in the positive sample folder are preprocessed. Furthermore, the preprocessed images of people who have fallen into the water are deduplicated using mean hashing and structural similarity methods, thereby removing similar images and improving the quality of the samples. Positive and negative samples are then labeled using a multi-label information fusion method. Image data augmentation is used to process the labeled samples to obtain the final deep learning dataset. The above method can quickly create a deep learning dataset for flood rescue scenarios. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart of image collection in an embodiment of the present invention;

[0041] Figure 2 This is a flowchart of image preprocessing in an embodiment of the present invention;

[0042] Figure 3 This is a flowchart illustrating the process of deduplicating preprocessed images of people who have fallen into the water using mean hashing and structural similarity methods in this embodiment of the invention.

[0043] Figure 4 This is a schematic diagram illustrating the tag attribute setting method based on multi-tag information fusion in an embodiment of the present invention;

[0044] Figure 5 This is a schematic diagram of image data augmentation processing in an embodiment of the present invention. Detailed Implementation

[0045] This invention discloses a method and system for creating deep learning datasets for flood relief scenarios, in order to improve the technical problems of difficulty and low efficiency in dataset creation in the prior art.

[0046] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] Example 1

[0048] This invention provides a method for creating a deep learning dataset for flood relief scenarios, comprising:

[0049] Using webpage element extraction technology to collect image data in flood relief scenarios;

[0050] A pre-trained target detection model was used to detect the collected image data, which was divided into images of people falling into the water and other images, and saved to the positive sample folder and the flood scene folder respectively.

[0051] Preprocess the images of people who have fallen into the water contained in the positive sample folder;

[0052] The method of mean hashing and structural similarity is used to deduplicate preprocessed images of people who have fallen into the water. Specifically, the images are resized to 500*500 pixels, grayscale is applied, and the average grayscale value is calculated. Based on the relationship between the grayscale value of each pixel and the average grayscale value, a hash value for each pixel is obtained, and the image fingerprint is further derived. The Hamming distance between two images is calculated based on their fingerprints. If the Hamming distance is greater than a threshold, the two images are considered similar, and the image to be removed is determined based on their resolution. An improved structural similarity function is used to perform pairwise calculations on the remaining images. Based on the relationship between the structural similarity threshold and the calculated function value, as well as the resolution, the image to be deleted is determined. In the improved structural similarity function, the exponent of the brightness contrast function is set to 0.5, the exponent of the contrast contrast function is set to 1, and the exponent of the structural similarity function is set to 1.5.

[0053] Images of people who fell into the water and kept in the positive sample folder were used as positive samples, while images in the flood scene folder were used as negative samples. The positive and negative samples were labeled using a multi-label information fusion method.

[0054] The labeled samples are processed using image data augmentation to obtain the final deep learning dataset.

[0055] In the specific implementation process, when creating the flood relief scenario dataset, the following difficulties and technical problems were found with the existing methods:

[0056] 1. The sample size for annotation should be greater than 10,000 images. Annotating 10,000 images first requires finding more than 10,000 images that match the specific scenario. In practice, it was found that the number of images matching the specific scenario is far greater than 10,000 (approximately 20,000), because many images matching the specific scenario are not suitable for annotation, or the annotations are not applicable to sample training. In the process of searching for these 20,000 images, it was found that they must not only match the specific rescue environment such as floods, rivers, and lakes, but also show people falling into the water and needing rescue. Searching through various search engines using search terms revealed that very few images simultaneously met all the criteria; each search yielded only a few dozen images.

[0057] 2. Download Issues. Initially, a web crawler was prepared to scrape data from the internet. However, many websites nowadays implement technical restrictions on web crawlers to protect data and privacy, rendering these tools unusable.

[0058] 3. Image Duplicate Issues. During the collection of the first 200 images, we encountered some duplicate images, though the number was relatively small. However, in subsequent data collection, as the image data volume increased, the number of duplicate images also increased non-linearly, reaching a peak of 20% in one set of images. The real challenge wasn't identical images, but rather similar ones. Completely identical images were rare; most were similar. Some of these images had slight color differences due to website compression, some had been cropped due to various online channels, and some were even taken from similar angles by reporters of the same scene and the same person falling into the water. In these cases, commonly used file comparison and image comparison software became ineffective.

[0059] 4. Image preprocessing issues. Most images come from news reports and videos, and some come from short video platforms such as TikTok. Therefore, the images captured contain watermarks, subtitles, and even large black borders. These are all elements that should not be present in the scene. If these elements are not removed, it will greatly affect the accuracy of the model trained by the algorithm.

[0060] 5. Multi-person collaboration issues. To obtain 10,000 labeled data images, we searched and downloaded more than 120,000 images and 441 videos. These massive amounts of data images required manual screening and labeling. Due to the enormous amount of data, during the image screening process, some people were very strict, keeping only 20 out of 1,000 images, while others selected more than 100, making the judgment criteria very vague. The labeling process was also different, with different criteria for drawing boxes for the targets of people who fell into the water.

[0061] This makes the creation of datasets difficult and slow. Therefore, this invention provides a method for creating deep learning datasets for flood relief scenarios.

[0062] Research revealed that web scraping tools struggle to bypass website technical barriers, and different websites require different methods of bypassing them, making it too time-consuming and labor-intensive. Furthermore, analysis of the data obtained through web scraping showed that most of the images did not depict the desired scenario of "people falling into the water and awaiting rescue in a flood," containing many irrelevant or only vaguely related images that did not match the actual situation. This significantly increased the workload for subsequent image filtering. Therefore, the web scraping method was abandoned for image acquisition.

[0063] This invention uses webpage element acquisition technology for image downloading, and considering the issue of multi-person collaboration, it uses Tencent Online Spreadsheet Management to implement mutual exclusion of search terms for collaborative work. Specific operations are as follows: Figure 1 As shown, it includes the following steps:

[0064] (1) Each person is assigned to a different country server.

[0065] (2) At the same time, considering the issue of repetition rate, each person conducted a separate search using detailed location as the search term.

[0066] Someone is using the Edge browser and the Bing search engine on a Chinese local server to search for floods in Jiangxi Province. To avoid duplicates, they add a specific location after "Jiangxi," such as "Shangrao, Jiangxi." To further avoid repetitive results, they scroll through several pages to find similar content and vary the final search terms, using terms like "waterlogging," "flood," "flood," and "drowning" in combination. If they use "Shangrao floods in Jiangxi," "Jiujiang floods in Jiangxi," etc., the initial search results show different images, but the later images become increasingly similar, converging to represent all flood disasters that have occurred in Jiangxi.

[0067] (3) Launch the IA Image Assistant tool (a browser plugin), search for relevant images on the webpage, and keep scrolling down until a large number of irrelevant images appear. Click "Extract Images from This Page" in the IA Image Assistant menu, and set the resolution, image format conversion, and save path folder. Then click the "Select All" button, and then click the "Download Selected" button to save the images to the preset folder named "Flood Scene".

[0068] (4) The pre-trained target detection model uses the EasyDL model to identify people in the folder, and then cut (different from copying) the images of people that are identified to another folder named "positive samples" for manual screening.

[0069] After filtering, the "positive samples" folder contains images of people falling into the water, while the "flood scenes" folder contains other images that were filtered out.

[0070] In one implementation, after using a pre-trained target detection model to detect the collected image data and classify it into images of people falling into the water and other images, the method further includes:

[0071] Check and confirm the images in the positive sample folder.

[0072] Specifically, the "positive samples" folder should theoretically contain all scene images of "people falling into the water and waiting for rescue in a flood scenario," but in reality, there are still quite a few errors in it.

[0073] Typical errors include: ① The person is not actually in the water, but standing on the shore or near the water's edge; ② The scene is not a flood scene, but a scene after a flood, with a lot of yellow mud; ③ The "Flood Scene" folder contains many unidentified people. One reason is that the person is too small, for example, only the head of a person is visible in the distance; another is that the person's head and face are not above the water, only the limbs are on the surface; and yet another is severe occlusion, obscuring most of the person. All of these situations will cause the EasyDL model for object detection-person to fail, resulting in the failure to identify people who have fallen into the water. Besides these typical errors, there are also very rare cases that cause identification errors, therefore manual review is required for screening.

[0074] After inspection and confirmation, the "positive sample" files all contained scene images of "people falling into the water and waiting for rescue in a flood scenario".

[0075] In one implementation, preprocessing is performed on the images of people who have fallen into the water contained in the positive sample folder, including:

[0076] Sort the images in the positive sample folder according to their resolution.

[0077] The system sequentially checks if there are black borders on the edges of the sorted images. If black borders are found, the corresponding images are moved to the folder where black borders need to be removed.

[0078] Remove black borders from the images in the folder containing the black border removal tool.

[0079] Character detection is performed on the images kept in the positive sample folder and the images after black border removal in the folder to be removed. If the number of characters in the image exceeds the character count threshold, the corresponding image is cut and moved to the folder to be removed.

[0080] Remove captions from the images in the folder containing the captions to be removed;

[0081] Cut the image after removing the subtitles to the positive sample folder.

[0082] Please see Figure 2 This is a flowchart of image preprocessing in an embodiment of the present invention.

[0083] In the specific implementation process, the workflow is as follows:

[0084] (1) Sort the images in the positive sample folder according to their resolution. This is because many similar images come from the same news article and have similar resolutions.

[0085] (2) Read the top 10 rows of pixels, bottom 10 rows of pixels, left 10 columns, and right 10 columns of the image respectively.

[0086] Perform pixel grayscale averaging on the four distinct edges. If the grayscale value is <= 10, it is considered close to black and a black edge exists. Cut (i.e., move) this type of image to another folder, named "Black Edges to be Removed".

[0087] (3) Use the software Photoshop to remove black borders from the images in the folder to be removed in batches;

[0088] (4) Start the Baidu Paddle-OCR tool and scan the images stored in the positive sample folder and the images after black border removal in the folder to be removed. If more than 10 characters are found, cut (move) the images to another folder and name the other folder "subtitles to be removed".

[0089] (5) Manually remove subtitles from the images in the folder containing the subtitles to be removed;

[0090] (6) Cut the image after removing the subtitles to the positive sample folder.

[0091] (7) Manually review the images to avoid errors. Since the images were sorted by resolution and numbered in step (1), refer to the image numbers in the "Black Borders to be Removed" and "Subtitles to be Removed" folders. Manually check the images with similar numbers in the "Positive Samples" folder (where black borders and subtitles were removed) to quickly identify any errors. Correct any individual errors.

[0092] In the "Black Borders to be Removed" and "Subtitles to be Removed" folders, images from the same source are mostly grouped together. Using software like "PhotoMagic" for batch cropping can effectively remove black borders and subtitles. However, watermarks are more difficult to remove. Images with watermarks can be manually assessed based on their overall impact on the image's content. If the impact is minor, they are left untreated; if the impact is significant, they are manually removed using Photoshop.

[0093] After the above processing steps, all images in the "Positive Sample" folder (with black border images and subtitle images removed) are also copied to the "Positive Sample 2" folder.

[0094] In one implementation, the structural similarity threshold includes a first threshold and a second threshold, with the second threshold being greater than the first threshold. The calculated function value includes a first function value and a second function value. Based on the relationship between the structural similarity threshold and the calculated function value, and the resolution, it is determined which image to delete, including:

[0095] If the calculated first function value is greater than the first threshold, then cut the picture with the smaller resolution into the positive sample - temporary folder;

[0096] Calculate pairwise the pictures in the positive sample - temporary folder using an improved structure comparison function, and determine whether the calculated second function value is greater than the second threshold. If it is greater, delete both pictures corresponding to the second function value; otherwise, copy the two pictures corresponding to the second function value to the positive sample folder.

[0097] In the specific implementation process, the present invention adopts an algorithm that mixes mean hash and ssim. As Figure 3 shown, the implementation steps are as follows:

[0098] (1) Grayscale the image. To prevent the influence of color, first grayscale the color image.

[0099] (2) Calculate the average grayscale value of the image, assume it is a, and then refer to the mean hash algorithm aHash to calculate the fingerprint according to the pixel mean (initialize the aHash of the input picture as "". Traverse each pixel of the matrix G row by row from left to right. If the element G(i,j) in the i - th row and j - th column >= a, then aHash += "1"; if the element G(i,j) in the i - th row and j - th column < a, then aHash += "0"). The difference between the above operation method and the original mean hash algorithm is that instead of unifying the picture size to 8*8, 64 pixels, resize the original picture to 500*500 (because the long side of the finally labeled image size is based on 500 pixels), and directly calculate the aHash value for 500*500, 250,000 pixels. This method is slower in calculation speed, but with a high - performance server as the computing power basis, and this method has higher accuracy compared to the original mean hash algorithm.

[0100] (3) After obtaining the aHash value of the picture, compare the Hamming distance of the aHash values of two pictures, and normalize it to the interval [0,1]. Set the threshold to 85%, that is, 0.85. Assume that the normalized Hamming distance between pictures A and B is greater than 0.85, then it is determined to be similar. Look up the original image resolution. If the resolution of A is greater than B, retain the picture A with the larger resolution and剔除 the picture B with the smaller resolution.

[0101] (4) For the pictures retained in step (3), perform ssim algorithm comparison. We have improved the ssim algorithm. The ssim algorithm measures from three aspects: brightness, contrast, and structure:

[0102] Brightness comparison function:

[0103]

[0104] Contrast comparison function:

[0105]

[0106] Structure comparison function:

[0107]

[0108] Combining the three functions above, we obtain the SSIM exponential function:

[0109]

[0110] Where, μ x μ is the average value of x. y Let y be the average value. Let x be the variance. Let σ be the variance of y. xy Let x be the covariance of x and y, and c1 be a constant with a value of (k1L). 2 c2 is a constant with a value of (k2L). 2 c3 is a constant, defined as c2 / 2, and L is the dynamic range of pixel values, typically taken as (2). bits per pixel -1), k1 and k2 are generally taken as 0.01 and 0.03 respectively.

[0111] In conventional methods, the values ​​are as follows: α = β = γ = 1, C3 = C2 / 2. Through extensive research and practice, the inventors discovered that in step (3), the mean hash algorithm already considers brightness. The remaining images are already quite different in brightness, and the brightness comparison function can no longer reflect the differences. However, for images with similar structures, the structure comparison function can effectively highlight the structural similarity. Therefore, this invention modifies the above parameters to α = 0.5, β = 1, γ = 1.5, and C3 = C2 / 2.

[0112] (5) Set the first threshold of SSIM to 0.3. Remove any images with a resolution greater than 0.3, cut the images with lower resolution to the folder "Positive Samples - Temporary Storage", and keep the images with the highest resolution in the folder "Positive Samples - OK". Keep the images with a resolution less than 0.3 and copy them to the "Positive Samples - OK" folder. Ensure that the images in the "Positive Samples - OK" folder are absolutely dissimilar.

[0113] (6) In the “Positive Samples - Temporary Storage” folder, there are images with an SSIM value greater than 0.3 that have been removed. Some of these images were mistakenly deleted. We re-execute step (4) in this folder and adjust the second threshold to 0.7. That is, images with a value greater than 0.7 are considered similar images and are completely deleted. Images with a value less than 0.7 are considered dissimilar images that were mistakenly deleted and are copied back to the “Positive Samples - OK” folder.

[0114] After completing steps (1)-(6) above, we obtained the "Positive Samples - Confirm" folder, in which the images are basically non-repeating. A very few similar images were manually deleted during the subsequent annotation process. The similarity rate was 0.46%, meaning that out of 10,000 images, only 46 were similar.

[0115] In one implementation, positive and negative samples are labeled using a multi-label information fusion method, including:

[0116] Difficult targets in images from both positive and negative samples are further categorized into two attributes: occlusion and blurring. Difficult targets are those with complex annotations.

[0117] Specifically, after obtaining the sample images, the next step is to annotate them. There are two main challenges in image annotation: one is to accurately draw the bounding box, and the other is to accurately assess the situation of a person falling into the water and waiting for rescue in a specific scenario such as a flood.

[0118] 1. For common targets, or images that are easy to annotate, precise bounding boxes are relatively easy to draw. However, for certain special cases, such as in some images where the feet in the background belong to a person who has fallen into the water, but are separated by water, they cannot be included in the bounding box; only the part of the person above the water can be precisely bounded. In some images, although the lower limbs of a person are visible, they cannot be included due to refraction and distortion in the water; only the part of the person above the water can be precisely bounded. Considering that flood scenes have poor water quality and are opaque, while river, lake, and sea scenes have better water quality and are more transparent, to ensure a unified standard for collaborative annotation by multiple people, we define that only the part above the water surface should be annotated.

[0119] 2. Precise scene assessment. We only mark targets in dangerous situations with bounding boxes. If the people in the image are not in a dangerous situation, the image is excluded.

[0120] In the specific implementation process, two sets of annotation software were developed: one is "image annotation software based on multi-label information fusion", and the other is "image data deduplication-amplification multi-information annotation software".

[0121] 1. The innovative aspects of the "image annotation software based on multi-label information fusion" are:

[0122] (1) This software refines the labels of targets that are difficult to annotate in the image, so that a target can have multiple label information. This enables the subsequent training algorithm to differentiate and process different types of targets.

[0123] For images with densely packed people, obstructions, and distant figures that are particularly blurry, classic labelimg software only offers a single "difficulty" metric for labeling challenging targets. However, in flood relief scenarios, this difficulty needs to be further subdivided into two distinct types: occlusion and blur. This embodiment refines the labeling method by further subdividing the difficulty type into "occlusion" and "definition," thus adding these two new label attributes, which are subordinate to the "difficulty" attribute. Figure 4 As shown.

[0124] (2) The software integrates the function of generating visualization images of annotation results. After a large number of images are annotated, visualization effects can be generated in batches and checked in batches in the folder.

[0125] (3) The software can generate an XML file corresponding to each sample, or it can merge multiple XML information into a single JSON format document in batches, which makes it convenient for the training algorithm to be called in various forms later.

[0126] (4) The PyQt5 GUI suite is developed using Python and can be used as open source software to make flexible adjustments to the deep learning algorithms in the future.

[0127] The "Image Data Deduplication-Amplification Multivariate Information Annotation Software" adds image deduplication and dataset expansion functions to the first software, with the following innovations:

[0128] (1) Added deduplication function.

[0129] (2) Image data augmentation functionality has been added, going beyond traditional color dithering, rotation, and grayscale conversion. Crucially, an HSV mask function has been added. The sample image undergoes HSV color model conversion, H channel features are extracted, and a mask is created to filter out background interference. This mask is then saved as a new sample image, thus augmenting the sample data. Figure 5 As shown.

[0130] The image on the left is the original image. After HSV color model conversion, the H channel features are extracted. The H value of the target object is used as the feature value. Then, with the feature value as the center, there are 10 gray values ​​to the left and right. That is, assuming the feature value is 'a', the range is [a-10, a+10]. Gray values ​​outside the range are used as background gray values ​​to create a black background mask. The mask is then ANDed with the original image on the left to obtain the image on the right.

[0131] This method of sample labeling can reduce interference from training samples and improve sample quality.

[0132] Example 2

[0133] Based on the same inventive concept, this embodiment provides a system for creating a deep learning dataset for flood relief scenarios, including:

[0134] The image data acquisition module is used to collect image data in flood relief scenarios using web page element acquisition technology.

[0135] The image detection module is used to detect the collected image data using a pre-trained object detection model. The images are divided into images of people falling into the water and other images, which are saved to the positive sample folder and the flood scene folder, respectively.

[0136] The preprocessing module is used to preprocess the images of people who have fallen into the water contained in the positive sample folder;

[0137] The method of mean hashing and structural similarity is used to deduplicate preprocessed images of people who have fallen into the water. Specifically, the images are resized to 500*500 pixels, grayscale is applied, and the average grayscale value is calculated. Based on the relationship between the grayscale value of each pixel and the average grayscale value, a hash value for each pixel is obtained, and the image fingerprint is further derived. The Hamming distance between two images is calculated based on their fingerprints. If the Hamming distance is greater than a threshold, the two images are considered similar, and the image to be removed is determined based on their resolution. An improved structural similarity function is used to perform pairwise calculations on the remaining images. Based on the relationship between the structural similarity threshold and the calculated function value, as well as the resolution, the image to be deleted is determined. In the improved structural similarity function, the exponent of the brightness contrast function is set to 0.5, the exponent of the contrast contrast function is set to 1, and the exponent of the structural similarity function is set to 1.5.

[0138] Images of people who fell into the water and kept in the positive sample folder were used as positive samples, while images in the flood scene folder were used as negative samples. The positive and negative samples were labeled using a multi-label information fusion method.

[0139] The labeled samples are processed using image data augmentation to obtain the final deep learning dataset.

[0140] Since the system described in Embodiment 2 of this invention is the system used to implement the method for creating a deep learning dataset in a flood rescue scenario in Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this system based on the method described in Embodiment 1 of this invention, and therefore will not be described again here. All systems used in the method of Embodiment 1 of this invention fall within the scope of protection of this invention.

[0141] Example 3

[0142] Based on the same inventive concept, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the method described in Embodiment 1.

[0143] Since the computer-readable storage medium described in Embodiment 3 of this invention is the same computer-readable storage medium used in implementing the method for creating a deep learning dataset in a flood relief scenario in Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this computer-readable storage medium based on the method described in Embodiment 1 of this invention, and therefore will not be repeated here. All computer-readable storage media used in the method of Embodiment 1 of this invention fall within the scope of protection of this invention.

[0144] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0145] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0146] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0147] Obviously, those skilled in the art can make various modifications and variations to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if these modifications and variations to the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention also intends to include these modifications and variations.

Claims

1. A method for making a deep learning dataset in a flood rescue scene, characterized in that, The method comprises the following steps: Collecting picture data in a flood rescue scene by using a webpage element acquisition technology; Detecting the collected picture data by using a pre-trained target detection model, and dividing the picture data into pictures of fallen personnel and other pictures, and saving the pictures into a positive sample folder and a flood scene folder respectively; Preprocessing the pictures of fallen personnel contained in the positive sample folder; Removing duplicates from the preprocessed pictures of fallen personnel by using a mean hash and a structural similarity method, specifically including: adjusting the size of the pictures of fallen personnel to 500*500, performing grayscale processing on the pictures with the adjusted size, calculating the average gray value of the images, obtaining the hash value of each pixel according to the relationship between the gray value of each pixel and the average gray value of the image, and further obtaining the fingerprint of the picture; according to the size of the fingerprints of two pictures, calculating the Hamming distance of the two pictures, if the Hamming distance of the two pictures is greater than a distance threshold, it is determined that the two pictures are similar, and according to the resolution of the two pictures, it is determined which picture to delete; calculating the improved structural contrast function for the remaining pictures two by two, and according to the relationship between the structural similarity threshold and the calculated function value and the size of the resolution, it is determined which picture to delete, wherein in the improved structural contrast function, the exponent of the brightness contrast function is set to 0.5, the exponent of the contrast ratio contrast function is set to 1, and the exponent of the structural contrast function is set to 1.5; Labeling the positive samples in the positive sample folder and the pictures contained in the flood scene folder as negative samples by using a multi-label information fusion based method; Processing the labeled samples by using an image data augmentation method to obtain a final deep learning data set.

2. The method of claim 1, wherein, After detecting the collected picture data by using a pre-trained target detection model and dividing the picture data into pictures of fallen personnel and other pictures, the method further comprises: Checking and confirming the pictures in the positive sample folder. 3.The method of claim 1, wherein, The preprocessing of the pictures of fallen personnel contained in the positive sample folder comprises: Sorting the pictures contained in the positive sample folder according to the resolution size; Detecting whether the edges of the sorted pictures exist black edges in turn, if the edges exist black edges, moving the corresponding pictures to a black edge removal folder; Removing black edges from the pictures in the black edge removal folder; Detecting characters of the pictures remaining in the positive sample folder and the pictures in the black edge removal folder after the black edge removal, if the number of characters of the pictures exceeds a character number threshold, cutting the corresponding pictures to a subtitle removal folder; Removing subtitles from the pictures in the subtitle removal folder; Cutting the pictures after the subtitle removal to the positive sample folder.

4. The method of claim 1, wherein, The structural similarity threshold comprises a first threshold and a second threshold, and the second threshold is greater than the first threshold, the calculated function value comprises a first function value and a second function value, and according to the relationship between the structural similarity threshold and the calculated function value and the size of the resolution, it is determined which picture to delete, comprising: If the first function value is greater than the first threshold, the picture with smaller resolution is cut to a positive sample-temporary folder; The improved structural contrast function is used to calculate the pictures in the positive sample-temporary folder two by two, and it is judged whether the second function value obtained by calculation is greater than the second threshold value, if greater, the two pictures corresponding to the second function value are deleted, otherwise, the two pictures corresponding to the second function value are copied to the positive sample folder. 5.The method of claim 1, wherein, The positive samples and negative samples are labeled by using a multi-label information fusion based manner, including: The difficult targets in the pictures in the positive samples and negative samples are further divided into two attributes of occlusion and blur, wherein the difficult target is a target with complex labeling condition. 6.A system for making a deep learning dataset in a flood rescue scene, characterized in that, Including: The picture data acquisition module is configured to collect picture data in a flood rescue scene by using a web element acquisition technology; The picture detection module is configured to detect the collected picture data by using a pre-trained target detection model, and divide the picture data into fallen person pictures and other pictures, and save the fallen person pictures and the other pictures into a positive sample folder and a flood scene folder respectively; The preprocessing module is configured to preprocess the fallen person pictures contained in the positive sample folder; The mean hash and structural similarity method is used to remove the duplicates of the preprocessed fallen person pictures, including: adjusting the size of the fallen person pictures to 500*500, performing grayscale processing on the size-adjusted pictures, calculating the image grayscale average value, obtaining the hash value of each pixel according to the relationship between the grayscale value of each pixel and the image grayscale average value, and further obtaining the fingerprint of the picture; according to the size of the fingerprints of two pictures, the Hamming distance of the two pictures is calculated, if the Hamming distance of the two pictures is greater than the distance threshold, it is determined that the two pictures are similar, and according to the resolution of the two pictures, it is determined which picture to delete; the improved structural contrast function is used to calculate the retained pictures two by two, according to the relationship between the structural similarity threshold and the calculated function value and the size of the resolution, it is determined which picture to delete, wherein in the improved structural contrast function, the exponent of the brightness contrast function is set to 0.5, the exponent of the contrast contrast function is set to 1, and the exponent of the structural contrast function is set to 1.5; The fallen person pictures retained in the positive sample folder are used as positive samples, the pictures contained in the flood scene folder are used as negative samples, and the positive samples and negative samples are labeled by using a multi-label information fusion based manner; The labeled samples are processed by using an image data augmentation manner to obtain a final deep learning data set.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method of any one of claims 1 to 5.

Images