A high-altitude dangerous object detection method, device and storage medium

Abstract

The application discloses a high-altitude dangerous object detection method and device and a storage medium, relates to the technical field of high-altitude dangerous object detection, and comprises the following steps: acquiring a high-altitude dangerous object detection model and a building group image of a community to be detected, wherein the high-altitude dangerous object detection model comprises a backbone network, a classifier and a detector; performing feature extraction on the building group image through the backbone network to obtain a target feature map, wherein the backbone network comprises a CBM component set and a Res component set, the CBM component set is used for performing feature extraction on the building group image and conveying an extraction result to the Res component set, and the Res component set is used for optimizing the extraction result and outputting the target feature map; and performing classification detection on the target feature map through the classifier and the detector to obtain a dangerous object category corresponding to the building group image. The application is beneficial to improving community safety.

Description

Technical Field

[0001] This application relates to the field of high-altitude hazardous materials detection technology, and in particular to a method, equipment and storage medium for high-altitude hazardous materials detection. Background Technology

[0002] With the development of urbanization in my country, high-rise buildings are springing up in both town and city communities. Many residents on high floors place items such as air conditioners and potted plants on their balconies or windows. However, these items all pose a risk of falling from heights and are therefore classified as "dangerous objects at heights," posing a significant safety hazard. Currently, community workers typically conduct inspections of dangerous objects at heights through door-to-door visits, which is extremely labor-intensive and not comprehensive enough to effectively improve community safety. Summary of the Invention

[0003] This application aims to address at least one of the technical problems existing in the prior art. To this end, this application proposes a method, device, and storage medium for detecting hazardous materials at high altitudes, which helps improve community safety.

[0004] The first aspect of this application provides a method for detecting hazardous materials at high altitudes, including:

[0005] The system acquires images of a high-altitude hazardous material detection model and buildings in the community to be detected. The high-altitude hazardous material detection model includes a backbone network, a classifier, and a detector.

[0006] The backbone network extracts features from the building complex image to obtain a target feature map. The backbone network includes a CBM component set and a Res component set. The CBM component set is used to extract features from the building complex image and send the extraction results to the Res component set. The Res component set is used to optimize the extraction results and output the target feature map.

[0007] The target feature map is classified and detected by classifier and detector to obtain the hazardous object category corresponding to the building complex image.

[0008] The first aspect of this application provides a model training method, which has at least the following beneficial effects: This application collects images of buildings in the community to be detected, and then combines a backbone network, a classifier, and a detector to identify hazardous objects in the building images. Compared with door-to-door visits and investigations, this application helps to reduce workload and improve investigation accuracy, thereby helping to improve community safety.

[0009] According to some embodiments of the first aspect of this application, the CBM component set includes a first CBM component and a second CBM component, the Res component set includes a plurality of Res components, and the step of extracting features from the building cluster image through the backbone network to obtain a target feature map includes:

[0010] The first CBM component is used to extract features from the training samples to obtain a first output feature map;

[0011] The second output feature map is obtained by downsampling the first output feature map using the second CBM component.

[0012] By using a preset number of Res components to extract features from the second output feature map, several target feature maps of different sizes are obtained.

[0013] According to some embodiments of the first aspect of this application, the header of the classifier and detector includes an SPP component composed of an average pooling algorithm, and the method further includes:

[0014] The target feature map is upsampled using the SPP component, and the upsampled result is then classified and detected by a classifier and a detector to obtain the hazardous material category corresponding to the building complex image.

[0015] According to some embodiments of the first aspect of this application, the high-altitude hazardous material detection model is obtained through the following training methods, including:

[0016] Obtain a sample set of building cluster images;

[0017] The high-altitude hazardous object detection model is iteratively trained based on the building cluster image sample set. Specifically, the high-altitude hazardous object detection model classifies and detects a building cluster image sample from the set to obtain prediction box parameters. The loss function is updated based on the difference between the square root of the prediction box parameters and the square root of the corresponding label box parameters of the building cluster image sample. The iteration ends when the loss function meets a preset condition.

[0018] According to some embodiments of the first aspect of this application, after the model training is completed, the method further includes:

[0019] Collect a set of images of the buildings in the target community;

[0020] The image set of buildings in the target community is input into the trained high-altitude hazardous object detection model, and the model parameters are fine-tuned.

[0021] According to some embodiments of the first aspect of this application, the loss function expression is as follows:

[0022]

[0023] , These are all parameters of the prediction box. , These are all label box parameters. The preset loss coefficient, , , is the weighting constant for each loss term.

[0024] According to some embodiments of the first aspect of this application, the step of classifying and detecting the target feature map using a classifier and a detector to obtain the hazardous material category corresponding to the building complex image includes:

[0025] Detect whether the target feature image has the feature of hanging objects around the balcony security grille;

[0026] Detect whether the target feature map has the feature of placing an item on the edge of the balcony or above the balcony wall;

[0027] Detect whether the target feature map has features of window damage;

[0028] Detect whether the target feature map contains features indicating that the outdoor unit of the air conditioner has shifted;

[0029] The system detects whether the target feature map contains the feature of an advertising billboard installed on an exterior wall.

[0030] According to some embodiments of the first aspect of this application, after obtaining the hazardous materials category corresponding to the building complex image, the method further includes:

[0031] Risk levels are predicted based on the hazardous material categories corresponding to the building complex images, resulting in prediction data.

[0032] A second aspect of this application provides an electronic device, including:

[0033] At least one memory;

[0034] At least one processor;

[0035] At least one program;

[0036] The program is stored in the memory, and the processor executes at least one of the programs to implement the controller design method as described in any embodiment of the first aspect of this application.

[0037] A third aspect of this application provides a computer-readable storage medium storing computer-executable signals for performing a controller design method as described in the first aspect and any embodiment of this application. Attached Figure Description

[0038] Additional aspects and advantages of this application will become apparent and readily understood in conjunction with the following description of the embodiments, in which:

[0039] Figure 1 This is a schematic diagram of the structure of the high-altitude hazardous materials detection model provided in the embodiments of this application;

[0040] Figure 2 Provided for the embodiments of this application Figure 1 A schematic diagram of the backbone network structure;

[0041] Figure 3 This is a schematic diagram of the backbone network structure in the related technologies provided in the embodiments of this application;

[0042] Figure 4 A schematic diagram of the head structure of the classifier and detector provided in the embodiments of this application;

[0043] Figure 5 A flowchart of a high-altitude hazardous material detection method provided in the embodiments of this application;

[0044] Figure 6 A flowchart illustrating feature extraction of the building complex image by the backbone network provided in this embodiment of the application;

[0045] Figure 7 A flowchart of a detection method provided in another embodiment of this application;

[0046] Figure 8 A flowchart of a detection method provided in another embodiment of this application;

[0047] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0049] It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown in the flowchart. The terminology in the specification, claims, and the foregoing drawings is used to distinguish similar objects and is not necessarily used to describe a specific order or sequence.

[0050] In the description of this application, the use of "first" and "second" is for the purpose of distinguishing technical features only, and should not be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated or the order of the technical features indicated.

[0051] In the description of this application, unless otherwise expressly defined, terms such as "setup," "installation," and "connection" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this application in conjunction with the specific content of the technical solution.

[0052] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0053] In recent years, reports of deaths and injuries caused by falling objects from high-rise buildings have been frequent. With the development of urbanization in my country, high-rise buildings are springing up in both urban and rural communities. Many residents illegally place air conditioners and potted plants on their balconies or windows, causing significant problems for community workers. Currently, the investigation of dangerous objects from high-rise buildings relies mainly on manual monitoring and door-to-door visits by community workers to promote community harmony. This method greatly increases the workload of community workers and also leads to situations where residents conceal information from them.

[0054] Based on this, this application proposes a model training method, a detection method, an apparatus, and a storage medium, which help improve community security.

[0055] The embodiments of this application will be further described below with reference to the accompanying drawings.

[0056] Reference Figure 1 , Figure 1 The schematic diagram of the high-altitude hazardous material detection model provided in the embodiments of this application includes an input image dataset module, a grid and anchor box generation module, a backbone network feature extraction module, a classifier and detector module, a loss function update module, output prediction box information, confidence level and item type module.

[0057] Regarding the input image dataset module, the image dataset is obtained in the following ways:

[0058] (1) Sampling of building clusters in the community, sampling of individual buildings by manually controlling drones, planning the patrol flight route so that the drones can semi-automatically shoot and sample objects on balconies and windows of the community around the buildings, cleaning the collected video stream (e.g., deleting images with high similarity between adjacent frames), and resizing the images to a fixed size to form a high-quality image dataset.

[0059] (2) Using "web crawler" technology, we crawled images of balcony panels and balcony items on the Internet, saved them, and then cleaned up images with high similarity and junk data to form an image dataset.

[0060] It should be noted that during model training, the input to the model is an image dataset. However, to better explain the model training process, this application uses a single building complex image as an example to demonstrate the training method.

[0061] based on Figure 1 , refer to Figure 5 , Figure 5 The flowchart is for a model training method provided in the embodiments of this application, which includes, but is not limited to, the following steps 501 to 503.

[0062] Step 501: Obtain the high-altitude hazardous object detection model and the building cluster images of the community to be detected. The high-altitude hazardous object detection model includes a backbone network, a classifier, and a detector.

[0063] Step 502: Extract features from the building complex image using the backbone network to obtain the target feature map. The backbone network includes a CBM component set and a Res component set. The CBM component set is used to extract features from the building complex image and send the extraction results to the Res component set. The Res component set is used to optimize the extraction results and output the target feature map.

[0064] Step 503: Classify and detect the target feature map using a classifier and detector to obtain the hazardous material category corresponding to the building complex image.

[0065] In one embodiment, the CBM component set includes a first CBM component and a second CBM component, and the Res component set includes several Res components. Step 502: Extract features from the building cluster image through the backbone network to obtain the target feature map, including:

[0066] Step 5021: Extract features from the training samples using the first CBM component to obtain the first output feature map;

[0067] Step 5022: The first output feature map is downsampled using the second CBM component to obtain the second output feature map;

[0068] Step 5023: Extract features from the second output feature map using a preset number of Res components to obtain several target feature maps of different sizes.

[0069] Understandably, referring to Figure 2 , Figure 2The schematic diagram of the backbone network provided in this application embodiment includes a first CBM component (CBM1), a second CBM component (CBM2), a first Res component (Res1), a second Res component (Res2), a third Res component (Res8), a fourth Res component (Res8), and a fifth Res component (Res4) arranged from left to right. Each CBM component includes a Conv component, a BN component, and a Mish component. Each Res component includes a CBM component and a Concat component. It should be noted that in ResX components, X represents the number of components. For example, Res8 represents eight Res components.

[0070] It is understood that the target feature map includes a first target feature map, a second target feature map, and a third target feature map. Step 5023: Features are extracted from the second output feature map using a preset number of Res components to obtain several target feature maps of different sizes, including: obtaining the first target feature map using the first Res component, the second Res component, the third Res component, and the second output feature map; obtaining the second target feature map using the fourth Res component and the first target feature map; and obtaining the third target feature map using the fifth Res component and the second target feature map. (Refer to...) Figure 2 , Figure 2 The squares 1, 2, and 3 in the diagram represent the first target feature map, the second target feature map, and the third target feature map, respectively, and are combined with... Figure 4 As can be seen, the output of the backbone network in this application is the input of the classifier and the detector.

[0071] It should be further noted that the Conv component in this application mainly includes effective convolution and downsampling convolution. Effective convolution uses kernel settings that do not change the original input parameters for each convolution, while downsampling convolution uses a 3×3 kernel with a stride of 2. The main purpose of this setting is to improve the system model's ability to focus on image features after multiple learning and recombination of image features. Considering the diverse types of high-altitude objects in the community and the need for small target detection, a deeper network structure is required to extract image features. Therefore, the Conv component is designed with a total of 53 convolutional layers. The main purpose of using 53 convolutional layers is to mine the features of the input image more deeply through image convolution operations, thereby improving the classification and detection accuracy of this system.

[0072] It should be further noted that the main function of the Batch Normalization (BN) component in this application is to improve the stability of the system during model training. The BN component uses Batch Normalization to ensure that the statistical distribution of each batch of training datasets is the same, primarily focusing on statistically reconstructing the mean and variance of the training dataset. Let's assume the mean and variance of the training dataset are... and And assume a dataset of size N. Based on this assumption, this system will use formulas (1) to (4) to calculate BatchNormalization and obtain the final y value. It is worth noting that a constant is added to formula (3) to prevent the denominator from being zero. After formula (3), The distribution will be guaranteed to be at a uniform level, but directly using Using image features as input to the system model will result in the loss of image features, which will degrade the model's performance to some extent. To address this, the model uses two parameters. and To conduct A linear transformation is applied to approximate the distribution as closely as possible to the unnormalized state. These two parameters do not require manual intervention; rather, they are obtained through backpropagation, training the network along with the backbone parameters to find the optimal solution.

[0073]

[0074]

[0075]

[0076]

[0077] It should be noted that the main function of the Mish component in this application is to improve the convergence speed of the system model based on the simulation of sparse connections in neural networks, and to prevent the problem of gradient vanishing during backpropagation in model training.

[0078] It should be noted that the CBMx component in this application is divided into CBM1 and CBM2. CBM1 is mainly responsible for feature extraction, and the size of the output feature map is consistent with that of the input feature map; CBM2 is mainly responsible for the downsampling function of the system model, and the downsampling rate is 1 / 2, which means that the feature map size is halved.

[0079] It should be further explained that the Resx and Concat components in this application are based on a residual network design, ensuring that the input baseline is maintained while deepening the system backbone. In this system, based on the idea of ​​retaining more feature information, the Concat component deepens the feature map to prevent further feature loss. Specifically, Resx adds the values ​​at corresponding positions in the feature map without changing the number of channels. For this system, this structure will further extract features from the original image, improving detection accuracy. The main function of the Concat component is to superimpose the output from the Resx component with the original input, without adding values ​​at each corresponding position in the feature map. The aim is to retain as much feature information as possible using the number of channels before downsampling.

[0080] It should be added that, referring to Figure 3 The schematic diagram of the backbone network in the related technologies shown can be seen that the backbone network of this application does not use the CSPX component. Figure 3 As shown in the CSPX component structure, the CSPX component needs to be composed of Res components. Therefore, compared with related technologies, this application directly uses Res components as the main network components, which makes the structure more concise, reduces the amount of computation and parameters involved in the model operation, and helps to improve detection efficiency.

[0081] In one embodiment, the classifier and detector header includes an SPP component composed of an average pooling algorithm, which classifies and detects the target feature map. The method further includes upsampling the target feature map through the SPP component, and then classifying and detecting the upsampled result through the classifier and detector to obtain the hazardous category corresponding to the building complex image.

[0082] Understandably, to obtain more concentrated and strong image semantic feature information while preserving strong localization feature information, this system uses two feature pyramid structures: bottom-up and top-down. The top-down structure means gradually upsampling from a small feature map size to a larger size, while the bottom-up structure means gradually downsampling from a large feature map size to a smaller size. We will use FPN to represent the top-down feature pyramid structure and PAN to represent the bottom-up feature pyramid structure. Based on this, this system is designed to use... Figure 4 The PAN-type network structure shown serves as the head of both the classifier and detector, where squares 1, 2, and 3 are... Figure 2 The output of the backbone network (FPN) shown is given by the following: assuming the input image size of the backbone network is 608*608, the three output feature maps are (76*76), (38*38), and (19*19).

[0083] Understandably, referring to Figure 4 An SPP component, consisting of average pooling, was added to the header of both the classifier and detector. Using the SPP component helps to prevent significant loss of image features in block 3 (the third target feature map, 19×19 in size). Furthermore, the main algorithm used in the SPP component is average pooling, which can strengthen the local semantic relationships in the feature map.

[0084] It should be further explained that this application upsamples blocks 1, 2, and 3 into Out_1, Out_2, and Out_3, with sizes of (19×19), (38×38), and (76×76) respectively. The feature map size at this point is inverted from the feature map size formed by FPN. Specifically, for the small-sized feature map formed by FPN, the relationship between local features is enhanced through SPP, thereby enriching the positional feature information. For the other two sizes, positional feature information is enhanced through upsampling. The classifier and detector use the same head structure because this system integrates the classifier and detector to perform prediction box processing. The predictions, among which, Indicates the coordinates of the prediction box. These represent the width and height of the prediction box, respectively.

[0085] It should be noted that the number of channels for the final Out_1, Out_2, and Out_3 is increased to ensure that the final output can complete the classification and detection tasks. Now, assume the user has n types of hazardous materials at height, combined with... With four parameters and the prediction box confidence c, the final number of prediction parameters is n+4+1. For each grid point, there are three prior boxes, and the number of grid points in each feature map corresponds to the size of the feature map. Therefore, the final dimension for Out_1 is (19×19×3×(3×(n+4+1))), the final dimension for Out_2 is (38×38×3×(3×(n+4+1))), and the final dimension for Out_3 is (76×76×3×(3×(n+4+1))).

[0086] In one embodiment, the high-altitude hazardous object detection model is obtained through the following training method: acquiring a set of building cluster images; iteratively training the high-altitude hazardous object detection model based on the set of building cluster images, wherein the high-altitude hazardous object detection model performs classification detection on a building cluster image sample in the set of building cluster images to obtain prediction box parameters; updating the loss function based on the difference between the square root of the prediction box parameters and the square root of the label box parameters corresponding to the building cluster image sample; and ending the iteration when the loss function meets a preset condition.

[0087] Understandably, this application classifies and labels a set of building cluster images based on hazardous material categories, and uses the labeled building cluster images as training samples. Then, the model is iteratively trained using these training samples to enable the model to detect hazardous material categories. Therefore, this model helps community workers to check whether there are hazardous materials in the buildings within the community. Compared with door-to-door visits, this application helps reduce workload and improve the accuracy of the investigation, thereby helping to improve community safety.

[0088] It should be added that labeling can be done using AI labeling software to label high-risk objects. Specifically, 25% of the images in the dataset are labeled first for model training. Then, the model is used to predict the labels for the remaining 75% of the data. Finally, erroneous labels are corrected through manual review, which greatly reduces the cost of purely manual labeling.

[0089] It should be noted that the labeling process also includes generating image grids and anchor boxes: the labeled building group image samples are gridded, for example, by dividing the grid into grids of size [size missing]. A pixel-sized image of buildings is evenly divided into several 8×8 grids. The number of grids is then as follows: The main purpose of the generated mesh is to provide an accurate reference pixel coordinate system for model prediction, where ( , The coordinates are the points in the image grid. After generating the image grid, anchor frames are added to the three RGB channels of the image: First, obtain the center coordinates of each grid in each channel (…). , Secondly, the K-means algorithm is used to cluster the label boxes in the dataset using the center point coordinates, resulting in 9 boxes around the center point, which are then used as prior boxes (anchor boxes). Now, let's assume the width and height of the obtained prior boxes are respectively... , It's worth noting that the generated prior boxes don't have fixed sizes; their dimensions are derived from the actual label boxes and the clustering algorithm. The clustering algorithm works as follows: first, labeled boxes with similar aspect ratios are grouped together; then, the average length and width of each group's label boxes are used as the size of that group's prior boxes (anchor boxes). The main purpose of generating anchor boxes is to provide a reference point for the predicted boxes; the anchor boxes serve as this reference. The predicted boxes will be based on the ( ) of the anchor boxes. , ) Calculate positional deviation and , Calculate the width and height. The specific calculations will be discussed in the classifier and detector section later.

[0090] In one embodiment, after the model training is completed, the method further includes: collecting a set of building cluster images of the target community; inputting the set of building cluster images of the target community into the trained high-altitude hazardous object detection model, and fine-tuning the model parameters.

[0091] It should be further explained that, based on the pre-trained model, a small number of images are collected by the drone in the target community. The system model is then fine-tuned using these few collected samples to make the model suitable for the specified community. This step helps improve the generalization ability of the detection model.

[0092] It should be further explained that the key to fine-tuning technology lies in the creation of a small sample dataset. The method used in this application to create the dataset involves processing the newly collected dataset and the old dataset with the same distribution. Specifically, the steps are as follows: 1. Using a drone to photograph the balcony surfaces of the target community, the photographed objects must be similar to those in the training dataset; 2. Applying autoencoder denoising technology to denoise the collected dataset; 3. Manually labeling the denoised images, ensuring the same number of labels for each category; 4. Resizing the labeled data to fit the input size of the system model. These steps basically solve the general user requirements for detecting hazardous objects at high altitudes. However, for communities with large balconies, the collected images are high-resolution images. Simply resizing these images will lead to the loss of image features and reduce the system model's ability to detect small targets. Therefore, for images with large resolution that cannot be resized, this system will use segmentation to convert them into input images suitable for the system model, setting a segmentation overlap rate of 20% to prevent the target object from being segmented exactly. After obtaining the new dataset, it is fed into the system model for retraining and fine-tuning. This allows the system model to be further optimized based on the system parameters that have already been trained. The training steps are the same as steps 501 to 504. However, the fine-tuning training should be limited to no more than 3 training iterations to prevent overfitting.

[0093] In one embodiment, the loss function is expressed as:

[0094]

[0095] It should be further explained that since the values ​​of large predicted boxes are much larger than those of small predicted boxes, taking the square root of both (predicted box parameters and label box parameters) can reduce the difference. This allows the model to focus on both types of predicted boxes simultaneously rather than the large ones, making the loss more stable. Furthermore, after calculating the square root, the selectable region of the values ​​becomes smaller, which implicitly increases the probability of overlap between the ground truth boxes and the predicted boxes, alleviating the problem of non-overlapping.

[0096] It should be added that the design of the loss function is crucial to the overall performance of the network. The parameters mentioned above are: The confidence level c and the predicted probability p of the class both need to be obtained through gradient update using the loss function. The loss function designed in this system is shown in formula (5), where The label bounding box parameters represent the data of the actual labels. The loss function consists of the bounding box coordinates loss, bounding box width and height loss, confidence loss for an object inside the box, confidence loss for no object inside the box, and class probability loss. λ is a weighting constant for each loss term, controlling the proportion between them. In the context of: When calculating the loss, the MSE (mean squared error) method is used. Since the confidence score and class label are output in binary classification (0 or 1), cross-entropy is used for loss calculation. For the coefficients in the loss... and Calculate the Interchange of Value (IOU) between any real label value and the predicted bounding box of the system model. The predicted bounding box with the highest IOU is selected, and each predicted bounding box can only be used once for matching calculation. The value is 1 if the Interchange of Union (IOU) is less than 0 otherwise; if the IOU calculated between a predicted bounding box and all the true label values ​​is less than the threshold set by the system, then... The value is 1 if the loss is positive and 0 otherwise. It is worth noting that the range of i in the loss expression is N1×N1, and the range of j is [0,2]. i represents the predicted bounding box data with the i-th grid point as the reference, and Equation (5) shows the loss function design for the first target feature map (N1×N1); j represents the three channels of the feature map, from channel 0 to channel 2. Therefore, as shown in Equation (6), the losses of the three feature maps can be superimposed to obtain the total loss.

[0097]

[0098] In one embodiment, the target feature map is classified and detected by a classifier and a detector to obtain the hazardous object category corresponding to the building complex image, including: detecting whether the target feature map has the feature of hanging objects around the balcony security grille; detecting whether the target feature map has the feature of placing items on the edge of the balcony or above the balcony wall; detecting whether the target feature map has the feature of window damage; detecting whether the target feature map has the feature of air conditioner outdoor unit displacement; and detecting whether the target feature map has the feature of billboards installed on the exterior wall.

[0099] It is understandable that wind chimes hanging from balcony security grilles are typically installed outside the grilles, in the middle of the balcony, posing a risk of rope breakage and falling from a height. Such falls could result in death or injury, threatening residents' lives. This application can check whether they are hanging outside the balcony security grilles. Many users place potted plants on the edge of their balconies or above the balcony walls for sunlight. Without sufficiently wide wire mesh protection, this poses a serious safety hazard, threatening residents' lives. This application can check whether potted plants are present on the edge of balconies or above the balcony walls. Some residents have damaged windows, posing a risk of glass shards falling from a height. This application can check each household for broken windows. Air conditioner outdoor units are generally installed on the side of the exterior wall of the balcony. Over time, due to neglect, screws may loosen or the frame may be bent, causing the unit to tilt and potentially fall from a height. This application can check whether the air conditioner outdoor unit has shifted from its fixed position. Some residents have installed billboards on the exterior walls of the community, all located directly in front of the exterior walls of their balconies. These billboards, including both large and small ones, pose a risk of instability and potential collapse. This application can be used to check for the presence of such billboards on exterior walls. Similarly, residents hanging clothes on balcony security bars pose a risk of clothes being blown down by the wind and falling from a height. This application can be used to check for the presence of clothes hanging on the exterior surfaces of balconies.

[0100] In one embodiment, after obtaining the hazardous material category corresponding to the building complex image, the method further includes: predicting the risk level based on the hazardous material category corresponding to the building complex image to obtain prediction data.

[0101] It should be added that the risk level can be generated by combining the identified types and sizes of hazardous materials at high altitudes.

[0102] Reference Figure 7 and Figure 8 , Figure 7 and Figure 8 A flowchart of a detection method provided in another embodiment of this application: During training, images of buildings, balconies, etc., and images obtained by "web crawling" technology are used as training samples to train the model. Based on the trained system model, fine-tuning is performed according to the usage scenario. During use, the system model identifies hazardous materials and judges the risk level of hazardous materials in the images collected by the drone, and uploads the judgment results to the cloud server. The cloud server sends the results to the user's mobile terminal via wireless transmission. The user can be a community worker or a community resident.

[0103] The proposed visual recognition-based method for detecting hazardous objects at high altitudes combines cutting-edge deep learning algorithms to enable precise inspection of such objects using drones. This method can inspect multiple communities within a specified timeframe, significantly improving inspection efficiency. Furthermore, by using drones equipped with communication software, the method can notify community workers and residents in real time of the risk index of items and loose objects on balconies or windowsills. This greatly reduces labor costs and allows for a more comprehensive inspection of hazardous objects at high altitudes. In terms of cost, the method only requires the cost of terminal software development, data sample labeling, and drone usage and battery life. It can also conduct inspections on multiple areas within a single community or simultaneously in multiple communities. Overall, the cost, both economically and in terms of time, is lower than current labor costs.

[0104] This application proposes a vision-based method for detecting hazardous objects at high altitudes. Using a drone as a carrier, it combines deep learning target detection algorithms with various mathematical models. Through real-time communication between the drone and a terminal, it can screen for hazardous objects at high altitudes. Furthermore, this invention is also applicable to the detection of hazardous objects on various high-rise building facades, more effectively and broadly protecting the lives of community residents, maintaining harmonious relationships among community members, and promoting harmonious urban development.

[0105] Reference Figure 9 , Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 900 includes: a memory 901, a processor 902, and a computer program stored in the memory 901 and executable on the processor 902. When the computer program is executed, it is used to perform the methods described above.

[0106] The processor 902 and the memory 901 can be connected via a bus or other means.

[0107] The memory 901, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs, such as the methods described in the embodiments of this application. The processor 902 implements the above-described methods by running the non-transitory software programs and instructions stored in the memory 901.

[0108] The memory 901 may include a program storage area and a data storage area. The program storage area may store the operating system and application programs required for at least one function; the data storage area may store the methods described above. Furthermore, the memory 901 may include high-speed random access memory and may also include non-transitory memory, such as at least one storage device, flash memory, or other non-transitory solid-state storage device. In some embodiments, the memory 901 may optionally include memory remotely located relative to the processor 902, and these remote memories may be connected to the electronic device 900 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0109] The non-transient software program and instructions required to implement the above method are stored in memory 901 and are executed by one or more processors 902.

[0110] This application also provides a computer-readable storage medium storing computer-executable instructions for performing the above-described methods.

[0111] In one embodiment, the computer-readable storage medium stores computer-executable instructions that are executed by one or more control processors to implement the method described above.

[0112] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0113] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, storage device storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically include computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0114] It should also be understood that the various implementation methods provided in this application can be combined arbitrarily to achieve different technical effects.

[0115] The above is a detailed description of the preferred embodiments of this application. However, this application is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A method for detecting hazardous materials at high altitudes, characterized in that, include: A high-altitude hazardous material detection model and building cluster images of the target community are acquired. The high-altitude hazardous material detection model includes a backbone network, a classifier, and a detector. The classifier and detector headers include SPP components composed of an average pooling algorithm. The backbone network includes a CBM component set and a Res component set. The CBM component set includes a first CBM component and a second CBM component connected in series. Each Res component in the Res component set includes a CMB1 component, a first CMB2 component, an Add component, a Concat component, and a second CMB2 component. The CMB1 component and... The CMB2 components are connected in series. The Add component in the Res component is used to concatenate the input of the CMB1 component and the output residual of the first CMB2 component. The Concat component in the Res component is used to concatenate the output of the Add component and the input of CMB1. The output of the Concat component in the Res component is the input of the second CMB2 component, and the output of the second CMB2 component is the output of the corresponding Res component. The Res component set includes a first Res component, a second Res component, a third Res component, a fourth Res component, and a fifth Res component connected in series in sequence. The first CBM component extracts features from the building cluster image of the community to be detected to obtain a first output feature map; The second output feature map is obtained by downsampling the first output feature map using the second CBM component. The first target feature map is obtained by using the first Res component, the second Res component, the third Res component, and the second output feature map; The second target feature map is obtained by using the fourth Res component and the first target feature map; The third target feature map is obtained by using the fifth Res component and the second target feature map; The third target feature map is upsampled using the SPP component to obtain Out_1; The third target feature map is upsampled using the SPP component and PAN-type network structure, and the upsampled result is superimposed with the first target feature map using the Concat component to obtain Out_3. The second target feature map is overlaid using the Concat component to obtain Out_2; Classification detection is performed on Out_1, Out_2, and Out_3 to obtain the hazardous material category corresponding to the building complex image.

2. The method according to claim 1, characterized in that, The high-altitude hazardous material detection model was obtained through the following training methods, including: Obtain a sample set of building cluster images; The high-altitude hazardous object detection model is iteratively trained based on the building cluster image sample set. Specifically, the high-altitude hazardous object detection model classifies and detects a building cluster image sample from the set to obtain prediction box parameters. The loss function is updated based on the difference between the square root of the prediction box parameters and the square root of the corresponding label box parameters of the building cluster image sample. The iteration ends when the loss function meets a preset condition.

3. The method according to claim 2, characterized in that, After the model training is complete, the method further includes: Collect a set of images of the buildings in the target community; The image set of buildings in the target community is input into the trained high-altitude hazardous object detection model to fine-tune the model parameters.

4. The method according to claim 2, characterized in that, The loss function expression is as follows: , These are all parameters of the prediction box. , These are all label box parameters. The preset loss coefficient, , , is the weighting constant for each loss term.

5. The method according to claim 1, characterized in that, The classification and detection of hazardous materials includes: detecting whether the target feature map has the feature of hanging objects around the balcony security grille; Detect whether the target feature map has the feature of placing an item on the edge of the balcony or above the balcony wall; Detect whether the target feature map has features of window damage; Detect whether the target feature map contains features indicating that the outdoor unit of the air conditioner has shifted; The system detects whether the target feature map contains the feature of an advertising billboard installed on an exterior wall.

6. The method according to claim 5, characterized in that, After obtaining the hazardous material category corresponding to the building complex image, the method further includes: Risk levels are predicted based on the hazardous material categories corresponding to the building complex images, resulting in prediction data.

7. An electronic device, characterized in that, include: At least one memory; At least one processor; At least one program; The program is stored in the memory, and the processor executes at least one of the programs to implement the method as claimed in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable signals for performing the method as described in any one of claims 1 to 6.

Images