Instance segmentation model-based target edge detection method and device

Abstract

The application discloses a kind of based on instance segmentation model target edge detection method and device, the method includes: the image collected is preprocessed, the labeling of the region to be detected is carried out and data set is made;In the main network of instance segmentation model, neck network and head network, the image data information in its training set is respectively operated to realize instance segmentation model training;The target detection of the region to be detected and other regions is carried out, and the initial target detection model and its corresponding final target detection model are acquired;The first detection frame of the region to be detected in final target detection model is acquired, and the vertex coordinates of the first detection frame are acquired using unmanned aerial vehicle;The segmentation detection algorithm is carried on unmanned aerial vehicle, the detection of the region to be detected is realized by the first detection frame, and the reproduction of the occluded part of the region to be detected is realized.The problem that the existence of irregular target edge part is similar to the gray value of other regions, and the contour recognition of model through image identification brings challenge.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing image processing, and in particular to a target edge detection method and device based on an instance segmentation model. BACKGROUND

[0002] Buildings, as an important artificial feature, reflect the evolution and layout of human society and economic development. Scientific and accurate target identification from remote sensing data lays the foundation for numerous practical applications, including urban planning, street navigation, national defense security, and post-disaster emergency response, and has important research value. With the continuous development of urbanization, the density of buildings is becoming higher and higher, and the characteristics of targets are becoming more and more complex. Traditional manual identification techniques not only consume a large amount of manual time and cost, but also are difficult to meet the actual demand in terms of detection accuracy.

[0003] The commonly used target edge detection method is to detect the obtained region original image to obtain the target region in the original image, and then perform target detection in the target region to obtain the position information and category information of the target edge in the region. However, in actual situations, there are a large number of mutual occlusions between city buildings, the contrast of the overlapping region is low, the scene is complex, and false detection is prone to occur. The target edge part has irregular characteristics, which brings difficulties to detection, and the gray value of the target edge region is similar to that of other regions, which brings challenges to the model to identify the contour through the image. SUMMARY

[0004] In the embodiments of the present application, by providing a target edge detection method based on an instance segmentation model, the problem of false detection caused by a large number of mutual occlusions between city buildings, low contrast of overlapping regions, complex scenes, and irregular characteristics of target edge parts in actual situations is solved, and the problem of challenges to the model to identify the contour through the image caused by the similar gray values of the target edge region and other regions is solved.

[0005] In the embodiments of the present application, by providing a target edge detection method based on an instance segmentation model, the problem of false detection caused by a large number of mutual occlusions between city buildings, low contrast of overlapping regions, complex scenes, and irregular characteristics of target edge parts in actual situations is solved, and the problem of challenges to the model to identify the contour through the image caused by the similar gray values of the target edge region and other regions is solved.

[0005] In the embodiments of the present application, by providing a target edge detection method based on an instance segmentation model, the problem of false detection caused by a large number of mutual occlusions between city buildings, low contrast of overlapping regions, complex scenes, and irregular characteristics of target edge parts in actual situations is solved, and the problem of challenges to the model to identify the contour through the image caused by the similar gray values of the target edge region and other regions is solved.

[0005] In the embodiments of the present application, by providing a target edge detection method based on an instance segmentation model, the problem of false detection caused by a large number of mutual occlusions between city buildings, low contrast of overlapping regions, complex scenes, and irregular characteristics of target edge parts in actual situations is solved, and the problem of challenges to the model to identify the contour through the image caused by the similar gray values of the target edge region and other regions is solved.

[0005] In the embodiments of the present application, by providing a target edge detection method based on an instance segmentation model, the problem of false detection caused by a large number of mutual occlusions between city buildings, low contrast of overlapping regions, complex scenes, and irregular characteristics of target edge parts in actual situations is solved, and the problem of challenges to the model to identify the contour through the image caused by the similar gray values of the target edge region and other regions is solved.

[0006] With reference to the first aspect, in a possible implementation manner, the operations of respectively performing corresponding operations on image data information in the training set of the instance segmentation model in the backbone network, the neck network and the head network of the instance segmentation model to implement the instance segmentation model training, comprise: performing feature extraction in the backbone network of the YOLOV8 model, obtaining a feature extraction result, and performing convolution operation on the image to scale the image to the same size; inputting the feature extraction result into the neck network of the YOLOV8 model to perform multi-feature aggregation; performing regression loss calculation in the head network of the YOLOV8 model to obtain the trained YOLOV8 model of the instance segmentation model.

[0007] With reference to the first aspect, in a possible implementation manner, the operations of performing target detection on the to-be-detected region and other regions, obtaining an initial target detection model and a final target detection model corresponding thereto, comprise: modifying parameters in a script file of the trained YOLOV8 model of the instance segmentation model to obtain an initial target detection model; performing model training on the initial target detection model, and adjusting related hyperparameters according to the test effect in the validation set; after all samples in the training set of the initial target detection model complete one round of training, testing the initial target detection model to obtain a test result; re-labeling related features obtained by the test result, and performing iterative training to obtain the final target detection model.

[0008] With reference to the first aspect, in a possible implementation manner, the first detection frame is a minimum bounding rectangle of the to-be-detected region in the final target detection model; and the operations of determining the minimum bounding rectangle of the to-be-detected region in the final target detection model comprise: finding coordinates of boundary points in the to-be-detected region to obtain a coordinate set, and preferentially selecting a leftmost or rightmost coordinate point in the coordinate set as a starting point; sorting other boundary points in the coordinate set except the starting point according to the polar angle size between the starting point and the other boundary points; sequentially adding each boundary point to a convex hull of the to-be-detected region according to the sorted order; selecting a connecting line of two adjacent points on the convex hull as a first pair of sides, calculating lengths of two sides perpendicular to the first pair of sides, so that an area formed by the first pair of sides and the two sides falls within an initial bounding rectangle; determining whether the area is smaller than the area of the initial bounding rectangle; if the determination result is yes, updating the area of the initial bounding rectangle and the length and width thereof; rotating the first pair of sides to a second pair of sides, and repeating the above process until rotating to the first pair of sides, and outputting the minimum bounding rectangle.

[0009] With reference to the first aspect, in a possible implementation manner, the step of carrying the segmentation detection algorithm on the unmanned aerial vehicle, detecting the to-be-detected region through the first detection frame, and realizing the occlusion reconstruction of the to-be-detected region comprises: sorting the targets in the image according to the values of the calculated confidence; if a target of interest is found, marking the serial number of the target, scanning the target from different angles to obtain first detection frames of different angles and set the first detection frames as a first image set, and performing panoramic image target splicing technology; performing feature matching on the first image set to obtain a feature matching result; calculating the transformation structure between images according to the feature matching result, and performing image superposition according to the transformation structure to realize image mapping; aligning feature points through an image splicing algorithm to realize image registration; performing image segmentation and adaptively selecting a splicing seam according to image content; and performing fusion processing on the image according to a multi-band image fusion technology, so as to realize the detection of the to-be-detected region and the occlusion reconstruction of the to-be-detected region.

[0010] With reference to the first aspect, in a possible implementation manner, if the first detection frames overlap due to line-of-sight overlap, the step of integrating overlapping frames is performed.

[0011] With reference to the first aspect, in a possible implementation manner, the step of integrating overlapping frames comprises: putting a plurality of first detection frames that overlap due to line-of-sight overlap into a second image set, and calculating the overlapping area; calculating the percentage of the overlapping area in the first detection frame with higher confidence, and judging whether the percentage is greater than a preset threshold; if the result of the judgment is yes, regarding the plurality of first detection frames as the same target, and finding the minimum circumscribed rectangle of the target as a second detection frame; and returning the coordinate information of the four vertex points of the second detection frame to the unmanned aerial vehicle, so as to complete the step of integrating overlapping frames.

[0012] In a second aspect, an embodiment of the present application provides a target edge detection device based on an instance segmentation model, which comprises: a data set making module, configured to pre-process collected images, label the to-be-detected regions of the pre-processed images, and make a data set; an instance segmentation module, configured to respectively perform corresponding operations on image data information in a training set of the instance segmentation model in a backbone network, a neck network and a head network of the instance segmentation model, so as to realize training of the instance segmentation model; a target detection module, configured to perform target detection on the to-be-detected regions and other regions, and obtain an initial target detection model and a final target detection model corresponding to the initial target detection model; a detection frame obtaining module, configured to obtain a first detection frame of the to-be-detected region in the final target detection model, and obtain vertex coordinates of the first detection frame by using an unmanned aerial vehicle; and an occlusion reconstruction module, configured to carry a segmentation detection algorithm on the unmanned aerial vehicle, detect the to-be-detected region through the first detection frame, and realize the occlusion reconstruction of the to-be-detected region.

[0013] With reference to the second aspect, in a possible implementation manner, the operations of respectively performing corresponding operations on image data information in the training set of the instance segmentation model in the backbone network, the neck network and the head network of the instance segmentation model to implement the instance segmentation model training, comprises: performing feature extraction in the backbone network of the YOLOV8 model, obtaining a feature extraction result, and performing convolution operation on the image to scale the image to the same size; inputting the feature extraction result into the neck network of the YOLOV8 model to perform multi-feature aggregation; performing regression loss calculation in the head network of the YOLOV8 model to obtain the trained YOLOV8 model of the instance segmentation model.

[0014] With reference to the second aspect, in a possible implementation manner, the operations of respectively performing corresponding operations on image data information in the training set of the instance segmentation model in the backbone network, the neck network and the head network of the instance segmentation model to implement the instance segmentation model training, comprises: performing feature extraction in the backbone network of the YOLOV8 model, obtaining a feature extraction result, and performing convolution operation on the image to scale the image to the same size; inputting the feature extraction result into the neck network of the YOLOV8 model to perform multi-feature aggregation; performing regression loss calculation in the head network of the YOLOV8 model to obtain the trained YOLOV8 model of the instance segmentation model.

[0015] With reference to the second aspect, in a possible implementation manner, the first detection frame is the minimum circumscribed rectangle of the to-be-detected region in the final target detection model; and the operations of determining the minimum circumscribed rectangle of the to-be-detected region in the final target detection model comprise: finding the coordinates of the boundary points in the to-be-detected region to obtain a coordinate set, and preferentially selecting the leftmost or rightmost coordinate point in the coordinate set as a starting point; sorting the boundary points in the coordinate set other than the starting point according to the polar angle size between the starting point; sequentially adding each boundary point to the convex hull of the to-be-detected region according to the sorted order; selecting the connecting line of two adjacent points on the convex hull as a first pair of sides, calculating the lengths of two sides perpendicular to the first pair of sides, so that the area formed by the first pair of sides and the two sides falls within the initial circumscribed rectangle; determining whether the area is smaller than the area of the initial circumscribed rectangle; if the determination result is yes, updating the area of the initial circumscribed rectangle and the length and width thereof; rotating the first pair of sides to a second pair of sides, and repeating the above process until rotating to the first pair of sides, and outputting the minimum circumscribed rectangle.

[0016] With reference to the second aspect, in a possible implementation manner, the carrying of the segmentation detection algorithm on the unmanned aerial vehicle, the detection of the to-be-detected region is implemented by the first detection frame, and the reconstruction of the occluded part of the to-be-detected region is implemented, and the method comprises the following steps: sorting the targets in the image according to the calculated values of the confidence degrees; if a target of interest is found, marking the serial number of the target, scanning the target from different angles to obtain first detection frames of different angles and setting the first detection frames as a first image set, and performing panoramic image target splicing technology; performing feature matching on the first image set to obtain a feature matching result; calculating a transformation structure between images according to the feature matching result, and implementing image mapping by image superposition according to the transformation structure; aligning feature points by an image splicing algorithm to implement image registration; performing image segmentation and adaptively selecting a splicing seam according to image content; and performing fusion processing on the images according to a multi-band image fusion technology, so as to implement the detection of the to-be-detected region and the reconstruction of the occluded part of the to-be-detected region.

[0017] With reference to the second aspect, in a possible implementation manner, if the first detection frames overlap due to line-of-sight overlap, an overlapping frame integration step is performed.

[0018] With reference to the second aspect, in a possible implementation manner, the overlapping frame integration step comprises the following steps: putting a plurality of first detection frames that overlap due to line-of-sight overlap into a second image set, and calculating an overlapping area; calculating a percentage of the overlapping area in a first detection frame with a higher confidence degree, and judging whether the percentage is greater than a preset threshold; if the result of the judgment is yes, regarding the plurality of first detection frames as a same target, and finding a minimum circumscribed rectangle of the target as a second detection frame; and returning four vertex coordinate point information of the second detection frame to the unmanned aerial vehicle, so as to complete the overlapping frame integration step.

[0019] In a third aspect, an embodiment of the present application provides a target edge detection server based on an instance segmentation model, comprising a memory and a processor; the memory is used to store computer executable instructions; and the processor is used to execute the computer executable instructions to implement the method in the first aspect or any possible implementation manner of the first aspect.

[0020] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores executable instructions, and a computer executes the executable instructions to implement the method in the first aspect or any possible implementation manner of the first aspect.

[0021] The one or more technical solutions provided in the embodiments of the present application have at least the following technical effects:

[0022] The embodiment of the present application provides a kind of based on instance segmentation model target edge detection method, this method is implemented, the image collected is preprocessed, the image after being preprocessed can be more clear, and image edge and detail information can be more clear, the image data information in the training set of the main network, neck network and head network of instance segmentation model is respectively operated to realize instance segmentation model training in the labeling of the region to be detected of image after being preprocessed and making data set, can realize instance segmentation of the region to be detected and other regions, target detection is carried out to the region to be detected and other regions, obtain initial target detection model and its corresponding final target detection model, obtain the first detection frame of the region to be detected in final target detection model, the vertex coordinates of first detection frame are obtained using unmanned aerial vehicle, the vertex coordinates of first detection frame are obtained using unmanned aerial vehicle, can facilitate unmanned aerial vehicle to scan, segmentation detection algorithm is carried on unmanned aerial vehicle, detection is realized to the region to be detected by first detection frame, and the reproduction of the occlusion part of the region to be detected is realized, so as to solve the situation that there is a large amount of mutual occlusion between city buildings in actual situation, the contrast of overlapping area is low, scene is complex, easy to produce misjudgment phenomenon, target edge part exists irregular characteristics, bring difficulty for detection, and the gray value of target edge region and other regions is similar, which brings challenge to the model through image recognition contour. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application or the prior art. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0024] Figure 1 A flowchart of a target edge detection method based on an instance segmentation model is provided for the embodiments of the present application.

[0025] Figure 2 A specific flowchart for training an instance segmentation model by respectively operating image data information in the training set of the main network, neck network and head network of instance segmentation model is provided for the embodiments of the present application.

[0026] Figure 3 A specific flowchart for target detection of the region to be detected and other regions, obtaining an initial target detection model and its corresponding final target detection model is provided for the embodiments of the present application.

[0027] Figure 4 A specific flowchart for realizing the determination of the minimum bounding rectangle of the region to be detected in the final target detection model is provided for the embodiments of the present application.

[0028] Figure 5 The flowchart below illustrates how a segmentation detection algorithm is mounted on a drone, using a first detection box to detect the region to be detected and reproduce the occluded portion of the region to be detected.

[0029] Figure 6 A flowchart illustrating the steps for merging overlapping frames when there is overlap of the first detection frames due to overlapping lines of sight, as provided in this embodiment of the application.

[0030] Figure 7 A schematic diagram of a target edge detection device based on an instance segmentation model provided in this application embodiment;

[0031] Figure 8 A schematic diagram of a target edge detection server based on an instance segmentation model provided in an embodiment of this application;

[0032] Figure 9 This is a schematic diagram of the minimum bounding rectangle of the output provided in the embodiments of this application. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0034] The following description of some technologies involved in the embodiments of this application is provided to aid understanding and should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, some descriptions of well-known functions and structures are omitted in the following description.

[0035] This application provides a target edge detection method based on an instance segmentation model, such as... Figure 1 As shown, the method includes steps S101 to S105. Wherein, Figure 1 This is merely one execution order shown in the embodiments of this application and does not represent the only execution order for a YOLOv8-based building rooftop edge detection method. Where the final result can be achieved, Figure 1 The steps shown can be performed in parallel or in reverse order.

[0036] S101: pre-process the collected image, label the to-be-detected region of the pre-processed image, and make a data set. The to-be-detected region is the roof edge region.

[0037] Specifically, modern remote sensing technologies such as unmanned aerial vehicles can be used for image collection. The images collected by the present application are building roof region images, which are mainly divided into roof edge regions and other regions. Since the roof edge region is closely connected with the other region and has similar gray values, there are problems such as occlusion between them, so it is necessary to pre-process the collected image, that is, to enhance the processing. The main methods are: increasing the contrast of the image, making the bright area brighter and the dark area darker, so that the image is clearer. Specifically, in Python, the cv2.convertScaleAbs() function of the OpenCV library can be used to achieve this. Perform image sharpening operation to make the image edge and detail information more clear. Specifically, a filter can be used to achieve this. Apply Gaussian filtering to achieve image smoothing and noise filtering. Specifically, in Python, the cv2.GaussianBlur() function of the OpenCV library can be used to achieve this.

[0038] Further, to ensure the consistency of the subsequent model images, scale adjustment is needed to adjust the image size to the same height and width.

[0039] S102: respectively perform corresponding operations on the image data information in the training set of the backbone network, neck network and head network of the instance segmentation model to realize instance segmentation model training.

[0040] Specifically, before step S102, the data set needs to be divided into training set, validation set and test set, and ensure that the labeled building roof region image data contains object category and bounding box coordinate information.

[0041] Figure 2 The specific flowchart for implementing step S102 of respectively performing corresponding operations on the image data information in the training set of the backbone network, neck network and head network of the instance segmentation model to realize instance segmentation model training is provided for the embodiments of the present application, as shown in Figure 2 , which includes steps S201 to S203.

[0042] It should be noted that in addition to the YOLOV8 model, the instance segmentation model can also be implemented by using instance segmentation algorithms (English: mask rcnn), dense sliding window instance segmentation framework (English: TensorMask), MEInst algorithm, YOLACT model, etc. The present application takes the YOLOV8 model as an example to implement instance segmentation model training, but the present application is not limited to the YOLOV8 model.

[0043] S201: Feature extraction is performed in the backbone network of the YOLOV8 model, and a feature extraction result is obtained. The image is subjected to a convolution operation to scale the image to the same size.

[0044] S202: The feature extraction result is input into the neck network of the YOLOV8 model, and multi-feature aggregation is performed.

[0045] S203: Regression loss calculation is performed in the head network of the YOLOV8 model, and a trained YOLOV8 model of the instance segmentation model is obtained.

[0046] Specifically, the image subjected to preprocessing is trained in the YOLOV8 model to obtain building roof area features to realize instance segmentation of the roof edge area and other areas. In the YOLOV8 model, the C2f module is used in the backbone network instead of the C3 module in the YOLOV5, which can realize lighter model while ensuring the accuracy of the model. In the setting of the loss function, the YOLOV8 model uses the distributed feature learning idea (DFL for short) and the anchor-free form, uses the variational focal loss (VFL Loss) as the classification loss, and uses the target detection loss function (CIoU Loss+DFL) as the regression loss. The network structure of the YOLOV8 model mainly consists of three parts, the backbone network (Backbone), the neck network (Neck), and the head network (Head). The CBS module in the YOLOV8 model represents the convolution (Conv), batch normalization (BN), and activation function (SiLU) operations. The C2f module represents the combination operation module of convolution, splitting and merging, which has a more abundant gradient flow.

[0047] Further, first, the image, i.e., the original image, is input into the backbone network of the YOLOV8 model for feature extraction. When the input original image passes through the CBS module in the YOLOV8 model, a convolution operation with a convolution kernel size of 3 and a step size of 2 is used to realize the size scaling of the original image. In this application, the original image is uniformly scaled to a size of 512x512. Of course, the original image can also be scaled to other sizes, and the application is not limited to the above-mentioned size. Specifically, the convolution process includes image convolution, regularization, and function activation once, and the calculation formula is as follows: where P represents the padding operation, K represents the convolution kernel size, Q(x, y) represents the pixel value of the new image at coordinates x and y after convolution, z represents the channel value, n represents the total number of channels of the original image, i and j represent the output index, and the SiLU activation function used in the CBS module is expressed as: wherein x represents an input signal of the neural network.

[0048] Further, by replacing the C3 module in the YOLOV5 model with the C2f module, the channel number is changed in the form of splitting or splicing according to the scaling coefficient, which can effectively reduce the complexity and capacity of the YOLOV8 model, and the application of the dense residual structure can enhance the expression ability of the features. The above output is transmitted into the SPPF module of the backbone network of the YOLOV8 model. First, a CBS convolutional layer is included to perform multiple max-pooling operations. The feature maps that have not been max-pooled are spliced with the feature maps obtained after each max-pooling operation. The SPPF module can increase the receptive field and capture feature information at different levels in the scene.

[0049] Specifically, the feature extraction result is input into the neck network of the YOLOV8 model for multi-feature aggregation. In the PAN-FPN module in the neck network of the YOLOV8 model, the FPN is used to strengthen the convolutional feature expression of the neck network, and the PAN is used to realize path aggregation to further enhance the expression ability of multi-scale features. The use of the C2f module in the neck network again enables the YOLOV8 model to obtain more rich gradient flow information while ensuring lightweight.

[0050] Further, the YOLOV8 model has a large change in the head network part compared with the YOLOV5 model. The decoupled head structure is used instead of the coupled head to separate the classification head and the detection head. The VFL Loss is used as the classification loss, and the CIoU Loss+DFL is used as the regression loss. The calculation formula of the VFL Loss is as follows: wherein q is the intersection over union of the predicted box and the real box, p is the score, i.e. the probability, is the confidence, is a hyperparameter. When the predicted box intersects with the real box, i.e. q>0, it is a positive sample, and when the predicted box does not intersect with the real box, i.e. q=0, it is a negative sample. The calculation formula of the target detection loss function is as follows: v, wherein, v represents a penalty term, b represents the center point of the predicted box, represents the center point of the target box, and IoU represents the intersection over union, represents the Euclidean distance between the predicted box and the target box, and C represents the diagonal distance of the closed area of the predicted box and the target box, represents a weight coefficient, and v represents a parameter for measuring the consistency of the relative proportion of the predicted box and the target box. Specifically, v is defined as: wherein gt represents a target box, represents a width of the target box, represents a height of the target box, represents a width of the prediction box, represents a height of the prediction box. The DFL loss function can make the head network quickly focus on the distribution of positions close to the target position. The calculation formula of the DFL loss function is as follows: wherein y represents a true value of the target position, represents a prediction value on the left side of the target position, represents a probability corresponding to the prediction value on the left side of the target position, represents a prediction value on the right side of the target position, represents a probability corresponding to the prediction value on the right side of the target position. Specifically, adding the target detection loss function and the DFL loss function can obtain a total loss. The regression loss calculation is mainly used to measure the gap between the predicted position and the true position of the model, and the gap is taken into account in the calculation, so as to reduce the loss range of the model, enable the model to quickly converge, and help improve the accuracy and performance of the model, so as to make it perform better in subsequent target detection tasks.

[0051] S103: performing target detection on the to-be-detected region and other regions to obtain an initial target detection model and a final target detection model corresponding thereto.

[0052] Figure 3 The specific flowchart of implementing step S103 of performing target detection on the to-be-detected region and other regions to obtain an initial target detection model and a final target detection model corresponding thereto provided by the embodiments of the present application is shown in FIG. 3, and includes steps S301 to S304. Figure 3

[0053] S301: modifying parameters in a YOLOV8 model script file after training of an instance segmentation model to obtain an initial target detection model.

[0054] Specifically, in the source code of the YOLOV8 model, open the train.py file and modify the opt content in the ain.py file.

[0055] S302: performing model training on the initial target detection model and adjusting related hyperparameters according to the test effect in the validation set.

[0056] S303: after all samples in the training set of the initial target detection model are trained for one time, testing the initial target detection model to obtain a test result.

[0057] ​S304: re-label the relevant features obtained by the test results, and perform iterative training to obtain a final target detection model.

[0058] S104: obtain a first detection frame of the to-be-detected region in the final target detection model, and obtain the vertex coordinates of the first detection frame by using the unmanned aerial vehicle. The first detection frame is the minimum circumscribed rectangle of the to-be-detected region in the final target detection model.

[0059] Figure 4 The specific flowchart for determining the minimum circumscribed rectangle of the to-be-detected region in the final target detection model provided by the embodiments of the present application is shown in FIG. 4, which includes steps S401 to S407. Figure 4

[0060] S401: find the coordinates of the boundary points in the to-be-detected region to obtain a coordinate set, and preferentially select the leftmost or rightmost coordinate point in the coordinate set as a starting point.

[0061] Specifically, the coordinates of the boundary points are the coordinates of the points on the boundary line.

[0062] S402: sort the boundary points other than the starting point in the coordinate set according to the polar angle size between the starting point and the boundary points.

[0063] S403: add each boundary point to the convex hull of the to-be-detected region in turn according to the sorted order.

[0064] Specifically, if the current boundary point and the two boundary points added to the convex hull of the to-be-detected region before it form a counterclockwise direction, i.e., an obtuse angle, the current boundary point is added to the convex hull, and if the current boundary point and the two boundary points added to the convex hull of the to-be-detected region before it form a clockwise direction, i.e., an acute angle, the last boundary point added to the convex hull before it is deleted, and the relationship between the current boundary point and the two boundary points added to the convex hull of the to-be-detected region before it in the new convex hull is judged again.

[0065] S404: select the connecting line of the two adjacent points on the convex hull as a first pair of sides, and calculate the lengths of the two sides perpendicular to the first pair of sides so that the area formed by the first pair of sides falls within the initial circumscribed rectangle.

[0066] Specifically, the first pair of sides can be the extension line of the connecting line of the two adjacent points on the convex hull.

[0067] S405: determine whether the area is smaller than the area of the initial circumscribed rectangle.

[0068] If the determination result is yes, step S406 is performed: updating the area of the initial circumscribed rectangle and its length and width.

[0069] ​S407: rotate the first pair of edges to the second pair of edges, repeat the above process until rotating to the first pair of edges, and output the minimum bounding rectangle.

[0070] Specifically, the center coordinates, length, width and rotation angle of the minimum bounding rectangle are output.

[0071] Figure 9 The schematic diagram of the output minimum bounding rectangle provided by the embodiment of the present application is shown in Figure 9 The rectangle is the minimum bounding rectangle of the to-be-detected region.

[0072] S105: mount the segmentation detection algorithm on the unmanned aerial vehicle, realize detection of the to-be-detected region by the first detection frame, and realize reproduction of the occluded part of the to-be-detected region.

[0073] Figure 5 The specific flowchart for realizing mounting the segmentation detection algorithm on the unmanned aerial vehicle in step S105, realizing detection of the to-be-detected region by the first detection frame, and realizing reproduction of the occluded part of the to-be-detected region provided by the embodiment of the present application is shown in Figure 5 , which includes steps S501 to S507.

[0074] Specifically, mounting the segmentation detection algorithm on the unmanned aerial vehicle can process the collected images during the cruising of the unmanned aerial vehicle, obtain images of different angles by scanning the specific target, i.e., the roof of interest, perform image processing, and realize reproduction of the occluded part of the to-be-detected region.

[0075] S501: sort the targets in the image according to the calculated values of the confidence degrees.

[0076] Specifically, the targets in the image refer to the roofs in the image.

[0077] S502: if the target of interest is found, mark the serial number of the target, perform scanning of different angles to obtain first detection frames of different angles and set them as a first image set, and perform panoramic image target splicing technology.

[0078] S503: perform feature matching on the first image set to obtain a feature matching result.

[0079] Specifically, feature matching is to find similar or identical feature points between different images, which can be realized by using algorithms such as SIFT, SURF and ORB. The above algorithms can find and extract unique feature points in the image, and then perform matching.

[0080] S504: calculate the transformation structure between the images according to the feature matching result, and perform image superposition to realize image mapping according to the transformation structure.

[0081] Specifically, the transformation structure between images can be calculated by a transformation matrix, i.e., mapping the source image onto the target image.

[0082] S505: Image registration is realized by aligning feature points through an image stitching algorithm.

[0083] Specifically, the image stitching algorithm (English: Area-based Robust Point Matching, abbreviated as: APAP) can make the feature points as aligned as possible between two images, so that the images can be better registered.

[0084] S506: Image segmentation and adaptive selection of stitching seams are performed according to image content.

[0085] S507: The image is fused according to a multi-band image fusion technique, so as to realize detection of the to-be-detected region and reproduction of the occluded part of the to-be-detected region.

[0086] Specifically, the multi-band image fusion technique (English: multi-band blending) can decompose the color value of each pixel of the image into multiple channels, and independently apply different fusion algorithms on each channel, which can better preserve the details and color accuracy of the image.

[0087] Figure 6 The specific flowchart of the overlapping box integration step provided by the embodiment of the present application when there is a case of overlapping first detection boxes due to line-of-sight overlap is shown in FIG. 6, which includes steps S601 to S604. Figure 6

[0088] S601: Put multiple first detection boxes overlapped due to line-of-sight overlap into a second image set, and calculate the overlapping area.

[0089] S602: Calculate the percentage of the overlapping area in the first detection box with higher confidence, and determine whether the percentage is greater than a preset threshold.

[0090] Specifically, the preset threshold can be set to 20%, but it can also be set to other values, and the present application is not limited to the above values.

[0091] If the determination result is yes, step S603 is performed: multiple first detection boxes are regarded as the same target, and the minimum circumscribed rectangle of the target is found as a second detection box.

[0092] S604: The four vertex coordinate point information of the second detection box is transmitted back to the unmanned aerial vehicle, so as to complete the overlapping box integration step.

[0093] The embodiment of the present application also provides an instance segmentation model-based target edge detection device 700, as shown in FIG. 7.​Figure 7 As shown, the device comprises a data set making module 701, an instance segmentation module 702, a target detection module 703, an acquisition detection frame module 704, and a realization occlusion reproduction module 705.

[0094] The data set making module 701 is configured to pre-process the collected images, label the images after pre-processing for detection regions, and make a data set

[0095] The instance segmentation module 702 is configured to respectively perform corresponding operations on image data information in the training set of the instance segmentation model in the backbone network, neck network and head network of the instance segmentation model, thereby realizing training of the instance segmentation model. The instance segmentation module 702 is specifically configured to perform feature extraction in the backbone network of the YOLOV8 model, obtain a feature extraction result, perform convolution operation on the image to scale the image to the same size; input the feature extraction result into the neck network of the YOLOV8 model to perform multi-feature aggregation; perform regression loss calculation in the head network of the YOLOV8 model to obtain the YOLOV8 model after training of the instance segmentation model.

[0096] The target detection module 703 is configured to perform target detection on the detection region and other regions, and obtain an initial target detection model and a final target detection model corresponding thereto. The target detection module 703 is specifically configured to modify parameters in a script file of the YOLOV8 model after training of the instance segmentation model to obtain the initial target detection model; perform model training on the initial target detection model, and adjust related hyperparameters according to the test effect in the validation set; after all samples in the training set of the initial target detection model complete one round of training, test the initial target detection model to obtain a test result; re-label related features obtained from the test result, and perform iterative training, thereby obtaining the final target detection model.

[0097] The detection frame acquisition module 704 is configured to acquire a first detection frame of a region to be detected in a final target detection model, and acquire vertex coordinates of the first detection frame by using a UAV. The first detection frame is a minimum circumscribed rectangle of the region to be detected in the final target detection model. The detection frame acquisition module 704 is specifically configured to find coordinates of boundary points in the region to be detected to obtain a coordinate set, preferentially select a leftmost or rightmost coordinate point in the coordinate set as a starting point, sort other boundary points in the coordinate set according to polar angles between the starting point and the other boundary points, sequentially add each boundary point to a convex hull of the region to be detected according to the sorting order, select a connecting line of two adjacent points on the convex hull as a first pair of sides, calculate lengths of two sides perpendicular to the first pair of sides, so that an area formed by the first pair of sides and the two sides falls within an initial circumscribed rectangle, determine whether the area is smaller than an area of the initial circumscribed rectangle, update the area of the initial circumscribed rectangle and lengths and widths thereof if the determination result is yes, rotate the first pair of sides to a second pair of sides, and repeat the above process until the first pair of sides is rotated to the second pair of sides, and output a minimum circumscribed rectangle.

[0098] The occlusion reproduction module 705 is configured to load a segmentation detection algorithm on a UAV, detect the region to be detected by using the first detection frame, and reproduce an occluded part of the region to be detected. The occlusion reproduction module 705 is specifically configured to sort targets in an image according to values of the calculated confidence, mark a serial number of a target of interest if the target of interest is found, perform scanning at different angles to obtain first detection frames at the different angles and set the first detection frames as a first image set, and perform panoramic image target splicing technology, perform feature matching on the first image set to obtain a feature matching result, calculate a transformation structure between images according to the feature matching result, perform image superposition according to the transformation structure to implement image mapping, perform feature point alignment by using an image splicing algorithm to implement image registration, perform image segmentation and adaptive selection of a splicing seam according to image content, and perform fusion processing on the images according to a multi-band image fusion technology, so as to detect the region to be detected and reproduce the occluded part of the region to be detected.

[0099] Further, the instance segmentation model-based target edge detection apparatus 700 provided in the embodiment of the present application further includes an integration module configured to put a plurality of first detection frames due to line-of-sight overlap into a second image set, calculate an overlap area, calculate a percentage of the overlap area in a first detection frame with a higher confidence, determine whether the percentage is greater than a preset threshold, regard the plurality of first detection frames as a same target if the determination result is yes, and find a minimum circumscribed rectangle of the target as a second detection frame, and return four vertex coordinate point information of the second detection frame to a UAV, so as to complete an overlap frame integration step.

[0100] Some of the modules in the apparatus described in the present application can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, classes, etc., that perform particular tasks or implement particular abstract data types. The present application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including memory storage devices.

[0101] The apparatus or modules described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. For the convenience of description, the above apparatus is described as various modules with functions. In the implementation of the embodiments of the present application, the functions of the modules can be implemented in one or more software and / or hardware. Of course, the modules implementing certain functions can also be implemented by a combination of multiple sub-modules or sub-units.

[0102] The methods, apparatuses or modules described in the present application can be implemented in a computer-readable program code in any appropriate manner, for example, the controller can take the form of, for example, a microprocessor or a processor, and a computer-readable medium storing computer-readable program code (such as software or firmware) executable by the (micro) processor, logic gates, switches, application specific integrated circuits (Application Specific Integrated Circuit, abbreviated as: ASIC), programmable logic controllers and embedded microcontrollers. Examples of the controller include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that, in addition to implementing the controller in a pure computer-readable program code manner, the same function can also be achieved by logically programming the method steps to implement the controller in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers. Therefore, such a controller can be considered as a hardware component, and the means included therein for implementing various functions can also be regarded as structures within the hardware component. Alternatively, the means for implementing various functions can also be regarded as both software modules implementing the method and structures within the hardware component.

[0103] As Figure 8As shown, the embodiment of the present application further provides a target edge detection server based on an instance segmentation model, comprising a memory 801 and a processor 802; the memory 801 is used for storing computer executable instructions; the processor 802 is used for executing the computer executable instructions to realize the target edge detection method based on the instance segmentation model according to the embodiment of the present application.

[0104] The embodiment of the present application further provides a computer readable storage medium, which stores executable instructions, and when a computer executes the executable instructions, the target edge detection method based on the instance segmentation model according to the embodiment of the present application can be realized.

[0105] From the above description of the embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software and necessary hardware. Based on such an understanding, the technical solutions of the present application can be embodied in the form of a software product or in the form of data migration in the implementation process. The computer software product can be stored in a storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions for causing a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the method according to the embodiments of the present application.

[0106] The embodiments in the specification are described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment mainly describes the differences from other embodiments. The whole or part of the present application can be used in many general or special computer system environments or configurations.

[0107] The above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the present application.

Claims

1. A method for detecting a target edge based on an instance segmentation model, characterized in that, The method comprises the following steps: Preprocessing the collected images, labeling the images after preprocessing and making a data set; Respectively performing corresponding operations on image data information in the training set of the instance segmentation model in the backbone network, neck network and head network of the instance segmentation model to realize training of the instance segmentation model; Detecting the target in the detection area and other areas to obtain an initial target detection model and a corresponding final target detection model; The method for detecting the target in the detection area and other areas to obtain an initial target detection model and a corresponding final target detection model comprises the following steps: Testing the initial target detection model to obtain a test result, re-labeling relevant features obtained from the test result and iteratively training to obtain the final target detection model; Obtaining a first detection frame of the detection area in the final target detection model and using a UAV to obtain vertex coordinates of the first detection frame; Carrying the segmentation detection algorithm on the UAV, detecting the detection area through the first detection frame and reproducing the occluded part of the detection area; 2. The method of claim 1, wherein, The method for carrying the segmentation detection algorithm on the UAV, detecting the detection area through the first detection frame and reproducing the occluded part of the detection area comprises the following steps: Sorting the target in the image according to the calculated value of the confidence, marking the serial number of the target of interest if the target of interest is found, scanning the target from different angles to obtain first detection frames at different angles and setting the first detection frames as a first image set, and performing panoramic image target splicing technology; Performing feature matching on the first image set to obtain a feature matching result, calculating the transformation structure between images according to the feature matching result, performing image superposition according to the transformation structure to realize image mapping, aligning feature points through an image splicing algorithm to realize image registration, performing image segmentation and adaptively selecting a splicing seam according to the image content, and performing fusion processing on the image according to multi-band image fusion technology to realize detection of the detection area and reproduction of the occluded part of the detection area. The method for respectively performing corresponding operations on image data information in the training set of the instance segmentation model in the backbone network, neck network and head network of the instance segmentation model to realize training of the instance segmentation model comprises the following steps:

3. The method of claim 2, wherein, Performing feature extraction in the backbone network of the YOLOV8 model to obtain a feature extraction result, performing convolution operation on the image to scale the image to the same size; Inputting the feature extraction result into the neck network of the YOLOV8 model to perform multi-feature aggregation; Performing regression loss calculation in the head network of the YOLOV8 model to obtain a YOLOV8 model after training of the instance segmentation model. The method for detecting the target in the detection area and other areas to obtain an initial target detection model and a corresponding final target detection model comprises the following steps: Modifying parameters in a script file of the YOLOV8 model after training of the instance segmentation model to obtain an initial target detection model; Model training is performed on the initial target detection model, and relevant hyperparameters are adjusted according to the test effect in the verification set thereof; After all samples in the training set of the initial target detection model are trained for one time, the initial target detection model is tested to obtain a test result; The relevant features obtained by the test result are relabeled, and iterative training is performed to obtain the final target detection model.

4. The method of claim 1, wherein, The first detection frame is the minimum circumscribed rectangle of the to-be-detected region in the final target detection model; Determining the minimum circumscribed rectangle of the to-be-detected region in the final target detection model comprises: Finding the coordinates of the boundary points in the to-be-detected region to obtain a coordinate set, and preferentially selecting the leftmost or rightmost coordinate point in the coordinate set as a starting point; The boundary points in the coordinate set other than the starting point are sorted according to the polar angle size between the starting point and the boundary points; Each boundary point is sequentially added to the convex hull of the to-be-detected region according to the sorted order; The connecting line of two adjacent points on the convex hull is selected as a first pair of sides, and the lengths of two sides perpendicular to the first pair of sides are calculated so that the area formed by the first pair of sides and the two sides falls within the initial circumscribed rectangle; It is judged whether the area is smaller than the area of the initial circumscribed rectangle; If the judgment result is yes, the area of the initial circumscribed rectangle and its length and width are updated; The first pair of sides is rotated to a second pair of sides, and the above process is repeated until the first pair of sides is rotated, and the minimum circumscribed rectangle is output.

5. The method of claim 1, wherein, Further comprising: If there is a situation that the first detection frame overlaps due to line-of-sight overlap, an overlapping frame integration step is performed.

6. The method of claim 5, wherein, The overlapping frame integration step comprises: Placing multiple first detection frames overlapped due to line-of-sight overlap into a second image set and calculating the overlapping area thereof; Calculating the percentage of the overlapping area in the first detection frame with higher confidence, and judging whether the percentage is greater than a preset threshold; If the judgment result is yes, multiple first detection frames are regarded as the same target, and the minimum circumscribed rectangle of the target is found as a second detection frame; The coordinate point information of the four vertices of the second detection frame is transmitted back to the unmanned aerial vehicle, thereby completing the overlapping frame integration step.

7. An instance segmentation model-based object edge detection apparatus, characterized by comprising: The device performs the method of any one of claims 1 to 6, comprising: A data set making module for pre-processing the collected images, labeling the to-be-detected regions of the pre-processed images, and making data sets; An instance segmentation module for performing corresponding operations on image data information in the training set of the instance segmentation model in the backbone network, neck network and head network of the instance segmentation model to realize instance segmentation model training; A target detection module for target detection on the to-be-detected regions and other regions to obtain an initial target detection model and a final target detection model corresponding thereto; An acquisition detection frame module for acquiring a first detection frame of the to-be-detected region in the final target detection model, and acquiring the vertex coordinates of the first detection frame by using an unmanned aerial vehicle. The realization of the shelter recurrence module is used to carry the segmentation detection algorithm on the unmanned aerial vehicle, realizes the detection to the region to be detected through the first detection frame, and realizes the recurrence of the shelter part of the region to be detected. 8.A server for instance segmentation model target edge detection, characterized in that, comprise a memory and a processor; The memory is used to store computer executable instructions; The processor is used to execute the computer executable instructions to realize the method in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores executable instructions, and the computer executes the executable instructions to realize the method in any one of claims 1-6.

Images