Ancient building ridge beast decoration piece inspection method

Abstract

The application discloses an ancient building ridge beast decoration piece inspection method, comprising the following steps: obtaining a training sample set, which comprises a large number of ridge beast images of labeled ridge beast targets; inputting the ridge beast images into a convolutional neural network model for training, and obtaining an optimal detection feature map corresponding to the ridge beast images from the convolutional neural network model; the convolutional neural network model comprises an aggregated convolution layer module for extracting fine-grained features of the ridge beast images and a multi-size convolution prediction head module with an integrated attention mechanism for extracting semantic information of the ridge beast images; a UAV regularly inspects the ancient building ridge beast decoration piece according to set inspection points and routes, takes a group of ridge beast images at each inspection point, uses the trained convolutional neural network model to detect the taken group of ridge beast images, compares and analyzes the detection results of multiple periods, and judges whether the ridge beast decoration piece is damaged or not. The method realizes rapid and automatic inspection of a large range of ridge beast decoration pieces.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of ancient building decoration detection. More particularly, the present application relates to a kind of ancient building ridge beast decoration piece inspection method. BACKGROUND

[0002] Ridge beast is a decorative component placed in the ridge of ancient building, which plays a role in stabilizing the ridge, preventing fire and rain and dispersing lightning, and contains rich symbolic meaning, with important artistic and historical value. Due to factors such as earthquakes and water erosion, ancient building ridge beast decorations are lost or damaged, which damages the aesthetic value of the building as a whole and causes the loss of valuable cultural resources. Regularly inspecting ridge beast decorations and counting their types and quantities can help discover and repair damaged components in a timely manner, which is of great significance to the daily management and maintenance of ancient buildings. Currently, the main inspection method for ridge beast decorations is to regularly dispatch staff for inspection, rely on visual interpretation by the human eye, and manually count the types and quantities of ridge beast decorations to record the locations of damaged or missing pieces. Then, the information is updated and entered into the database management system according to the inspection report. Due to the large number of buildings, large area, and high frequency of conventional inspection (e.g., once a week) in ancient building communities, the current manual inspection method has certain limitations: ① high labor cost, requiring a large number of personnel to participate in the inspection work and training staff to establish their knowledge of ancient buildings, which is costly; ② low efficiency, as ridge beasts are located at a high elevation in ancient buildings and are easily affected by tourists, visual inspection is not efficient; and ③ slow and untimely information update, as manual statistics and records result in slow information update. SUMMARY

[0003] An object of the present application is to solve at least the above problems and / or deficiencies and to provide at least the advantages described later.

[0004] Another object of the present application is to provide an ancient building ridge beast decoration inspection method that achieves rapid and automated inspection of a large range of ridge beast decorations and can automatically determine and locate missing or damaged ridge beast decorations.

[0005] To achieve these objects and other advantages of the present application, a method for inspecting ancient building ridge beast decorations is provided, comprising:

[0006] obtaining a training sample set, which includes a large number of ridge beast images with labeled ridge beast targets;

[0007] input the ridge beast image into a convolutional neural network model for training, and obtain an optimal detection feature map corresponding to the ridge beast image from the convolutional neural network model, the convolutional neural network model comprising: an aggregated convolutional layer module composed of n convolutional layer modules, and a multi-size convolutional prediction head module integrated with an attention mechanism, wherein n is an integer greater than or equal to 1, the aggregated convolutional layer module is used to extract fine-grained features of the ridge beast image, and the multi-size convolutional prediction head module is used to extract semantic information of the ridge beast image;

[0008] The unmanned aerial vehicle periodically inspects the ridge beast decoration piece of the ancient building according to the set inspection point and the flight route, takes a group of ridge beast images at each inspection point, detects the group of taken ridge beast images using the trained convolutional neural network model, compares and analyzes the detection results of multiple periods, and judges whether the ridge beast decoration piece is damaged.

[0009] Preferably, in the aggregated convolutional layer module, the first convolutional layer module is a basic convolutional layer, the second convolutional layer module to the n-th convolutional layer module have a unified structure, and each includes a convolutional layer with a step of 2 and a residual block composed of a plurality of depth aggregation convolutions for skip connection; wherein the specific processing process of the depth aggregation convolution is:

[0010] Step S1, input the feature map T input is HxWxM, a filter set D w is used to calculate a two-dimensional convolution on each of the M channels of the input feature map T input independently and one by one to obtain a feature map T d ; wherein H is the height of the input feature map, W is the width of the input feature map, and M is the number of channels of the input feature map; wherein the filter set D w includes M filters, and each filter has a size of 3x3;

[0011] Step S2, using a KxK sliding window with a sliding step of 1, applying a summation function SUM() to each center pixel on the M channels of the feature map T d , calculating the algebraic sum of the elements in the calculation range to obtain a feature map T sum , and concatenating the input feature map T input and the feature map T sum in the channel dimension to obtain a feature map T concat ;

[0012] Step S3, along the depth direction, using a filter P w to calculate the point convolution depth aggregation feature of the feature map T concat , wherein the filter P w has a size of 1x1;

[0013] Step S4, repeating step S3 F times, the output feature map T obtained output is HxWxF.

[0014] Preferably, the ancient building ridge beast decoration piece inspection method, the specific processing process of the poly convolution layer module is that when the ridge beast image sequentially passes through the first convolution layer module, the second convolution layer module, the third convolution layer module, the fourth convolution layer module and the nth convolution layer module in the poly convolution layer module, the resolution of the input ridge beast image is sequentially reduced to 1 / 2, and the number of channels is sequentially expanded to 2 times, so that the fine-grained features of the ridge beast image are obtained.

[0015] Preferably, the ancient building ridge beast decoration piece inspection method, the multi-size convolution prediction head module includes an SE module for compression operation and excitation operation, and a multi-size convolution structure module for fusing different regional range information, the multi-size convolution structure module includes two standard convolutions of 1x1 and 3x3 connected in parallel, a splicing module for splicing feature maps output by the two standard convolutions connected in parallel, and two 3x3 standard convolutions with filter numbers of C and 2C connected in series in sequence.

[0016] Preferably, the ancient building ridge beast decoration piece inspection method, the specific processing process of extracting semantic information of the ridge beast image by using the multi-size convolution prediction head module is:

[0017] The feature map output by the nth convolution layer module is input into the SE module for processing, and then the first prediction output tensor is obtained after processing by the multi-size convolution structure module, wherein the feature map after the 3x3 standard convolution with the filter number of 2C is output to the first prediction output tensor via the two-dimensional convolution layer with bias, and the feature map after the 3x3 standard convolution with the filter number of C is sent to the first up-sampling convolution module for resolution expansion processing by using the nearest interpolation function.

[0018] The feature map processed by the first up-sampling convolution module is spliced with the feature map output by the nth-1 convolution module, and then the spliced feature map is input into the SE module for processing, and then the second prediction output tensor is obtained after processing by the multi-size convolution structure module, wherein the feature map after the 3x3 standard convolution with the filter number of 2C is output to the second prediction output tensor via the two-dimensional convolution layer with bias, and the feature map after the 3x3 standard convolution with the filter number of C is sent to the second up-sampling convolution module for resolution expansion processing by using the nearest interpolation function.

[0019] The feature map processed by the second up-sampling convolution module is spliced with the feature map output by the n-2 convolution module, and then the spliced feature map is input into the SE module for processing, and then the processed feature map is input into the multi-size convolution structure module for processing to obtain a third prediction output tensor, wherein the feature map after 3*3 standard convolution with a filter number of 2C is output to the third prediction output tensor by a two-dimensional convolution layer with bias.

[0020] After the first prediction output tensor, the second prediction output tensor and the third prediction output tensor are processed by position inverse calculation and non-maximum suppression algorithm, the optimal prediction result is obtained, that is, the category and position of the ridge beast image are extracted.

[0021] Preferably, in the aggregation convolution layer module, n is 6, the first convolution layer module expands the channel number of the input ridge beast image from 3 to 16, and the channel numbers after processing by the second convolution layer module to the sixth convolution layer module are 32, 64, 128, 256 and 512 in sequence; the number of residual blocks in the second convolution layer module is 1, the number of residual blocks in the third convolution layer module is 2, the number of residual blocks in the fourth convolution layer module and the fifth convolution layer module is 4, and the number of residual blocks in the sixth convolution layer module is 2; in step S2, K is 3.

[0022] Preferably, in the SE module, the compression operation adopts a global average pooling operation, and the process of the excitation operation is that the feature map after the global average pooling operation is sequentially input into a first full connection layer, a Relu excitation function, a second full connection layer and a Sigmoid excitation function, and then multiplied by the input feature map matrix before the global average pooling operation to realize the redistribution of image features.

[0023] Preferably, in the ancient building ridge beast decoration piece inspection method, the training sample set includes ridge beast images obtained by fixed-point aerial photography and undamaged ancient building ridge beast images obtained from Internet crowd source data; 14 categories of ridge beast decorations are labeled in the ridge beast image using a labeling tool, including immortals, dragons, phoenixes, lions, sea horses, heavenly horses, fish, tigers, tortoises, bulls, lines, hanging beasts, kissing beasts and sleeve beasts.

[0024] Preferably, in the ancient building ridge beast decoration piece inspection method, the specific process of comparing and analyzing the detection results of multiple periods and judging whether the ridge beast decoration piece is damaged is as follows:

[0025] If more than 70% of the ridge beast image detection results are the same after a group of ridge beast images photographed at an inspection point are detected by the trained convolutional neural network model, it is determined that the inspection point is stably detectable, and the result is retained as the detection result of the inspection point in the current period.

[0026] The detection result of the inspection point in the current period is compared with the detection result of the inspection point in the last period, specifically: the intersection over union of each ridge beast decoration part in the last period detection result and the current period detection result is calculated in sequence, if the intersection over union is not less than the set threshold, the matching is successful, the detection state of the ridge beast decoration part of the inspection point is updated, if there is a ridge beast decoration part with an intersection over union less than the set threshold, the matching fails, and a damage warning is given.

[0027] Preferably, before the ridge beast image is input into the convolutional neural network model for training, the size of the 9 anchor boxes is determined by the K-Means clustering algorithm, the size of the input ridge beast image is 640*640, and the scaling method is used to unify the irregular images.

[0028] The present application at least includes the following beneficial effects:

[0029] I. The unmanned aerial vehicle periodically inspects the large-scale ancient building ridge beast decoration part according to the set inspection point and route, collects multiple clear ridge beast images at each inspection point, uses the trained convolutional neural network model to detect and identify the collected ridge beast images, obtains the stable detection result of the ridge beast decoration part, extracts the type and quantity of the ridge beast decoration part, and then compares the detection results of multiple periods to determine whether the ridge beast decoration part is damaged, thereby realizing the rapid and automatic inspection of the large-scale ridge beast decoration part.

[0030] II. The deep aggregation convolution used in the present application constructs the residual block structure in the aggregation convolution layer module, which can make the constructed convolutional neural network model learn more distinctive features of the ridge beast image.

[0031] III. In the multi-size convolution prediction head module, first, the SE module is used to model the multi-scale feature map channel relationship, then multi-size convolution structure is used after the SE module, that is, 1*1 and 3*3 standard convolution are connected in parallel to fuse multi-scale features, the feature maps output by the two standard convolutions connected in parallel are spliced, and then two 3*3 standard convolutions with filter numbers of C and 2C are connected in series to further extract semantic information.

[0032] Other advantages, objects and features of the present application will be partially embodied by the following description, and partially understood by those skilled in the art through research and practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 Structure relationship diagram of convolutional neural network model in one embodiment of the present application;

[0034] Figure 2 Ancient building ridge beast decoration piece inspection method flowchart in one embodiment of the present application;

[0035] Figure 3 Unmanned aerial vehicle inspection point and flight route diagram in one embodiment of the present application;

[0036] Figure 4 Corresponding diagram of various types of ridge beasts in one embodiment of the present application;

[0037] Figure 5 Structure diagram of second convolutional layer module to sixth convolutional layer module in one embodiment of the present application;

[0038] Figure 6 Calculation process diagram of deep aggregation convolution in one embodiment of the present application;

[0039] Figure 7 Receptive field expansion process diagram in one embodiment of the present application;

[0040] Figure 8 Relationship diagram of multi-size convolutional structure module in one embodiment of the present application;

[0041] Figure 9 Relationship diagram of SE module in one embodiment of the present application;

[0042] Figure 10 Relationship diagram of up-sampling convolutional module in one embodiment of the present application;

[0043] Figure 11 Detection effect diagram of using trained convolutional neural network model in one embodiment of the present application;

[0044] Figure 12(a) is a result diagram of using YOLOv3 method for detection in the prior art;

[0045] Figure 12(b) is a result diagram of using the method of the embodiment of the present application for detection;

[0046] Figure 13(a) is a result diagram of using YOLOv3 method for detection in the prior art;

[0047] Figure 13(b) is a result diagram of using the method of the embodiment of the present application for detection. DETAILED DESCRIPTION

[0048] The application will be further described in detail below with reference to the accompanying drawings, so that those skilled in the art can implement the application according to the description.

[0049] Figures 1-2 As shown in the drawings, the embodiment of the application provides an ancient building ridge beast decoration piece inspection method, which comprises the following steps:

[0050] S100, a training sample set is acquired, and the training sample set comprises a large number of ridge beast images of labeled ridge beast targets.

[0051] The convolutional neural network model needs a large number of data samples as a basis for training, and the optimal detection effect can be achieved by optimizing the model parameters. Therefore, the embodiment of the application adopts rich ridge beast data sets such as actual aerial images and Internet crowd source data to construct the training sample set of the ridge beast image.

[0052] The specific steps of acquiring the data set are as follows:

[0053] a, acquiring actual aerial images: along the direction of the black arrow shown in the drawings, adjusting the best lens angle, and segmenting the video data, segmenting the video stream frame number and frame rate, and selecting fixed interval single-frame images as candidate samples. Figure 3

[0054] b, acquiring Internet crowd source data: combining multiple keywords, batch capturing related images from open source libraries and search engines, and selecting ancient building ridge beast images that are not damaged and meet the format (JPEG / PNG / BMP) as candidate samples.

[0055] c, accurate labeling of ridge beast targets: using a labeling tool to accurately label the ridge beast target in each candidate sample, and the categories and numbers of the ridge beast are as shown in the drawings. Figure 4

[0056] S101, inputting the ridge beast image into a convolutional neural network model for training, and acquiring an optimal detection feature map corresponding to the ridge beast image from the convolutional neural network model, the convolutional neural network model comprising: an aggregated convolutional layer module composed of n convolutional layer modules and a multi-size convolutional prediction head module integrated with an attention mechanism, wherein n is an integer greater than or equal to 1, the aggregated convolutional layer module is used to extract fine-grained features of the ridge beast image, and the multi-size convolutional prediction head module is used to extract semantic information of the ridge beast image.

[0057] ​​The size of the 9 anchor boxes is determined by the K-Means clustering algorithm before the ridge beast image is input into the convolutional neural network model for training, the input ridge beast image has a size of 640*640, and the scaling method is used to unify the irregular image. The specific processing process is: taking the longest side of the image as the reference side to calculate the scaling coefficient of 640 for image scaling, and then using RGB(128, 128, 128) to fill the image into a regular square; the data augmentation technology is combined for expansion, and the specific method includes: mirror transformation; random offset in horizontal or vertical direction; color jitter in HSV color space.

[0058] In a specific implementation, when the convolutional neural network model is used to train the ridge beast image, the Adam optimizer is used to train the convolutional neural network model of the ridge beast image, and the set parameters are: the initial learning rate is set to 0.0001, beta_1 is 0.9, beta_2 is 0.999, the learning rate decay rate is 0.5, and the model parameters are saved when the loss is lowest as the final state.

[0059] In a specific implementation, n is preferably 6, and the aggregation convolution layer module includes 6 convolution layer modules. The convolution layer module can be used for image feature extraction, and the multi-layer convolution layer module can obtain deeper feature maps, so that the six convolution layer modules are used to extract image fine-grained features in the embodiment of the application.

[0060] The attention mechanism is a series of attention allocation coefficients, that is, a series of weight parameters, which can be used to emphasize or select important information of the target processing object and suppress some irrelevant detailed information. The prediction head can be used to detect targets of different scales, so that the multi-size convolution prediction head module with the attention mechanism can make the image features more directional, and can also fuse multi-scale features and extract semantic information. Therefore, when the ridge beast image is input into the convolutional neural network model for training, the optimal detection feature map corresponding to the ridge beast image can be obtained from the convolutional neural network model.

[0061] S102, the unmanned aerial vehicle regularly inspects the ridge beast decoration of the ancient building according to the set inspection point and the flight route, takes a group of ridge beast images at each inspection point, uses the trained convolutional neural network model to detect the group of ridge beast images, compares and analyzes the detection results of multiple periods, and judges whether the ridge beast decoration is damaged.

[0062] In a specific implementation, before the unmanned aerial vehicle inspects, the inspection point and the flight route need to be initialized, and the design of the inspection point and the corresponding flight route is as follows: Figure 3As shown, a simple example is given, the square is the ridge beast position of the ancient building roof, the circle is the unmanned aerial vehicle inspection point, and the arrow is the flight direction. After on-site investigation of the site and full understanding of the flight conditions of the airspace, the hand-controlled unmanned aerial vehicle device flies to P1 to P8 points in the clockwise direction in turn to perform air hovering, and the coordinates of the point positions are recorded in sequence in the flight control system as the subsequent inspection point positions. In the automatic inspection process of the unmanned aerial vehicle, the recorded inspection point coordinates are called.

[0063] The specific process of comparing and analyzing the detection results of multiple periods and judging whether the ridge beast decoration part is damaged or not is:

[0064] a. If more than 70% of the ridge beast image detection results are the same after a group of ridge beast images taken at an inspection point are detected by the trained convolutional neural network model, it is determined that the inspection point is stably detected, and the result is retained as the detection result of the inspection point in the current period;

[0065] b. The detection result of the inspection point in the current period is compared with the detection result of the inspection point in the last period, specifically: the intersection over union is calculated for each ridge beast decoration part of the last period detection result and the current period detection result in sequence, if the intersection over union is not less than the set threshold value, the matching is successful, the detection state of the ridge beast decoration part of the inspection point is updated, and if there is a ridge beast decoration part with an intersection over union less than the set threshold value, the matching fails, and a damage warning is given.

[0066] In specific implementation, for the ridge beast target in the first period of aerial image, the detection state, the class and the position coordinates are initialized; the second period of aerial image is obtained, the trained convolutional neural network model is used for detection, all detection results are obtained, based on the first period detection state, each ridge beast decoration part is matched from the second period detection result according to the intersection over union threshold value, the intersection over union threshold value is set to 0.5, if the matching is successful, the detection state of the inspection point is updated, otherwise a damage warning is given, and the third period, and so on. It should be noted that the detection effect diagram using the trained convolutional neural network model is as shown in Figure 11 As shown, each ridge beast decoration part has a rectangular frame, by calculating the intersection over union of each ridge beast decoration part of the inspection point in adjacent two inspection periods, the change of the ridge beast decoration part of the inspection point can be easily judged.

[0067] Therefore, the ancient building ridge beast decoration part inspection method adopted by the embodiment of the application can realize rapid and automatic inspection of a large range of ridge beast decoration parts.

[0068] In order to more clearly illustrate the specific implementation process of obtaining the optimal detection feature map from the convolutional neural network model in the last embodiment, in another specific embodiment of the present application, the aggregation convolutional layer module in step S101, the first convolutional layer module 1 is a basic convolutional layer, that is, a conventional convolutional layer or a standard convolution, the purpose of which is to expand the number of channels of the input image to obtain more abundant information, and the channel number of the input dinosaur image is expanded from 3 to 16, the second convolutional layer module 2, the third convolutional layer module 3, the fourth convolutional layer module 4, the fifth convolutional layer module 5 and the sixth convolutional layer module 6 have a uniform structure, which includes a convolutional layer with a step of 2 and a residual block composed of N depth aggregation convolutions for skip connection, as shown in Figure 5 .

[0069] In the above embodiment, when N is 1, the number of residual blocks arranged in the convolutional layer module is 1, when N is 2, the number of residual blocks arranged in the convolutional layer module is 2, and the value of N is the same as the number of residual blocks arranged in the convolutional layer module. In the specific implementation, the number of residual blocks in the second convolutional layer module is set to 1, the number of residual blocks in the third convolutional layer module is set to 2, the number of residual blocks in the fourth convolutional layer module and the fifth convolutional layer module is set to 4, and the number of residual blocks in the sixth convolutional layer module is set to 2, so that the network learns the residual of the inter-layer potential mapping to maintain the linear mapping, so as to solve the problem of network degradation with the increase of depth.

[0070] The convolutional layer module performs down-sampling on the feature map, and the resolution of the feature map is reduced by 1 / 2 and the number of channels is expanded by 2 times each time. Therefore, when the input dinosaur image size is 416x416x3, the output feature map size is 416x416x16 after the first convolutional layer module 1, the output feature map size is 208x208x32 after the second convolutional layer module 2, the output feature map size is 104x104x64 after the third convolutional layer module 3, the output feature map size is 52x52x128 after the fourth convolutional layer module 4, the output feature map size is 26x26x256 after the fifth convolutional layer module 5, and the output feature map size is 13x13x512 after the sixth convolutional layer module 6.

[0071] As shown in Figure 6 , when constructing the residual block, the specific processing process of the depth aggregation convolution applied is as follows:

[0072] Step 201, input feature map T input is HxWxM, and a filter set D w is used to independently calculate a two-dimensional convolution on each of the M channels of the input feature map T input to obtain a feature map Td Where H is the height of the input feature map, W is the width of the input feature map, and M is the number of channels in the input feature map; where the filter bank D w It includes M filters, each filter having a size of 3×3;

[0073] Step 202: Using a 3×3 sliding window with a sliding step size of 1, respectively on feature map T d On each of the M channels, the summation function SUM() is applied to the center pixel to calculate the algebraic sum of the elements within the range, resulting in the feature map T. sum The input feature map T input With feature map T sum Feature map T is obtained by concatenating the data along the channel dimension. concat ;

[0074] Step S203: Along the depth direction, use filter P w Calculate feature map T concat The point convolutional depth aggregation features, wherein the filter P w The size is 1×1;

[0075] Step S204: Repeat step S203 F times to obtain the output feature map T. output It is H×W×F.

[0076] In the specific implementation process, regarding depthwise convolution, since the input feature map T... input With feature map T sum The number of channels is M, therefore, the spliced ​​feature map T concat The number of channels is 2M. When using filter bank P... w Processing feature map T concat At that time, 1 P w This will result in a feature map, which can be used with multiple P... w This will result in multiple feature maps, i.e., multiple feature map channels. Specifically, in step S204, regarding the number of times step S203 is repeated, F is 32 in the second convolutional layer module, 64 in the third convolutional layer module, 128 in the fourth convolutional layer module, 256 in the fifth convolutional layer module, and 512 in the sixth convolutional layer module.

[0077] As can be known from the specific processing procedure of the depth polymetric convolution, in the embodiment, the depth polymetric convolution can make the network learn more different features of the ridge beast. Based on the feature aggregation idea, the depth polymetric convolution (DPC) is proposed by improving the depth separable convolution by using a local summation function, and a residual structure in the polymetric convolution layer module is constructed using the depth polymetric convolution, thereby reducing the parameter quantity and improving the context information utilization rate. For the depth separable convolution, the local summation function is used to aggregate internal features, expand the convolution receptive field, and enhance the feature difference, then the input features and the aggregated features are spliced, the point convolution is used to integrate the original single-pixel features and the field features, and the depth polymetric features are learned. In step S202, SUM() is a summation function, and if there are matrices As Figure 7 shown, the depth polymetric convolution uses a summation function to aggregate D w The convolution features do not bring additional parameters, but can combine features in a larger area, thereby increasing the flow of information between channels and the difference of depth features. The depth polymetric convolution enables the convolutional neural network model to learn depth polymetric features, and the convolution weight coefficients in the kernel determine which local signals are combined, which can enhance the extraction ability of fine-grained features to some extent.

[0078] In another specific embodiment, the ancient building ridge beast decoration piece inspection method, the multi-size convolution prediction head module 7 includes a SE module performing compression operation and excitation operation, and a multi-size convolution structure module fusing information of different area ranges, as Figure 8 shown, the multi-size convolution structure module 7 includes two standard convolutions of 1×1 and 3×3 connected in parallel, a splicing module splicing feature maps output by the two standard convolutions connected in parallel, and two 3×3 standard convolutions of filter numbers C and 2C connected in sequence.

[0079] In the above embodiment, as Figure 9As shown, the compression operation in the SE (Squeeze and Excitation) module adopts a global average pooling operation, and the process of the excitation operation is as follows: the feature map after the global average pooling operation sequentially passes through a first full connection layer, a Relu excitation function, a second full connection layer, and a Sigmoid excitation function, and then is multiplied by the input feature map matrix before the global average pooling operation, to realize the redistribution of image features. The first full connection layer has 10% of the input feature dimension as the number of neurons, and the second full connection layer has the same number of neurons as the input feature dimension. The SE module is a kind of channel domain attention, which first pools and squeezes the global channel features to generate channel descriptors, then the channel descriptors pass through a full connection neural network for calculation and transmission, model the mutual relationship between channels, and are excited to be channel weight coefficients that can express the relative importance, and finally the matrix multiplication is performed to realize the redistribution of features. The SE module finally uses the Sigmoid excitation function to map each weight coefficient to a scaling factor in the interval (0, 1), so that the response of the interference signal can be suppressed, and the features are more directional.

[0080] To better fuse information in different regional ranges, a multi-size convolution structure module is used after the SE module. Two standard convolutions of 1x1 and 3x3 connected in parallel can fuse multi-scale features, and then perform splicing. Further, two 3x3 standard convolutions with filter numbers of C and 2C connected in series can further extract semantic information. The standard convolution Convblock is composed of a Con2D convolution layer, a Batch Normalization layer (Batch Normalization), and a LeakyRelu excitation function layer. The feature map after the 3x3 standard convolution with a filter number of C is sent to an up-sampling convolution module (as shown in Figure 10 After the resolution is expanded using the nearest interpolation function, the large-scale features are spliced to form a feature pyramid, so as to improve the utilization rate of features and improve the detection effect of multi-scale targets. The feature map after the 3x3 standard convolution with a filter number of 2C is sent to a two-dimensional convolution layer with a bias to predict an output tensor.

[0081] In the convolutional neural network model, the specific processing process of using the multi-size convolution prediction head module to extract semantic information of the dinosaur image is as follows: Figure 1

[0082] ​S301, the feature map output by the sixth convolutional layer module is input into the SE module for processing, and then the first prediction output tensor is obtained after processing by the multi-size convolutional structure module, wherein the feature map after 3x3 standard convolution with a filter number of 2C is used to predict the first prediction output tensor by a two-dimensional convolution layer with bias, and the feature map after 3x3 standard convolution with a filter number of C is sent to the first up-sampling convolution module 8 for resolution expansion processing by using a nearest-neighbor interpolation function.

[0083] S302, the feature map processed by the first up-sampling convolution module 8 and the feature map output by the fifth convolutional module 5 are spliced, and then the spliced feature map is input into the SE module for processing, and then the second prediction output tensor is obtained after processing by the multi-size convolutional structure module, wherein the feature map after 3x3 standard convolution with a filter number of 2C is used to predict the second prediction output tensor by a two-dimensional convolution layer with bias, and the feature map after 3x3 standard convolution with a filter number of C is sent to the second up-sampling convolution module 9 for resolution expansion processing by using a nearest-neighbor interpolation function.

[0084] S303, the feature map processed by the second up-sampling convolution module 9 and the feature map output by the fourth convolutional module 4 are spliced, and then the spliced feature map is input into the SE module for processing, and then the third prediction output tensor is obtained after processing by the multi-size convolutional structure module, wherein the feature map after 3x3 standard convolution with a filter number of 2C is used to predict the third prediction output tensor by a two-dimensional convolution layer with bias.

[0085] S304, after the first prediction output tensor, the second prediction output tensor and the third prediction output tensor are processed by position inverse calculation and non-maximum suppression algorithm, the optimal prediction result is obtained, that is, the category and position of the ridge beast image are extracted.

[0086] In the above embodiment, the first prediction output tensor, the second prediction output tensor and the third prediction output tensor are prediction values of the dinosaur class and the relative position, and after position inversion of the three prediction values, all prediction results of the image are obtained, and then all prediction results are processed by using a non-maximum suppression algorithm to filter low confidence and redundant predictions and to reserve final prediction results. Specifically, when the feature map size output by the sixth convolutional block is 13x13x512, the first prediction output tensor obtained is 13x13x3x(4+1+C), when the feature map size output by the fifth convolutional layer block is 26x26x256, the second prediction output tensor output is 26x26x3x(4+1+C), and when the feature map size output by the fourth convolutional layer block is 52x52x128, the third prediction output tensor output is 52x52x3x(4+1+C), wherein C is 14, which is the number of dinosaur species, and 4+1 represents position information. After detection of the dinosaur image by the convolutional neural network model, type information and position information of the dinosaur can be obtained. In the processing process of the above embodiment, the cubic interpolation method is used for the adjacent interpolation function.

[0087] As described above, the ancient building dinosaur decoration piece inspection method provided by the embodiment of the application realizes rapid and automatic inspection of a large range of dinosaur decoration pieces, and automatically judges and positions missing and damaged dinosaur decoration pieces.

[0088] Compared with the method YOLOv3 used in the prior art, the mAP is increased by 3.05% and the parameter amount is decreased by about 70% by using the convolutional neural network model provided by the embodiment of the application for detection, as shown in the quantitative comparison table, which effectively reduces the missed detection and false detection of small-size dense dinosaur pieces and can meet the rapid inspection demand of dinosaur decoration pieces. The qualitative comparison is shown in FIG. 12(a), FIG. 12(b), FIG. 13(a) and FIG. 13(b), wherein FIG. 12(a) and FIG. 13(a) are detection effect diagrams of YOLOv3, and FIG. 12(b) and FIG. 13(b) are detection effect diagrams of the embodiment of the application. As can be seen from the diagrams, the method used in the embodiment of the application can detect small-size dense dinosaur decoration pieces and will not cause missed detection.

[0089] Quantitative comparison table

[0090]

[0091] Although the embodiments of the present application have been disclosed as above, it is not limited to the application listed in the specification and the embodiments. It can be fully applied to various fields suitable for the present application. Additional modifications can be easily realized by those skilled in the art. Therefore, the present application is not limited to specific details and examples shown and described herein without departing from the general concept defined by the claims and the equivalent scope.

Claims

1. A method for inspecting ridge beast decorative parts of ancient buildings, characterized in that, include: Obtain a training sample set, which includes a large number of annotated vertebrate images; The image of the vertebrate animal is input into a convolutional neural network model for training, and the optimal detection feature map corresponding to the image of the vertebrate animal is obtained from the convolutional neural network model. The convolutional neural network model includes: an aggregated convolutional layer module consisting of n convolutional layer modules and a multi-size convolutional prediction head module incorporating an attention mechanism, where n is an integer greater than or equal to 1. The aggregated convolutional layer module is used to extract fine-grained features of the vertebrate animal image, and the multi-size convolutional prediction head module is used to extract semantic information of the vertebrate animal image. The drone regularly inspects the roof ornaments of ancient buildings according to the set inspection points and routes. At each inspection point, a set of roof ornament images are taken. The trained convolutional neural network model is used to detect the set of roof ornament images. The detection results of multiple periods are compared and analyzed to determine whether the roof ornaments have been damaged. In the aggregated convolutional layer module, the first convolutional layer module is the basic convolutional layer, and the second to nth convolutional layer modules have a unified structure, all including a convolutional layer with a stride of 2 and a residual block formed by skip-layer connections of several depth aggregated convolutions; wherein, the specific processing of the depth aggregated convolution is as follows: Step S1: Input feature map For H×W×M, use a filter bank In the input feature map respectively Two-dimensional convolutions are independently computed on each of the M channels to obtain the feature map. Where H is the height of the input feature map, W is the width of the input feature map, and M is the number of channels in the input feature map; where the filter bank It includes M filters, each filter having a size of M. ; Step S2, using A sliding window with a step size of 1 is used to navigate between feature maps. Apply a summation function to each center pixel across the M channels. Calculate the algebraic sum of the elements within the range to obtain the feature map. Input feature map With feature map The feature map is obtained by concatenating the data along the channel dimension. ; Step S3: Apply a filter along the depth direction. Calculate feature map The point convolutional depth aggregation features, wherein the filter The size is 1×1; Step S4: Repeat step S3 F times to obtain the output feature map. for; The multi-size convolutional prediction head module includes an SE module that performs compression and activation operations, and a multi-size convolutional structure module that fuses information from different regions. The multi-size convolutional structure module includes: two standard convolutions, 1×1 and 3×3, connected in parallel; a stitching module that stitches together the feature maps output by the two standard convolutions; and two standard convolutions, 3×3, with filter numbers of C and 2C respectively, connected in series in sequence.

2. The method for inspecting ridge beast decorative parts of ancient buildings as described in claim 1, characterized in that, The specific processing procedure of the aggregated convolutional layer module is as follows: when the vertebrate image passes through the first convolutional layer module, the second convolutional layer module, the third convolutional layer module, the fourth convolutional layer module and up to the nth convolutional layer module in the aggregated convolutional layer module, the resolution of the input vertebrate image is reduced to 1 / 2 in sequence, and the number of channels is expanded to 2 times in sequence, thereby obtaining the fine-grained features of the vertebrate image.

3. The method for inspecting ridge beast decorative parts of ancient buildings as described in claim 1, characterized in that, The specific processing procedure for extracting semantic information from vertebrate images using the multi-size convolutional prediction head module is as follows: The feature map output by the nth convolutional layer module is input into the SE module for processing, and then processed by the multi-size convolutional structure module to obtain the first prediction output tensor. The feature map after 3×3 standard convolution with 2C filters is predicted and output as the first prediction output tensor by a two-dimensional convolutional layer with bias. The feature map after 3×3 standard convolution with C filters is fed into the first upsampling convolution module and the resolution is increased by using the nearest neighbor interpolation function. The feature map processed by the first upsampling convolution module is concatenated with the feature map output by the (n-1)th convolution module. The concatenated feature map is then input into the SE module for processing, and then processed by the multi-size convolution structure module to obtain the second prediction output tensor. The feature map after a 3×3 standard convolution with 2C filters is predicted and output as the second prediction output tensor by a two-dimensional convolutional layer with bias. The feature map after a 3×3 standard convolution with C filters is fed into the second upsampling convolution module, where the resolution is increased using the nearest neighbor interpolation function. The feature map processed by the second upsampling convolution module is concatenated with the feature map output by the (n-2)th convolution module. The concatenated feature map is then input into the SE module for processing. After processing by the multi-size convolution structure module, the third prediction output tensor is obtained. The feature map after 3×3 standard convolution with 2C filters is predicted and output as the third prediction output tensor by a two-dimensional convolution layer with bias. After processing the first, second, and third predicted output tensors using position inverse calculation and nonmaximum suppression algorithms, the optimal prediction result is obtained, which extracts the category and location of the vertebrate image.

4. The method for inspecting the ridge beast decorations of ancient buildings as described in claim 2, characterized in that, In the aggregated convolutional layer module, n is 6. The first convolutional layer module expands the number of channels of the input vertebrate image from 3 to 16. The number of channels processed by the second to sixth convolutional layer modules are 32, 64, 128, 256, and 512 respectively. The number of residual blocks in the second convolutional layer module is 1, the number of residual blocks in the third convolutional layer module is 2, the number of residual blocks in the fourth and fifth convolutional layer modules is 4, and the number of residual blocks in the sixth convolutional layer module is 2. In step S2, K is 3.

5. The method for inspecting ridge beast decorative parts of ancient buildings as described in claim 1, characterized in that, The compression operation in the SE module uses global average pooling. The activation process is as follows: the feature map after global average pooling passes through the first fully connected layer, the ReLU activation function, the second fully connected layer, and the Sigmoid activation function in sequence, and then is multiplied with the input feature map matrix before the global average pooling operation to achieve the redistribution of image features.

6. The method for inspecting ridge beast decorative parts of ancient buildings as described in claim 1, characterized in that, The training sample set includes aerial images of roof ridge beasts taken from fixed points and images of undamaged roof ridge beasts from ancient buildings obtained from crowdsourced data on the Internet. The roof ridge beast images are labeled with 14 categories of roof ridge beast decorations, including immortals, dragons, phoenixes, lions, seahorses, celestial horses, xiyu (a type of fish), suanni (a mythical lion-like creature), xiezhi (a mythical beast), douniu (a type of bull), xingshi (a type of mythical beast), chuisheng (a type of mythical beast), wenseng (a type of mythical beast), and taoseng (a type of mythical beast).

7. The method for inspecting ridge beast decorative parts of ancient buildings as described in claim 1, characterized in that, The specific process of comparing and analyzing the test results from multiple periods and determining whether the ridge beast ornaments have been damaged is as follows: If, after a set of images of vertebrate animals taken at a patrol point is detected by a trained convolutional neural network model, more than 70% of the images show the same detection result, then the patrol point is determined to be a stable detection point, and the result is retained as the detection result of the patrol point in this cycle. The detection results of this inspection point in the current cycle are compared with the detection results of the same inspection point in the previous cycle. Specifically, the cross-union ratio (CUI) of each ridge beast decoration from the previous cycle is calculated and matched with the detection results of this cycle in sequence. If the CUI is not less than the set threshold, the match is successful and the detection status of the ridge beast decoration at this inspection point is updated. If there are ridge beast decorations with an CUI less than the set threshold, the match fails and a damage warning is issued.

8. The method for inspecting ridge beast decorative parts of ancient buildings as described in claim 1, characterized in that, Before inputting the vertebrate images into the convolutional neural network model for training, the size of the nine anchor boxes in the input vertebrate images is determined by the K-Means clustering algorithm. The size of the input vertebrate images is 640×640. The scaling method is used to unify the irregular images.

Images