Method for identifying country dwellings in view of unmanned aerial vehicle
Through neural network, the tilted photography residential images collected by drones were processed, and the edge features were extracted, which solved the problem of drones measuring the size of residential houses in rural transportation, achieved high-precision and widely applicable residential size prediction, and improved transportation safety Sex and efficiency.
Patent Information
- Application Number
- CN202410948007.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-08
- Publication Date
- 2024-11-08
AI Technical Summary
The prior art is difficult to accurately measure the length, width and height dimensions of houses in rural transportation of drones, especially when the drone heading is not fixed, which limits the measurability and accuracy of the measurement of house sizes.
Using a neural network-based method, the images of houses are collected through tilt photography, edge extraction and Hoff line transformation are performed, and the length, width and height edge characteristics of houses are extracted, and the house size prediction model is established to realize the online prediction and identification of house size.
It realizes accurate prediction of the three-direction dimensions of houses of various specifications, with wide applicability, high prediction accuracy, and strong robust model, and can accurately measure under uncertain heading, simplifying operation steps and improving real-time performance.
Smart Images

Figure CN118918463A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of unmanned aerial vehicle (UAV) rural transportation application, and specifically relates to a rural residential identification method under the field of view of an unmanned aerial vehicle (UAV) and a rural flight control method based on the size of the residential buildings. Background Art
[0002] The application of drones in rural transportation has broad potential and practical value. Drones are widely used in many fields, including medical supplies distribution, agricultural monitoring and management, express package delivery, post-disaster rescue and reconstruction, environmental monitoring and protection, infrastructure inspection, education and publicity, etc.
[0003] In terms of infrastructure inspection and maintenance, drones are used to inspect power lines, communication base stations, bridges and other infrastructure. In terms of e-commerce logistics, drones can provide fast delivery services for small packages, letters and express deliveries. In the future, it is expected to support the logistics and distribution of rural e-commerce platforms, shorten the circulation time of agricultural products and daily necessities; at the same time, it will provide reverse logistics transportation, provide farmers with convenient agricultural product sales channels, and quickly transport agricultural products to urban markets.
[0004] Drone transportation has the advantages of high efficiency and wide coverage. Drones can reach the destination quickly and save time, especially in rural areas with inconvenient transportation. Drones can also provide a full range of services because they can cover inconvenient or remote rural areas.
[0005] In rural transportation, whether it is a normal landing or an emergency landing, drone docking needs to consider safety, convenience and efficiency. In order to avoid rural houses or choose a suitable docking point in the target residential environment during drone rural transportation, accurate house length, width and height measurement data is required as the basis for planning navigation. On the one hand, during the flight, when setting the minimum flight altitude of the drone, it is necessary to judge the height of rural houses; on the other hand, when flying around houses or landing on roofs, it is necessary to measure the size and length of the houses to ensure a safe lateral and longitudinal distance.
[0006] In the prior art, there are several solutions for measuring buildings. For example, the invention patent application with application number 201811029714.X discloses a spatial measurement method in the fields of intelligent buildings, surveying and mapping, in which a laser emits a light pulse to the measured target, and the distance is obtained by the flight time required for the measured object to reflect. Then, a camera is used to obtain a measurement image to measure the distance between two points in the image; this method requires that the camera optical axis is facing the target plane, and can only measure the distance between two points on the single-orientation target plane. The invention patent application with application number 202310233220.8 discloses a method for identifying, obtaining and calculating the ratio of the length unit to the pixel value of a CAD drawing, using an AI method to automatically obtain the pixel value on the CAD drawing and the number of pixels corresponding to it on the drawing, thereby calculating the ratio of the length of the pixel to the actual space, and automatically calculating the length of the actual space represented by the pixels of the architectural CAD image; obviously, this method requires a priori ratio of the length of the pixel to the actual space, that is, an actual length needs to be measured or marked in advance. The invention patent application with application number 202310416333.1 discloses a field survey method for photovoltaic power stations. It uses any variety of tools such as infrared level, tape measure, tape measure, and laser rangefinder to measure the actual area and length and width of the roof and cross-check, so that users can judge whether a photovoltaic power station needs to be installed. Through on-site photography, on-site measurement and background system review, it is judged whether the roof is suitable for installing a photovoltaic power station based on the surrounding obstacles and the condition of the roof. It can be seen that this application also needs to use actual measurement methods to obtain the size of the roof.
[0007] The invention patent application with application number 202110395821.X discloses a method for monitoring building height based on high-resolution optical remote sensing satellite images based on corner points. It uses spatial distribution to pair corner points, and then uses the geometric model of shadow imaging to calculate the height of the building. It uses the geometric relationship between the total length of the building's shadow on the remote sensing image, the solar altitude angle, the solar azimuth angle and the building height.
[0008] From the above technical solutions, it can be seen that the current image-based measurement of house size is either based on prior data measured on the spot or directly calculated based on pixel ratio. Among them, the latter relies on an accurate proportional relationship, that is, the pixel width value of the measured line is required to correspond to a priori determined ratio. Obviously, when this solution is applied to the drone system, it requires that the drone video screen is fixed at the same relative azimuth angle to be comparable, which undoubtedly greatly limits the measurability of residential size.
[0009] Therefore, a method and system for identifying the length, width and height of rural houses with strong generalization performance and wide applicability is needed. Summary of the invention
[0010] The purpose of the present invention is to provide a method, system and flight control method for identifying rural houses in the field of vision of a drone, which are used to identify the characteristics of rural houses based on images in the field of vision of the drone, and then measure their height, length and width of the roof, etc., without limiting the drone to maintain a certain precise orientation when sampling images.
[0011] In order to solve the above technical problems, the present invention provides a method for identifying rural houses in the field of vision of a drone, comprising the following steps:
[0012] S1, initialization, establishing a neural network for predicting rural residential size;
[0013] S2, determine whether the current state is offline network training or online size prediction, if it is the latter, go to S9, otherwise go to S3;
[0014] S3, using oblique photography to collect images of rural houses of different sizes from the drone's field of view at a certain height, and record the actual length, width and height of the houses;
[0015] S4, selecting a residential area from the rural residential image, removing the part outside the residential area to form a residential target image, performing edge extraction on the residential target image to obtain a residential edge image;
[0016] S5, performing Hough line transform on the residential edge image, and obtaining two relative edge lines of the residential building that sandwich the long edge, the wide edge, or the high edge as the target edge and are approximately perpendicular to the target edge based on the surfaces of the residential building, respectively, to form a residential edge line image and record the corresponding relationship between the target edge and its two relative edge lines;
[0017] S6, according to any two relative edge lines obtained in the residential edge line image, calculating the geometric feature quantity of each edge line relative to the image center; recording the geometric feature quantity of the two relative edge lines and the size of the corresponding target edge as a training sample;
[0018] S7, determine whether the current training sample set needs to be supplemented, if so, go to step S3, otherwise go to step S8;
[0019] S8. Based on the obtained training sample set, a rural residential size prediction neural network is trained to obtain a trained residential size prediction neural network as a residential size prediction model;
[0020] S9, in an online state, perform residential size identification in an actual rural scene; detect whether there is an exit instruction, if so, proceed to step S14, otherwise, proceed to the next step S10 after the residential size identification instruction is triggered;
[0021] S10, acquiring a rural residential image, determining a ROI region containing residential buildings, extracting residential buildings from the ROI region as foreground objects, and setting all non-foreground object regions in the rural residential image to zero to form a residential building target image;
[0022] S11, extracting the edge of the residential target image to obtain the residential edge image;
[0023] S12, performing Hough line transform on the residential edge image, and obtaining a residential edge image including three pairs of edge lines in the residential outline edge through heuristic search, wherein the first pair of edge lines of the three pairs of edge lines are two opposite edge lines of the top and bottom of the residential building with a pair of side edges in between, and the second pair of edge lines and the third pair of edge lines of the three pairs of edge lines are two opposite edge lines with a pair of horizontal long sides and horizontal wide sides in between, respectively;
[0024] S13, from the residential boundary image, sequentially obtain geometric features of two opposite boundary lines relative to the image center for any pair of the three boundary lines, input the geometric features into the trained residential dimension prediction model, obtain a residential dimension prediction value, and output the dimension prediction value; and then go to step S9;
[0025] S14. End the rural residential size prediction.
[0026] Preferably, in step S3, rectangular residential models of various specifications are built, and residential image samples are obtained by obliquely photographing the models from different directions;
[0027] In the step S4, after retaining only the residential area portion of the acquired residential image sample, grayscale and binarization processing is performed, and the edge of the binarized residential target image is extracted to obtain a residential edge image.
[0028] Preferably, in step S3, the size of the cuboid used in the residential model is a first preset multiple S1 of the actual residential size, and the multiple S1 is set according to the drone's rural cruising flight altitude and camera resolution; the size difference between cuboids of different specifications corresponds to S1 times the actual residential size difference SD. Preferably, the first preset multiple S1 is between 1 / 50 and 1 / 20, and the actual residential size difference SD is 0.5 to 3 meters.
[0029] Preferably, in step S3, during the collection of the residential image sample set, the residential models are arranged in different areas of the ground field of view, so that the residential models in the sample set are distributed in each square of the nine-square grid formed by the top, middle, bottom, left, middle and right in the image plane.
[0030] Preferably, in step S13, the coordinate information and orientation information of the current residential location are also obtained, and after the size of the residential building is determined, the size of the residential building and the corresponding coordinate and orientation information are output.
[0031] Preferably, in step S6, the slope of each edge line and the vertical distance from the image center to the edge line are calculated as the geometric feature quantity of each edge line relative to the image center; and the slopes k of the two relative edge lines are calculated. 1 , k 2 And the perpendicular distance D from the center of the image to the two opposite side lines 1 , D 2 A total of 4 geometric features are used as inputs of the training samples, and the size of the target side corresponding to the two relative side lines, i.e., the length, width, or height of the residential building, is used as the output of the training samples;
[0032] In step S13, the slopes k of the two relative edge lines are obtained. 1 , k 2 And the perpendicular distance D from the center of the image to the two opposite side lines 1 , D 2 A total of 4 geometric features are used as geometric features relative to the image center and input into the trained residential size prediction model to obtain a predicted value of the residential size; wherein, if the two relative side lines correspond to the first, second or third pair of side lines, the residential size is the height, length or width of the residential size accordingly.
[0033] Preferably, in step S6, the method for calculating and extracting geometric features for any pair of edge lines among the three pairs of edge lines comprises:
[0034] Assume that the coordinates of the two endpoints of a residential boundary are (x 1 ,y 1 )、(x 2 ,y 2 ), then its slope is k = y 1 -y 2 / x 1 -x 2 ;
[0035] The vertical distance from the center of the image to the edge of the residential building is
[0036] In the formula, (x 0 ,y 0 ) represents the coordinates of the center of the image, A and B are the coefficients of the general formula of the straight line equation, C is the constant term and: A = y 2 -y 1 , B = x 1 -x 2 , C = -x 1 (y 2-y 1 )+y 1 (x 2 -x 1 );
[0037] According to the calculation formula of k and d, the slope k of any pair of edge lines at the edge of the residential area is calculated respectively. 1 , k 2 And the perpendicular distance D from the center of the image to the two side lines 1 , D 2 A total of 4 quantities are used as the geometric features of the dwellings in the rural dwelling images.
[0038] Preferably, in step S8, training the neural network for predicting the size of rural houses includes: T1, initializing the weights and biases of the network; T2, reading the network parameters and the training sample set recorded in the document; T3, normalizing the sample data; T4, calculating the error between the predicted output value and the expected value of each sample; T5, correcting the network weights and biases by reverse calculation based on the error; T6, if the conditions for ending the network training are met, ending the training, otherwise going to T4 to continue the training; T7, analyzing and outputting the results.
[0039] Preferably, the step S12 comprises: after performing Hough line transform on the residential edge image, selecting the straight line segments with the longest length in the vertical and horizontal directions as candidate straight line segments according to the polar coordinate space parameter characteristics in the Hough line transform, and determining two relative edge lines sandwiching a pair of long sides, wide sides and high sides according to the orientation characteristics of the candidate straight line segments, as a pair of edge lines of the length, width and height of the residential building, and forming three pairs of edge lines of the residential building together;
[0040] For each sideline of any pair of the three pairs of sidelines, in combination with the corresponding residential edge image, the coordinates of the two end points of the straight line are used to describe it, and the slope of the straight line and the perpendicular distance from the center point of the image to the straight line are respectively calculated based on the two end points.
[0041] Preferably, in the heuristic search of step S12, when the angle is represented by the slope, when the absolute value of the difference between the slopes of the two edge lines is less than a very small first slope threshold, the two edge lines are merged; at the same time, if the angle tangent value or the slope is greater than a very large second slope threshold, the edge line is determined to be a vertical line perpendicular to the horizontal axis.
[0042] Preferably, the step S12 comprises: when performing Hough line transform on each edge pixel point in the residential edge image, accumulating and obtaining a ρ-θ plane matrix corresponding to the image, wherein ρ represents the distance of the vertical segment of the straight line from the origin, and θ is the angle of the vertical line to the horizontal axis;
[0043] Based on the accumulated ρ-θ plane matrix, search for the two or three longest edges in a range close to the vertical direction as the side edges of the residential building;
[0044] In the range of the non-side edge direction, the plane matrix element values, i.e., the lengths of the straight line segments, are sorted from large to small, and two secondary straight line segments close to the main straight line segment in direction are searched in sequence from the main straight line segment with the largest length, and the main straight line segment and the corresponding two secondary straight line segments form a candidate straight line group;
[0045] For any straight line segment in the candidate straight line group, calculate the distance between it and the upper and lower endpoints of the side edge of the residential building. If the minimum value of the distance corresponds to the distance between the straight line segment and the upper endpoint of a side edge, then mark the straight line segment as a roof candidate straight line, otherwise mark it as a ground candidate straight line. Loop from straight lines in different directions to obtain multiple groups of candidate straight line groups.
[0046] Preferably, the step S12 further comprises: in one of the candidate straight line groups, the first pair of sidelines are formed by two relative sidelines, namely, the residential roof candidate straight line and the ground candidate straight line, which sandwich a pair of side edges. In one of the candidate straight line groups, the second pair of sidelines and the third pair of sidelines are formed by two relative sidelines, namely, the two residential roof candidate straight lines, which sandwich a pair of other straight line segments; or, in the side edges of the residential buildings, the second pair of sidelines and the third pair of sidelines are formed by two relative sidelines, namely, the two residential side edges, which sandwich a pair of other straight line segments.
[0047] Preferably, in step S1, a target classification model based on the ASFF-YOLOv5s network is also established.
[0048] After collecting rural scene images with a bird's-eye view camera, the network is offline trained using an image sample set consisting of rural scene images with rural residential areas labeled to obtain a rural scene target classification model;
[0049] In step S10, the rural scene target classification model performs recognition processing on the rural scene image including residential houses, i.e., the rural residential house image, to obtain a rural residential area anchor frame, and uses the area within the anchor frame as the ROI area of the rural residential houses.
[0050] Preferably, the ASFF-YOLOv5s network adds three layers 24, 25, and 26 between the 23rd and the next layer of the YOLOv5s network. These three layers receive the outputs of the 17th, 20th, and 23th layers as input, respectively, and after the ASFF adaptive spatial feature fusion module performs feature fusion of different scales, they replace the original 17th, 20th, and 23th layers in turn and input them into the 24th layer of the original output layer, i.e., the 27th layer of the new network.
[0051] Preferably, the step S12 comprises: when performing the Hough line transform on each edge pixel point in the residential edge image, accumulating to obtain the ρ-θ plane matrix corresponding to the image, wherein ρ represents the distance of the vertical segment of the straight line from the origin, and θ is the angle of the vertical line to the horizontal axis; in the accumulating, the foreground pixel point (u, v) in the image and a straight line element l i,j (ρ i,j ,θ i,j ) The corresponding judgment method is:
[0052] Take a point H(ρ i,j ·cosθ i,j , ρ i,j sinθ i,j ), recorded as H(u′, v′), calculate J = |vv′-(uu′)·tanθ i,j |, if |J|≤ε, then the pixel point is determined to be on the straight line and accumulated, where ε is a preset small positive number.
[0053] Preferably, multiple residential size prediction models based on BP neural network are established offline, and each residential size prediction model corresponds to a flight altitude and a bird's-eye view angle of the image acquisition unit; when the residential size prediction is performed online in step S12, the geometric features obtained after the current acquisition process are input into the residential size prediction model corresponding to the flight altitude and the bird's-eye view angle during acquisition in step S10, and the model output value is obtained as the residential size prediction value.
[0054] Preferably, in the BP neural network of the residential size prediction model, the input layer receives the slopes k of the straight lines on both sides of the residential building from the general processing module. 1 and k 2 , the perpendicular distance D from the center of the image to the straight lines on both sides of the residential building 1 and D 2 , as well as the flight altitude and the overlooking angle during collection, a total of 6 input quantities, and the output of the output layer is the size of the residential buildings.
[0055] In another embodiment of the present invention, a rural flight control method based on residential size is provided, comprising the following steps:
[0056] R1. Initialization, establishing a rural residential size prediction neural network to map the geometric features of residential boundary lines in oblique photography images to residential sizes, collecting training samples and training the rural residential size prediction neural network to obtain a residential size prediction model;
[0057] R2. In response to the residential size recognition command during flight, take oblique photos of the rural scene to obtain images of rural residential buildings;
[0058] R3, determining a ROI region including residential buildings in the rural residential image, extracting residential buildings from the ROI region as foreground objects, and setting all non-foreground object regions in the rural residential image to zero to form a residential building target image;
[0059] R4. For the obtained residential target image, edge detection is performed to obtain a residential edge image; Hough line transform is performed on the residential edge image, and a residential edge image containing two straight line edges of the residential building is obtained through heuristic search;
[0060] R5. Perform Hough line transform on the residential edge image, and obtain a residential edge image including three pairs of edge lines in the residential outline edge through heuristic search, wherein the first pair of edge lines of the three pairs of edge lines are two opposite edge lines of the top and bottom of the residential building with a pair of side edges in between, and the second pair of edge lines and the third pair of edge lines of the three pairs of edge lines are two opposite edge lines with a pair of horizontal long sides and horizontal wide sides in between respectively;
[0061] R6. From the residential boundary image, for any pair of the three pairs of boundary lines, obtain the geometric feature quantities of the two opposite boundary lines relative to the image center; input the geometric features into the trained residential dimension prediction model to obtain the residential dimension prediction value, and output the dimension prediction value;
[0062] R7, determine whether the path planning is to fly over the current residential building or to stop and land. If it is to fly over, control the minimum flight altitude to be greater than the residential building height in the current residential building size, which is a set threshold; otherwise, if it is to stop and land, go to step R8;
[0063] R8, determine whether the length and width of the target parking site in the current residential size data are both greater than the preset safety distance threshold. If so, take the center of the target parking site as the landing point and go to step R9; otherwise, issue an alarm to indicate that the site size is insufficient and go to step R10;
[0064] R9, sending the landing point location information to the flight control platform, after which the flight control platform controls the drone to land, proceed to step R11;
[0065] R10, fly to the next rural residence, during which the trained target classification model based on the deep network is used to identify and process the rural image in the current field of view. When a rural residence target is identified in the field of view, a residential size recognition instruction is issued, and the process goes to step R2;
[0066] R11. End rural flight control based on house size.
[0067] Preferably, in step R8, after finding a site with a sufficient safety distance, residential images are collected from multiple angles and size prediction is performed to check the safety distance of the site. Preferably, the size of the residential buildings can also be displayed in the form of a list on a display device, and the operator selects one of the residential buildings as the target stop point.
[0068] Preferably, in step R8, the identified residential size is displayed in segments on the display screen in the form of a curve, and the coordinates and positions corresponding to the identified size are marked on the curve. Preferably, the heading information is obtained through the positioning unit, and the direction identified based on the heading, such as the due north direction, and the direction corresponding to the axis of the residential roof are also recorded and marked in the output position information.
[0069] Compared with the prior art, the scheme of the present invention has the following advantages: the present invention aims at the unpredictable relative orientation relationship between the posture of the drone when collecting images and the orientation of the houses, analyzes the characteristic representation of the size of the houses in the rural house images in the image space, decomposes the size of the houses into three values of length, width and height in three-dimensional directions, and proposes to use two relative sidelines respectively sandwiching a pair of long sides, wide sides and high sides to describe the size of the houses in three directions respectively, and uses the four geometric features of the slope of the two relative sidelines of the rural houses in any direction and the vertical distance from the center point of the image to the two sidelines as constraints to establish a neural network to predict the size of the houses. Samples are collected to identify and train the nonlinear mapping relationship between the size of rural dwellings and the four proposed geometric features. Each sample can select rural dwellings by classifying and identifying targets in aerial images. The three pairs of edge lines corresponding to the length, width and height of the edge of the dwelling outline are obtained by Hough line transform screening. The four geometric features are extracted for two edge lines in any pair of extracted edge lines, and the results are input into the trained dwelling size prediction model to make online predictions of the length, width and height of rural dwellings. The residential size prediction model based on neural network established by the present invention has good generalization performance, can accurately predict the three-dimensional sizes of residential houses of various specifications according to the rural residential images collected from the uncertain navigation direction, and has a wide range of applications. Since the features are extracted from the straight line and do not rely on the precise pixel start and end points, the prediction accuracy is high; at the same time, since the established model is highly robust, when extracting the residential edge lines, the step size in the accumulation matrix of the Hough line transform does not need to be set too small, and the correspondence between the edge points and the straight line elements in the matrix simplifies the judgment, reduces the amount of calculation, speeds up the search speed, and further improves the real-time recognition of rural residential houses; and based on the heuristic search, the horizontal edges and vertical side edges of the ground and roof of the residential houses can be automatically found and marked, simplifying the operation confirmation steps.
[0070] It should be understood that all combinations of the aforementioned concepts and the additional concepts discussed in more detail below (provided that such concepts are not mutually inconsistent) can be considered as part of the inventive subject matter disclosed herein. In particular, all combinations of the claimed subject matter appearing in the disclosed solution can be considered as part of the inventive subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS
[0071] Figure 1 This is a schematic diagram of a rural residential scene;
[0072] Figure 2 The structure diagram of the device and system for identifying rural residential buildings in the field of vision of a drone using the method of the present invention is shown in FIG.
[0073] Figure 3A This is the structure diagram of the host unit / controller in the rural residential identification system under the vision of the drone. Figure 3B To obtain and preprocess the module structure diagram, Figure 3C This is the structural diagram of the feature extraction module;
[0074] Figure 4A This is a schematic diagram of the neural network model structure of the residential size prediction module. Figure 4B It is a schematic diagram of the partial structure of the host unit / controller in the method of identifying rural houses under the field of view of drones;
[0075] Figure 5A This is a schematic diagram of the drone's posture during sample collection. Figure 5B This is a schematic diagram for comparing residential models of different sizes;
[0076] Fig. 6A , Figure 6B This is a schematic diagram of residential models photographed at different angles during sample collection. Figure 6C This is a schematic diagram of the single-sided outline treatment of residential buildings. Fig.6D The residential model image after Hough line transformation straight line detection;
[0077] Fig. 7A , Fig. 7E The schematic diagram of the proposed residential model edge constraints and geometric features is shown in Figure 2. Figure 7B , Figure 7C , Fig.7D The diagrams of length, width and height constraints and geometric features of rural houses are as follows;
[0078] Figure 8 Fitting curve between actual and predicted values of residential model width in the test set;
[0079] Fig. 9 Fitting curve between actual and predicted values of residential model length in test set;
[0080] Fig.10Fitting curve between actual value and predicted value of residential model height in test set;
[0081] Fig.11 Schematic diagram of target annotation for rural drone field of view images;
[0082] Fig.12 This is a schematic diagram of small, medium and large-scale targets in rural scenes;
[0083] Fig.13A Local diagram of YOLOv5s network configuration file, Fig. 13B This is a schematic diagram of the feature fusion of ASFF-3 and learning of features at different scales. Fig. 13C , Fig.13D They are the partial diagram of the ASFF-YOLOv5s network configuration file and the schematic diagram of the network structure;
[0084] Fig.14A , Fig. 14B A schematic diagram comparing the loss curves of the training set and validation set before and after improving the network;
[0085] Fig.15 This is a schematic diagram of the residential target anchor frame and residential size identification in the drone field of view image.
[0086] Fig.16 This is a schematic diagram of the processing flow of the rural residential identification method under the vision of drones.
[0087] Among them: 1000 rural residential identification systems under the vision of drones, 900 rural residential identification devices under the vision of drones, 100 host units / controllers, 200 user interface units, 300 handheld terminals, 400 servers, 500 image acquisition units, 600 flight control platforms, 700 positioning units, 800 rural residential environments / scenes;
[0088] 110 processor, 120 storage module, 130 input and output module; 111 general processing module, 112 acquisition and preprocessing module, 113 feature extraction module, 114 iterative learning module, 115 residential size prediction module;
[0089] 1121 image acquisition unit, 1122 target detection unit, 1123 residential area extraction unit; 1131 edge detection unit, 1132 straight line extraction (screening) unit, 1133 slope feature extraction unit, 1134 distance feature extraction unit, 1135 side edge extraction unit, 1136 top and bottom edge extraction unit, 1137 roof parallel line extraction unit; 1141 first connection matrix, 1142 second connection matrix; 1101 neural network model, 131 input unit, 132 output unit; 210 display screen, 220 operation panel. DETAILED DESCRIPTION
[0090] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, but the present invention is not limited to these embodiments. The present invention covers any substitution, modification, equivalent method and scheme made within the spirit and scope of the present invention.
[0091] Embodiment 1:
[0092] When drones are performing missions in rural areas, they need to measure the height of residential buildings to set the minimum flight altitude when flying over villages, and they need to measure the length and width of residential buildings when landing on rooftops or flying over houses. Therefore, identifying and measuring the size of rural residential buildings in the field of vision of drones is the basis of rural application of drones.
[0093] like Figure 1 As shown in the figure, in rural scenes, there are mostly farmland and vegetable fields with residential houses scattered among them. Drones provide transportation for rural areas and can provide landing sites on the roofs of residential houses or on the flat ground around residential houses. To ensure the safety of drones flying over and landing, it is necessary to understand the size of rural residential houses to make flight and landing decisions.
[0094] Since rural houses are often not of the same specifications, their sizes vary; and the property rights also belong to different entities, it is difficult to collect size data a priori. To this end, the present invention provides a method for identifying rural houses in the field of vision of a drone, which does not rely on prior three-dimensional size measurement and can estimate the size of rural houses by collecting images of rural houses under a non-fixed heading during flight.
[0095] like Fig.16 As shown, the present invention provides a method for identifying rural residential buildings in the field of vision of an unmanned aerial vehicle, comprising the following steps:
[0096] S1, initialization, establishing a neural network for predicting rural residential size;
[0097] S2, determine whether the current state is offline network training or online size prediction, if it is the latter, go to S9, otherwise go to S3;
[0098] S3, using oblique photography to collect images of rural houses of different sizes from the drone's field of view at a certain height, and record the actual length, width and height of the houses;
[0099] S4, selecting a residential area from the rural residential image, removing the part outside the residential area to form a residential target image, performing edge extraction on the residential target image to obtain a residential edge image;
[0100] S5, performing Hough line transform on the residential edge image, and obtaining two relative edge lines of the residential building that sandwich the long edge, the wide edge, or the high edge as the target edge and are approximately perpendicular to the target edge based on the surfaces of the residential building, respectively, to form a residential edge line image and record the corresponding relationship between the target edge and its two relative edge lines;
[0101] S6, according to any two relative edge lines obtained in the residential edge line image, calculating the geometric feature quantity of each edge line relative to the image center; recording the geometric feature quantity of the two relative edge lines and the size of the corresponding target edge as a training sample;
[0102] S7, determine whether the current training sample set needs to be supplemented, if so, go to step S3, otherwise go to step S8;
[0103] S8. Based on the obtained training sample set, a rural residential size prediction neural network is trained to obtain a trained residential size prediction neural network as a residential size prediction model;
[0104] S9, in an online state, perform residential size identification in an actual rural scene; detect whether there is an exit instruction, if so, proceed to step S14, otherwise, proceed to the next step S10 after the residential size identification instruction is triggered;
[0105] S10, acquiring a rural residential image, determining a ROI region containing residential buildings, extracting residential buildings from the ROI region as foreground objects, and setting all non-foreground object regions in the rural residential image to zero to form a residential building target image;
[0106] S11, extracting the edge of the residential target image to obtain the residential edge image;
[0107] S12, performing Hough line transform on the residential edge image, and obtaining a residential edge image including three pairs of edge lines in the residential outline edge through heuristic search, wherein the first pair of edge lines of the three pairs of edge lines are two opposite edge lines of the top and bottom of the residential building with a pair of side edges in between, and the second pair of edge lines and the third pair of edge lines of the three pairs of edge lines are two opposite edge lines with a pair of horizontal long sides and horizontal wide sides in between, respectively;
[0108] S13, from the residential boundary image, sequentially obtain geometric features of two opposite boundary lines relative to the image center for any pair of the three boundary lines, input the geometric features into the trained residential dimension prediction model, obtain a residential dimension prediction value, and output the dimension prediction value; and then go to step S9;
[0109] S14. End the rural residential size prediction.
[0110] like Figure 2As shown, a rural residential identification device 900 in the field of view of a drone using the method of the present invention comprises a host unit 100 and a user interface unit 200; wherein the host unit 100 comprises a processor 110, a storage module 120 and an input / output module 130; the host unit 110 is configured to: obtain a rural residential image, determine an ROI region containing residential buildings in the rural residential image, extract residential buildings from the ROI region as a residential building target image; obtain a residential building edge image by edge detection of the residential building target image; perform Hough line transformation on the residential building edge image, and obtain three pairs of residential building contour edges by heuristic search. A residential boundary line image of the boundary line, wherein the first pair of the three pairs of boundary lines are two opposite boundary lines of the top and bottom of the residential building with a pair of side edges in between, and the second pair of boundary lines and the third pair of boundary lines of the three pairs of boundary lines are two opposite boundary lines with a pair of horizontal long sides and horizontal wide sides in between respectively; from the residential boundary line image, for any pair of the three pairs of boundary lines in turn, obtain geometric features of the two opposite boundary lines relative to the image center, input the geometric features into a trained residential size prediction model based on a neural network, obtain a residential size prediction value, and output the size prediction value by the output unit 132 in the input-output module 130.
[0111] See also Figure 1 As shown, in the images collected in the rural scenes, the residential houses are approximately displayed as a rectangular shape; and generally, when the drone collects images, the lateral direction of the imaging surface is parallel to the horizontal plane, and the longitudinal axis in the middle of the imaging surface is approximately parallel to the image formed by the plumb line. However, due to the perspective transformation effect, in the image, any two parallel edges or side edges of the actual residential houses appear to converge at a vanishing point at infinity. On the one hand, it is obvious that when measuring the distance from the pixel points at both ends of the edge segment of the residential house in the image and then converting it back to the actual width according to the proportional relationship, the proportional relationship is difficult to accurately determine, and the ratio will be directly related to the front-back or left-right position of the edge in the image. On the other hand, referring to Figure 1 In actual applications, the scenes shown in the left and right figures cannot force the drone to face a precise direction of a certain surface of the residential building in order to perform proportional measurement, that is, to shoot directly at the residential building to collect images. Otherwise, the applicability of the identification will be greatly limited.
[0112] Although the physical size of rural houses in the field of view is certain, when it is sampled from different headings, its edge in the image space is uncertain in terms of pixel width and direction. This certain physical size and uncertain image features are in a contradictory relationship that is difficult to unify. Therefore, this contradiction needs to be decoupled and modeled. The heading of the drone is not fixed during flight, and the orientation of the houses varies greatly; therefore, the relative relationship between the heading of the drone and the orientation of the houses is unpredictable. In order to improve the versatility and applicability of the size identification of houses, it is necessary to study how to characterize the relationship between the two in various relative orientations in the image space, that is, what kind of features in the image can determine the size of the houses?
[0113] After observing the length, width and height edges of residential buildings in various directions in the image, it is found that the key lies in the two straight lines of residential edge on both sides of the length, width or height edge; but the residential edge itself is not the only way to determine the size of the residential building, and the angle of the straight line itself can only reflect the relative angle difference between it and the heading. After a large number of experiments and tests, it is also found that the value of the length, width or height, that is, the relative distance between the two straight lines of residential edge on both sides, will affect the imaging effect of the two parallel sides in the image. So, how to characterize the characteristics of the two sides? Through further research, hypothesis and reasoning, it is proposed to use the slope k of the two parallel lines at different spacings 1 , k 2 And the perpendicular distance D from the center point of the image to the straight lines on both sides of the straight line 1 , D 2 A total of 4 geometric feature inputs are used as constraints for the distance between two lines, i.e., the size of one of the orientation edges of the houses sandwiched by them, in the image space. Furthermore, how to correspond this constraint to the size of the houses? Since there is a nonlinear and coupled mapping relationship between the two, inspired by the fitting application of neural networks in system nonlinear modeling, the present invention proposes to use neural networks to establish a prediction model for the size of houses.
[0114] A neural network consists of an input layer, a hidden layer, and an output layer. Figure 4A As shown, the residential size prediction model of the present invention adopts a three-layer BP neural network with one hidden layer, and its input layer receives the slope k of a pair of side lines corresponding to the length, width, or height of the residential building from the general processing module respectively. 1 , k 2 And the perpendicular distance D from the center point of the image to the pair of edge lines 1 , D 2 There are 4 geometric feature inputs in total, and the output of the output layer is the length, width or height of the residential building corresponding to the pair of edge lines.
[0115] To extract the above four input quantities or geometric features of rural houses in the image space, it is necessary to first obtain the two straight lines corresponding to the length, width or height of the houses from the image, that is, the two edge lines. Figure 3A As shown, in the processor 110 of the host unit or controller 100, multiple processing modules are set, namely, a general processing module 111, an acquisition and preprocessing module 112, a feature extraction module 113, an iterative learning module 114 and a residential size prediction module 115. Among them, the general processing module 111 responds to the message and the operation of the user interface unit, and starts the identification of residential houses in the collected image, schedules the identification processing flow, inputs the rural residential images collected in real time into the acquisition and preprocessing module 112, and outputs the size data identified and predicted by the residential size prediction model in the residential size prediction module 115.
[0116] The acquisition and preprocessing module 112 acquires the rural residential image taken by the image acquisition unit 500, removes high-frequency noise through filtering, and then determines the ROI area including the residential area according to deep learning features or color texture and other features; removes the part outside the ROI area in the original rural residential image to form a residential target image, and then forms a binary residential edge image through edge detection; after performing Hough line transformation on the binary residential edge image, the straight line segments with the largest length in the vertical and horizontal directions are respectively selected as candidate straight line segments according to the polar coordinate space parameter features in the Hough line transformation, and According to the orientation features of the candidate straight line segments, two relative sidelines with a pair of long sides, wide sides and high sides in between are determined respectively as a pair of sidelines of the length, width and height of the residential buildings, and together form three pairs of sidelines of the residential buildings; for each sideline of any pair of sidelines in the three pairs of sidelines, the feature extraction module combines the corresponding binary residential edge image, describes it with the coordinates of the two end points of the straight line, and obtains the slope of the straight line and the perpendicular distance from the center of the image to the straight line based on the two end points; the slope of the straight line on both sides of the residential building and the perpendicular distance from the center of the image to the straight line are used as the geometric features of the residential buildings in the rural residential building image. Preferably, the slope of the straight line can be obtained according to the Hough line transformation accumulation, that is, the angle corresponding to the straight line in the accumulator array.
[0117] like Figure 3BAs shown, in the acquisition and preprocessing module 112, it is further divided into an image acquisition unit 1121, a target detection unit 1122, and a residential area extraction unit 1123. The image acquisition unit 1121 acquires the rural residential image acquired in real time by the external image acquisition unit 500 through the general processing module 111, or acquires the pre-stored residential image from the storage module 120 in an offline state. The residential area extraction unit 1123 preliminarily determines the ROI area including the residential area in the image; preferably, a target classification model based on the YOLOv5s network can be established in the target detection unit 1122, and after the rural scene image is acquired by the overhead camera in the image acquisition unit, the network is offline trained with an image sample set composed of the rural scene images labeled with various target areas such as rural residential areas to obtain a rural scene target classification model; when running online, refer to Fig.15 As shown, the residential area extraction unit 1123 uses the rural scene target classification model to identify and process the rural scene image including residential houses, i.e., the rural residential image, to obtain a rural residential area anchor frame, and uses the area within the anchor frame as the ROI area of the rural residential houses.
[0118] In the YOLOv5 model series, the YOLOv5s model has the smallest size and the least training parameters. Comparative experiments show that YOLOv5s can balance recognition accuracy and image processing frame rate, and is suitable for deployment on mobile devices. The classic YOLOv5s network structure is divided into three parts. The first part is Backbone, which is called the backbone network part, used to extract features from input images; the second part is Neck, which is used to extract features of different scales from network structures of different depths in Backbone for feature fusion, so that feature information of different depths can be retained to the maximum extent, which is more conducive to the learning of network model parameters during training; the third part is Output, that is, output prediction, which mainly predicts the input image with tensors of different scales after feature fusion.
[0119] In rural transportation scenes, there are often some targets with rural characteristics, such as high-voltage towers, puddles, small rivers, small bridges, signal towers, houses and greenhouses. Under non-inclement weather conditions, a large number of target images in rural scenes are obtained through drone aerial photography. In order to improve the richness of the data set, the images are taken in sunny weather and cloudy weather, so that different lighting conditions are also taken into account. After the data set is collected, the targets to be inspected in the image are annotated so that the images can be supervised when they are sent to the neural network training. Fig.11It shows the marking of the target object as a residential building; after marking all the target objects, click the Save button to complete the labeling of a picture and save it. The labeled file is saved in txt format, and the number of label files corresponds to the number of picture files. Each line in the label file represents a target position in the picture, and the saved information is from left to right, including the target type, the x-coordinate of the target center point, the y-coordinate of the target center point, the width of the target, and the height of the target; the data uses normalized values.
[0120] The data set is randomly divided into a training set and a validation set according to a certain ratio. After training the training set, the validation set is used to validate the network. The results show that the AP of residential buildings on the validation set is not high, only 0.669. Therefore, after analyzing the characteristics of the sample and the characteristics of the deep network, it is decided to start with the characteristics of the sample to improve the performance of the classification network.
[0121] In rural scenes, the size of objects varies significantly. Fig.12 As shown, the high-voltage tower, residential buildings and small river correspond to small targets, medium targets and large targets in the image respectively. Based on this feature, the present invention proposes to introduce the ASFF adaptive spatial feature fusion module into the deep learning network to enhance its prediction and recognition capabilities for features of different scales.
[0122] like Fig. 13B As shown in Figure 2, the process of learning weights of feature maps of different scales in the ASFF module is as follows: 1) Upsampling the feature maps of Level 1 and Level 2 to the same scale as Level 3, and the transformation process is recorded as X 1→3 , X 2→3 and X 3→3 ; 2) Use a 1*1 convolution kernel to compress the number of channels at three scales to a uniform length; 3) Concatenate the compressed number of channels; 4) Use a 1*1 convolution kernel to further compress the number of channels to 3; 5) Obtain the weight information of the three different scale spaces through the Softmax classification function, i.e., α 3 , β 3 and γ 3 For the i-th layer, the corresponding ASFF fusion feature formula is:
[0123] ASFF i =X 1→i α i +X 2→i β i +X 3→i γ i
[0124] A partial picture of the network configuration file after adding ASFF is shown below Fig. 13C As shown. Fig.13A , Fig. 13CAs shown, the ASFF-YOLOv5s network established by the rural scene target classification model of the present invention adds three layers 24, 25, and 26 between the 23rd and the next layer of the YOLOv5s network. These three layers receive the outputs of the 17th, 20th, and 23th layers as inputs, respectively, and after the features of different scales are fused by the ASFF adaptive spatial feature fusion module, they replace the original 17th, 20th, and 23th layers in turn and are input to the 24th layer of the original output layer, i.e., the 27th layer of the new network. Fig.13D Schematic diagram of the ASFF-YOLOv5s network structure.
[0125] After the ASFF-YOLOv5s model is trained, its performance is compared with the classic YOLOv5s. Fig.14A , Fig. 14B The training set loss curve and validation set loss curve of the two are compared. It can be seen that the loss value of ASFF-YOLOv5s is significantly lower than that of the YOLOv5s benchmark model, and the convergence speed is also significantly faster. When the training round is 100 epochs, the train_loss of ASFF-YOLOv5s and YOLOv5s are 0.1437 and 0.1140 respectively, and the val_loss are 0.0951 and 0.0915 respectively.
[0126] Compared with the decrease in the loss value of the network model, we are more concerned about the detection accuracy of ASFF-YOLOv5s on the validation set for each category. The comparison of AP of each category before and after the improvement of the network shows that the AP value of the ASFF-YOLOv5s model on the validation set is improved compared with the classic YOLOv5s benchmark model, among which the high-voltage tower is improved by 3.9%, the puddle is improved by 9.9%, the small river is improved by 5%, the small bridge is improved by 7.8%, the residential house is improved by 8%, the signal tower is improved by 8%, the greenhouse is improved by 3.8%, and the mAP is improved by 6.5% overall; it can be seen that the improved network model has significantly improved the detection accuracy of targets of various sizes.
[0127] After the target classification model based on the ASFF-YOLOv5s network locates the residential target with an anchor frame to identify the ROI area of the residential target, the residential target image is extracted to facilitate the subsequent extraction of the residential edge straight line.
[0128] like Figure 3C As shown, in the feature extraction module 113, an edge detection unit 1131, a straight line extraction and screening unit 1132, a side edge extraction unit 1135, a top and bottom edge extraction unit 1136, a roof parallel line extraction unit 1137, a slope feature extraction unit 1133, and a distance feature extraction unit 1134 are also set.
[0129] For residential target images, the edge detection unit 1131 performs edge extraction on the residential target images. Edge extraction can be performed directly on the binary residential target images, and for color or grayscale residential target images, the color images are first converted into single-channel images, and then the Canny edge detection operator is used to detect edges.
[0130] The straight line extraction and screening unit 1132 extracts the edge straight lines of the straight boundaries of the residential buildings, including the long sides, wide sides and high sides, from the binary residential edge image. First, analyze how to reflect the long sides, wide sides and high sides of the residential buildings? If you directly look for the corresponding line segments, then if the end point position of the line segment moves, it will cause obvious errors; more importantly, even if a boundary line segment is obtained, how do you know its actual length? After modeling analysis, the present invention describes the boundary edge with two relative edge lines on both sides of any boundary edge of length, width and height, perpendicular to the boundary edge and sandwiching the boundary edge. Secondly, analyze how to extract such relative edge lines. The present invention adopts Hough line transform.
[0131] Hough line transform maps points in image space to curves in parameter space, such as polar coordinate space, and then finds the intersection of curves in parameter space to detect straight lines. In image space, the equation of a straight line can be expressed as: y = kx + b; using polar coordinate notation, the equation of a straight line can be expressed as: ρ = xcosθ + ysinθ, where ρ is the vertical distance from the straight line to the origin of the coordinate system, and θ is the angle between the perpendicular line of the straight line and the positive direction of the x-axis.
[0132] In order to detect straight lines, a plane matrix, namely an accumulator array (ρ, θ), is created in the ρ-θ plane based on the Hough line transform, and the parameter space is divided into discrete grids; then, for each edge point (x, y) in the image space, the ρ value of ρ = xcosθ + ysinθ under different θ is calculated, and the corresponding position in the accumulator array is voted; after voting, peak detection is performed, and the element with a larger value in the accumulator array is the detected straight line parameter (ρ, θ). Simultaneously, a heuristic search is performed to obtain a residential edge image including three pairs of edge lines in the residential contour edge.
[0133] Specifically, refer to Figure 1 As shown, the drone is in a level flight state when collecting images; therefore, generally, the wall edge of the plumb bob on the ground appears to be approximately parallel to the longitudinal axis in the image. According to the heuristic rule, the side edge extraction unit 1135 searches for the two or three longest side lines as the side edges of the residential building within the range of the straight line close to the vertical direction based on the accumulator array (ρ, θ); wherein the vertical line close to the vertical direction has θ within the vicinity of 0 degrees. Generally, three or more side edges can be found in residential buildings, and only when the drone is facing the side wall of the residential building, there are only two side edges.
[0134] After excluding the straight lines in the side edge direction, find the two straight lines with the longest length and close directions in the set of straight lines obtained by voting to form a candidate straight line pair. According to the degree of proximity of the θ values, several subsets of straight lines with close directions can be clustered. For each subset, 3 to 5 candidate straight lines can be screened. In the range of non-side edge directions, the lengths of the straight line segments are sorted from large to small according to the values of the accumulator array elements, and the two secondary straight line segments close to the main straight line segment with the longest length are searched in sequence. The main straight line segment and the corresponding two secondary straight line segments form a candidate straight line group. Preferably, 2 to 3 candidate straight line groups are obtained; some residential buildings have multi-story stepped shapes, and multiple pairs of parallel edges will appear.
[0135] Combination Figure 1 As shown, it can be found that the horizontal boundaries on the roofs and ground of the houses on the left and right sides have straight edges with nearly parallel directions in the field of view. To this end, all side edge groups are recorded in array form, and the upper and lower endpoints of each side edge are obtained by comparing the pixel point set corresponding to the same (ρ, θ) recorded in the Hough line transform accumulation process with the minimum and maximum values of the ordinate. For any straight line segment in the obtained candidate straight line group, calculate its distance from the upper and lower endpoints of the side edge of the residential building. If the element with the smallest value in the distance set corresponds to the distance between the straight line segment and the upper endpoint of a side edge, then the straight line segment is identified as a roof candidate straight line. Otherwise, if the element with the smallest value in the distance set corresponds to the distance between the straight line segment and the lower endpoint of a side edge, it is identified as a ground candidate straight line.
[0136] According to the directions corresponding to different candidate straight line groups and the identifiers of the candidate straight lines in the groups, the top and bottom edge extraction unit 1136 and the roof parallel line extraction unit 1137 respectively determine the three pairs of sidelines corresponding to the length, width and height of the residential outline edge. The candidate straight line groups are sorted according to the length of the main straight line segments in their respective groups, and the two candidate straight line groups with the largest corresponding lengths are respectively recorded as the first candidate straight line group and the second candidate straight line group. First, the top and bottom edge extraction unit 1136 forms the first pair of sidelines with two relative sidelines of the residential roof candidate straight line and the ground candidate straight line sandwiched between a pair of side edges in one of the candidate straight line groups. Then, the roof parallel line extraction unit 1137 forms the second pair of sidelines with two relative sidelines of the two residential roof candidate straight lines sandwiched between a pair of candidate straight lines in the first candidate straight line group in the second candidate straight line group; similarly, in the first candidate straight line group, the third pair of sidelines are formed with two relative sidelines of the two residential roof candidate straight lines sandwiched between a pair of candidate straight lines in the second candidate straight line group.
[0137] The first pair of side lines sandwich a pair of side edges or high edges of the residential building. The vertical coordinates of the upper and lower endpoints of a side edge can be accommodated between the vertical coordinates of each corresponding endpoint of the first pair of side lines, i.e., the roof candidate straight line and the ground candidate straight line; that is, the interval covered by the vertical coordinates of each corresponding endpoint of the roof candidate straight line and the ground candidate straight line basically approaches or includes the vertical coordinates of the upper and lower endpoints of the side edge.
[0138] In the Hough line transform, the ρ distance resolution is usually 1 pixel, while the θ angle resolution is usually π / 180. Since it is found that the boundary straight line features in rural dwellings have relatively obvious characteristics, the Hough line transform processing process can be optimized. As a preference, when traversing and accumulating the element values in the ρ-θ plane matrix, the spacing between elements in the value interval can take a larger value according to a priori testing, that is, the spacing between distances and angles is increased accordingly, and the number of rows and columns of the matrix is reduced, thereby greatly reducing the amount of calculation in the calculation judgment and shortening the calculation time.
[0139] Preferably, before peak detection, the elements in the accumulator array (ρ, θ) are merged. When the values of several adjacent elements in the array are above a preset pixel point threshold, i.e., the minimum length of a straight line, the centroid of the polygon enclosed by the coordinates of the several adjacent elements is obtained, and the (ρ, θ) corresponding to the obtained centroid is used as the residential boundary line.
[0140] For any pair of residential sidelines found, the slope feature extraction unit 1133 and the distance feature extraction unit 1134 respectively calculate the slope of the line and the vertical distance from the center point of the image to the line. Fig. 7A The schematic diagram of the residential model edge constraints and geometric features shown in the figure shows a pair of two sides of the residential building. Among them, the angles and slopes of the straight lines contained in the two side lines on the left are α(k1) and β(k2), respectively, and the perpendicular distances from the center point of the image to the two side lines of the residential building on this side are D1 and D2 respectively; correspondingly, the angles and slopes of the straight lines contained in the two side lines on the right are α′(k1′) and β′(k2′), respectively, and the perpendicular distances from the center point of the image to the two side lines of the residential building on this side are D1′ and D2′ respectively. In the oblique photography image plane, the two parallel lines will intersect at a point O at the far end, and point O is called the vanishing point.
[0141] The algebraic equation of a line can be expressed in three common forms, namely the general form, the two-point form, and the slope-intercept form. The general form is: Ax+By+C=0, where A and B are not zero at the same time, and the slope of the line k=-A / B. In the Cartesian rectangular coordinate system, the intercept of the line on the y-axis b=-C / / B, and the general form applies to all line equations.
[0142] The two-point formula is: The two-point equation is the equation of a straight line passing through the two points when the coordinates of the two points are known. The slope of the straight line is k = (y 2 -y 1 ) / (x 2 -x 1 ), it can be seen from the two-point equation that this equation is suitable for describing straight lines that are not perpendicular to the x-axis and y-axis.
[0143] The slope-intercept form is: y=kx+b. The slope-intercept form explicitly expresses the slope k and intercept b of the straight line, and is suitable for representing straight lines that are not perpendicular to the x-axis.
[0144] The straight line detected by Hough line transform in the image is described by the coordinates of the two end points of the straight line, and the two end point coordinates are marked as (x 1 ,y 1 )、(x 2 ,y 2 ), substitute the coordinates of the two points into the two-point equation, and then transform the two-point equation into the slope-intercept equation form to obtain: Its corresponding slope k = y 1 -y 2 / x 1 -x 2 The only difference is that the calculation of the slope of the line is completed in the image coordinate system (the y-axis is downward in the positive direction), so the slope of the line in the image is equal to that in the Cartesian rectangular coordinate system, but has the opposite sign.
[0145] Combining the two-point equation, the perpendicular distance from the center of the image to the edge of the residential building can be calculated as: In the formula, (x 0 ,y 0 ) represents the coordinates of the center of the image, A and B are the coefficients of the general formula of the straight line equation, C is the constant term and: A = y 2 -y 1 , B = x 1 -x 2 , C = -x 1 (y 2 -y 1 )+y 1 (x 2 -x 1 ). According to the calculation formula of k and d, the slope k of the straight line on both sides of the residential area is calculated respectively. 1 , k 2 And the perpendicular distance D from the center point of the image to the two straight lines 1 , D 2 A total of 4 quantities are used as the geometric features of the dwellings in the rural dwelling images.
[0146] Combination Figure 1 The actual residential scene in Fig. 7AThe concept of Figure 7B , 7C , 7D respectively illustrate the constraints and geometric characteristics of the length, width, and height of rural dwellings; among them, Fig.7D The two middle side lines are the first pair of side lines with a pair of side edges or high sides in the middle. Figure 7B , Figure 7C The two side lines in are the second pair of side lines and the third pair of side lines with a pair of long sides and wide sides in the middle. See also Fig. 7E As shown in the figure, for the determined residential boundary line, a vertical line is drawn from the center point of the image, represented by a dotted line. Fig. 7E In the figure, since the foot of the perpendicular is outside the range of the roof edge, the roof edge is extended to the foot of the perpendicular.
[0147] As a preference, a human-computer interaction interface is provided in the user interface unit. For residential buildings with complex outline structures, after automatic edge screening, the operator confirms and marks three pairs of edge lines through the human-computer interaction interface, obtains geometric features of the marked residential edge lines, and then predicts the size of the residential building. Figure 1 Taking the residential building shown in the figure as an example, there is a structure on the roof of the residential building on the left. In the human-computer interaction interface, the edge lines corresponding to the length and width of the structure can be further marked, and the geometric features of the marked structure can be calculated to predict its size. Finally, when calculating the length and width of the roof plane, the length and width obtained after subtracting the size of the structure from the overall length and width are used as the basis for determining the safe landing distance of the drone. Figure 1 For the residential house on the middle right, the roof edge is almost unobstructed, so it can be automatically screened first and then simply confirmed by the operator. The roof size can be directly obtained from the residential house size prediction model, which greatly simplifies the measurement operation and realizes the automatic measurement of the size of rural residential houses.
[0148] In order to reveal the nonlinear mapping structure of the residential size prediction model of the present invention, a rural residential scale model is established and then the model is tilted to obtain training image samples. Figure 5A As shown in FIG. 1 , when the drone is shooting the ground in the air, the angle between the optical axis of the onboard lens and the horizontal plane is α, which can be selected from a common working angle such as 45 degrees. Since the actual rural houses have different sizes, in order to facilitate modeling, the present invention uses a rectangular parallelepiped to make residential models of various specifications, such as Figure 5B shown.
[0149] In an offline state, a drone is used to collect images of the residential model and form a sample set, and the residential size prediction model is trained with the sample set. The residential model is in the shape of a cuboid, and the size of the cuboid is a first preset multiple S1 of the actual residential size. The multiple S1 is set according to the rural cruising flight altitude of the drone and the camera resolution; the different cuboids used in the residential model have a size range corresponding to S1 times the common size range of rural residential buildings, and the size difference between different cuboids corresponds to S1 times the actual residential size difference SD. For example, when the flight altitude is 100 meters, the sampling height of the corresponding model is 2 meters, so as to ensure that the residential model can occupy a significant image space during imaging, so as to coordinate the image resolution and the recognition accuracy of the residential size prediction model, and make it unnecessary for the drone to lower its altitude specifically for predicting the residential size, thereby increasing the scope of application of the prediction.
[0150] Preferably, the first preset multiple S1 is between 1 / 50 and 1 / 20, and the actual residential size difference SD is between 0.5 and 3 meters; the length, width and height of each rectangular block used as a residential model can preferably form an arithmetic progression with S1 times SD as the tolerance.
[0151] Using the residential model, three different actual drone cruising altitudes were simulated under laboratory conditions, and the camera was tilted at a 45° angle to shoot. real Compared with the aerial photography height h under laboratory conditions model The corresponding relationship is: when h real When the distance is 30m, 40m, or 50m, h model 667mm, 889mm, 1111mm, the scaling ratio is:
[0152]
[0153] refer to Fig. 6A As shown, due to the projective transformation, the straight lines that are parallel in the original three-dimensional cube become non-parallel in the two-dimensional image. The residential model is a three-dimensional model. The long side, wide side and high side of the residential building belong to different spatial positions and are combined into different planes in pairs. The present invention uses two opposite sides that are perpendicular to and sandwich the target side on a plane on a certain plane to express geometric feature constraints. For this purpose, feature extraction is performed from the edge lines on different planes.
[0154] Since the image sampling of the residential model is an oblique image, in order to ensure the diversity of the features representing the residential model in the image as much as possible and improve the generalization ability of the network model for predicting the size of the residential building, the residential model is photographed from 0°-180°. At the same time, in order to highlight the distance relationship between the drone and the actual location of the residential building, the residential models are arranged in different areas of the ground field of view, so that the residential models in the sample set are distributed in each square of the nine-square grid formed by the top, middle, bottom, left, middle, and right in the image plane. To simplify the explanation, Figure 6B Indicated Fig. 6A The top surface of the middle cuboid is at different positions in the image space when the sample set is collected. To improve the accuracy of the model, the image is shot with a black ground as the background. For the convenience of display, the image is inverted.
[0155] The present invention subdivides the size of residential models and forms an interval sequence, and shoots residential images from various relative angles during sampling, thereby ensuring the sufficiency of samples, thereby improving the generalization ability of the neural network and reducing the prediction error of the residential size.
[0156] For the rural residential images based on residential models, the collected images are filtered and grayed, then binarized to form residential target images, and then edge detection is performed to obtain residential edge images. Taking the top plane image of a residential model at a certain location as an example, the preprocessing results of each stage are as follows: Figure 6C As shown in the figure, from left to right they correspond to the processing results of grayscale, binarization, and edge detection. The Hough line transform is performed on the residential edge image after Canny edge detection. The transformed image is shown in Fig.6D As shown in the figure, the left and right figures are the Hough line transformation results of a pair of edge lines on both sides of the wide side. Afterwards, a heuristic search is performed on the straight lines obtained after the transformation to select the corresponding two side straight lines.
[0157] To ensure the correct detection and identification of residential boundary lines, a manual confirmation step can be added to the boundary search. Specifically, in the process of collecting straight line features representing residential buildings and establishing a training sample library, the interference of non-residential boundary straight lines is eliminated through the human-computer interface. With the help of the HighGUI module provided by openCV, a graphical user interface is established and the mouse callback function setMouseCallback() is used. In the callback function, a left mouse button click event and a right mouse button click event are created. Each time the left mouse button is clicked, a straight line detected by Hough line transform is drawn in the image, and it is confirmed after being compared with the original residential image.
[0158] Refer to the above Fig. 7AAs described in the description, for each model residential image, the four geometric features of the edge lines on both sides of a boundary are calculated, and the corresponding actual boundary size, i.e., length, width, or height, is recorded as the real value of the residential size. The geometric features and the real value of the residential size are formed into samples, and after the samples are collected and recorded to form a sample set, they are input into the residential size prediction model based on the neural network for offline training. Preferably, the sample set can be stored in the form of a data table composed of input and output real values as a record.
[0159] The BP neural network used in the residential size prediction model in this embodiment is as follows: Figure 4A As shown, the model is a three-layer structure including a hidden layer, in which
[0160] The output of the jth node in the hidden layer is
[0161] The output of the output layer node is
[0162] In the formula, the f() function is taken as the tansig function, w ij and v j are the connection weights from the input layer to the hidden layer and from the hidden layer to the output layer, respectively. j and b are the hidden layer and output layer thresholds respectively, m=4 is the number of input variables, k is the number of hidden layer nodes, and the gradient descent method is used for network training.
[0163] When predicting the size of residential buildings online, a pair of edge lines corresponding to the length, width and height of the residential buildings are obtained from the residential building model image. The length, width and height of the residential building model are predicted based on the four feature quantities of each pair of edge lines calculated. The prediction model also maps the model back to the actual residential building size as the actual output value based on the ratio of the model to the actual size.
[0164] Preferably, a plurality of the neural network-based residential size prediction models are created in the host unit, each residential size prediction model corresponds to a flight altitude and a bird's-eye view angle of an image acquisition unit; when performing residential size prediction online, the geometric features obtained after the current acquisition process are input into the residential size prediction model corresponding to the flight altitude and the bird's-eye view angle during acquisition, and the model output value is obtained as the residential size prediction value.
[0165] The sample data sets at three heights are randomly divided into training sets and test sets, both of which cover various residential models. The geometric features of the test set are input into the trained network model as input to obtain the predicted value. The fitting curves of the actual model value and the predicted value of the residential width at different heights are shown in Figure 1. Figure 8 As shown; accordingly, the fitting curve of the model actual value and predicted value of residential length is shown as Fig. 9As shown; the fitting curve of the model actual value and predicted value of residential height is as follows Fig.10 shown.
[0166] Analyze the regression results at three different laboratory heights, and the coefficient of determination R in the width dimension at three different heights 2 The R values of the three heights are 0.99815, 0.99863 and 0.99649 respectively. The closer the coefficient of determination is to 1, the better the regression effect is, indicating that the model size prediction models at the three heights have high reliability. The root mean square error (RMSE) at the three different heights is 2.6277, 1.632 and 1.8552 respectively, indicating that the predicted value is close to the true value and the network model has good generalization ability. 2 Both the accuracy and RMSE indicate that the model has high reliability and good generalization ability.
[0167] Example 2
[0168] Based on the previous embodiment, Figure 2 As shown, this embodiment provides a rural dwelling identification system 1000 under the field of view of a drone using the method of the present invention, which includes: an image acquisition unit 500 for image sensing and acquisition of rural scenes, a user interface unit 200 for parameter input and operation and rural dwelling information display, a server 400 for data management and information transmission, a handheld terminal 300 for remote operation and rural dwelling information interaction, and a host unit 100 connected to the user interface unit 200, the server 400, and the image acquisition unit 500.
[0169] Preferably, the rural residential identification system 1000 under the drone vision may not include the server 400 and the handheld terminal 300, but the rural residential identification system under the drone vision is connected to the server 400 and the handheld terminal 300 based on a communication network during application. A positioning unit 700 for acquiring coordinates may also be provided in the rural residential identification system 1000 under the drone vision.
[0170] Preferably, the user interface unit 200 is provided with a display screen 210 and an operation panel 220. The rural residential identification system 1000 under the vision of the drone further includes a flight control platform 600. In response to the user's operation on the operation panel 220 or the instruction of the host unit 100, the flight control platform 600 executes the instruction and flies toward the designated residential site.
[0171] The rural residential identification system 1000 under the field of view of the drone continuously detects multiple frames of rural residential images and predicts the size of the residential buildings. When the safety distance of the residential buildings meets the requirements, the sizes of multiple residential buildings are compared, and the building with the largest safety distance is determined as the target landing point. The coordinate position information of the target landing point is output to the flight control platform 600 through the output unit, and the flight control platform controls the power unit to make the drone fly to the target landing point.
[0172] Preferably, when the drone is used in a group system such as logistics and transportation, the system may have multiple handheld terminals and drone platforms accordingly. Through the server, the host unit and the remote client deployed at the handheld terminal can send messages of residential size. Preferably, an MQTT server can be deployed, and the server pushes messages to the client subscribed to the topic. Multiple devices can subscribe to multiple topics at the same time, and when publishing messages, those who subscribe to the same topic can receive the same message. The publish-subscribe mode is a messaging mode that decouples the client that sends the message, i.e., the publisher, from the client that receives the message, i.e., the subscriber, so that the two do not need to establish a direct connection or know each other's existence.
[0173] The server provides subscription and publishing services, and the host unit is also configured to: subscribe to the topic of obtaining residential information of the handheld terminal, and transmit the detected residential size and corresponding coordinates and location information to the handheld terminal through the server in response to the topic message; or when it is detected that the residential size reaches a set safety distance threshold, transmit the residential size and corresponding coordinates and location information to the server in the form of a residential size event topic message.
[0174] Preferably, the user interface unit is provided with an operation panel for parameter input, and is also provided with a display screen for assisting parameter input and displaying the residential size recognition results in the form of a list; the host unit also displays the recognized residential size in segments in the form of a curve on the display screen, and marks the coordinates and positions corresponding to the recognized size on the curve.
[0175] Preferably, the positioning unit 700 in the system also obtains heading information, and also records and identifies the direction identified based on the heading, such as the north direction, and the direction corresponding to the axis of the roof of the residential building in the output position information.
[0176] Preferably, the host unit may also communicate directly with the handheld terminal via a wireless network.
[0177] The host unit can be a local control unit deployed separately on the ground, or a control unit deployed on a drone platform. When deployed on a drone platform, the host unit on the drone platform performs edge computing for online residential size prediction. Due to the limited resources of the edge computing unit, the processing calculation of the host unit is optimized.
[0178] In the general Hough line transform, for each edge point (x, y) in the image space, the ρ value of ρ = xcosθ + ysinθ under different θ is calculated, and the corresponding position in the accumulator array is voted. This process of calculating ρ requires two floating-point multiplication operations.
[0179] To this end, based on the obvious boundary straight line features in rural residential buildings, the pixel check accumulation process of Hough line transform is optimized. Specifically, when the Hough line transform is performed on each foreground pixel point (u, v) in the residential edge image, in the accumulation statistics, it is determined whether the pixel point is adjacent to a straight line element l i,j (ρ i,j ,θ i,j ) The corresponding judgment method is: take a point H(ρ i,j ·cosθ i,j , ρ i,j sinθ i,j ), recorded as H(u′, v′), calculate J = |vv′-(uu′)·tanθ i,j |, if |J|≤ε, then the pixel point is determined to be on the straight line and accumulated, where i and j are the row and column labels in the ρ-θ plane matrix, and ε is a preset small positive number.
[0180] By setting the fulcrum, a tangent operation is performed once before traversal judgment, so that each time the foreground pixel is judged, only two subtraction operations and one multiplication operation are required to perform judgment accumulation, thereby avoiding two time-consuming floating-point operations for each pixel, further saving computing costs, and thus suitable for edge computing deployed on drone platforms.
[0181] In the heuristic search for obtaining residential edge lines, when the angle is expressed as a slope in a rectangular coordinate system, when the absolute value of the difference in the edge line slopes is less than a very small first slope threshold, the two edge lines are merged; at the same time, if the angle tangent value or the slope is greater than a very large second slope threshold, the edge line is determined to be a vertical line perpendicular to the horizontal axis, thereby preventing the slope from being judged as different lines due to slight differences when it is very large, and avoiding the false detection of short edge lines on the residential edge lines.
[0182] like Figure 4AAs shown, as a preferred embodiment, in the sample collection, the present embodiment can group and record samples from the two sets of relative edges of the front side and the upper surface of the cube, and the output layer corresponds to the height, length and width respectively. Based on the subdivided sample set, different neural networks can be established for the height, length and width of the residential buildings respectively, and trained based on the corresponding sample sets. As a preferred embodiment, the width and length on the horizontal plane can also be combined for sampling and recording, without distinguishing between length and width.
[0183] Different from the first embodiment, in the artificial neural network of the residential size prediction model, the input layer of the present embodiment can receive the slopes k of the straight lines on both sides of the residential building from the general processing module respectively. 1 and k 2 , the perpendicular distance D from the center of the image to the straight lines on both sides of the residential building 1 and D 2 , and the flight altitude and bird's-eye view angle during collection, a total of 6 input quantities, and the output of the output layer is the size of residential buildings. After establishing a neural network with this structure, the input quantity of each sample in the training set is supplemented, and after supplementing enough samples at different heights and bird's-eye view angles, it is trained with a new training sample set. When applied online, the new model can further simplify the image sampling requirements for rural residential prediction; it uses the generalization performance of neural networks to reduce the number of models required for rural residential size prediction, and also simplifies the system structure, so that one model can cover different flight sampling scenarios, further improving the applicability of the system.
[0184] Example 3
[0185] Based on the above embodiments, this embodiment provides a rural flight control method based on road width, comprising the following steps:
[0186] R1. Initialization, establishing a rural residential size prediction neural network to map the geometric features of residential boundary lines in oblique photography images to residential sizes, collecting training samples and training the rural residential size prediction neural network to obtain a residential size prediction model;
[0187] R2. In response to the residential size recognition command during flight, take oblique photos of the rural scene to obtain images of rural residential buildings;
[0188] R3, determining a ROI region including residential buildings in the rural residential image, extracting residential buildings from the ROI region as foreground objects, and setting all non-foreground object regions in the rural residential image to zero to form a residential building target image;
[0189] R4. For the obtained residential target image, edge detection is performed to obtain a residential edge image; Hough line transform is performed on the residential edge image, and a residential edge image containing two straight line edges of the residential building is obtained through heuristic search;
[0190] R5. Perform Hough line transform on the residential edge image, and obtain a residential edge image including three pairs of edge lines in the residential outline edge through heuristic search, wherein the first pair of edge lines of the three pairs of edge lines are two opposite edge lines of the top and bottom of the residential building with a pair of side edges in between, and the second pair of edge lines and the third pair of edge lines of the three pairs of edge lines are two opposite edge lines with a pair of horizontal long sides and horizontal wide sides in between respectively;
[0191] R6. From the residential boundary image, for any pair of the three pairs of boundary lines, obtain the geometric feature quantities of the two opposite boundary lines relative to the image center; input the geometric features into the trained residential dimension prediction model to obtain the residential dimension prediction value, and output the dimension prediction value;
[0192] R7, determine whether the path planning is to fly over the current residential building or to stop and land. If it is to fly over, control the minimum flight altitude to be greater than the residential building height in the current residential building size, which is a set threshold; otherwise, if it is to stop and land, go to step R8;
[0193] R8, determine whether the length and width of the target parking site in the current residential size data are both greater than the preset safety distance threshold. If so, take the center of the target parking site as the landing point and go to step R9; otherwise, issue an alarm to indicate that the site size is insufficient and go to step R10;
[0194] R9, sending the landing point location information to the flight control platform, after which the flight control platform controls the drone to land, proceed to step R11;
[0195] R10, fly to the next rural residence, during which the trained target classification model based on the deep network is used to identify and process the rural image in the current field of view. When a rural residence target is identified in the field of view, a residential size recognition instruction is issued, and the process goes to step R2;
[0196] R11. End rural flight control based on house size.
[0197] Combination Figure 2 , Figure 3A , Figure 4BAs shown, a controller 100 for identifying rural residential buildings in the field of vision of a drone using a rural flight control method based on road width of the present invention includes a processor 110, a storage module 120 and an input / output module 130. The processor 110 includes a general processing module 111, an acquisition and preprocessing module 112, a feature extraction module 113, an iterative learning module 114 and a residential building size prediction module 115. Among them, the general processing module 111 responds to the operation of the message and user interface unit, and starts the identification of residential buildings in the collected image, schedules the identification processing flow, inputs the real-time collected rural residential building images into the acquisition and preprocessing module 112, and outputs the size data identified and predicted by the residential building size prediction module 115.
[0198] Combination Figure 3A , Figure 4B As shown, the rural residential image is input into the controller by the input unit 131 in the input-output module 130; the residential size prediction model established in the residential size prediction module 115 has an input layer receiving the geometric features of the residential buildings in the rural residential image extracted by the feature extraction module 113 from the general processing module 111, and an output layer transmitting the geometric features to the iterative learning module 114 and the general processing module 111 through the first connection matrix 1141 and the second connection matrix 1142 respectively.
[0199] When the residential size prediction model is trained offline, the iterative learning module 114 adjusts the connection weights of the network according to the actual values of the residential size and the output values of the neural network in the model respectively input by the general processing module 111 and the residential size prediction model through the first connection matrix 1141; in the field environment, the first connection matrix 1141 is disconnected, and the residential size prediction model neural network predicts the size of the residential buildings in the current rural residential image and outputs it to the general processing module 111 through the second connection matrix 1142, and is processed and analyzed by the general processing module 111 and output through the output unit 132 in the input-output module 130.
[0200] The steps of iterative training include: STEP1, initialize the weights and biases of the network; STEP2, read the network parameters and the training sample set written in the document above; STEP3, normalize the sample data; STEP4, calculate the error between the predicted output value and the expected value of each sample; STEP5, reversely calculate and correct the network weights and biases; STEP6: if the conditions for the end of network training are met, end the training, otherwise go to STEP4 to continue training; STEP7, analyze and output the results.
[0201] The general processing module 111 in the controller also obtains the coordinate information and orientation information of the current location of the residential building through the positioning unit 700, and after determining the size of the residential building, outputs the size of the residential building and the corresponding coordinate and orientation information through the output unit 132.
[0202] like Fig.15 As shown, after the residential size is identified, the residential size is also marked in the residential edge segment corresponding to the extracted residential edge segment in the rural residential image, such as by using an arrow to mark the corresponding direction segment. Preferably, the acquired coordinates and location information, as well as the residential size, are also stored in a list of the storage module 120, and the list information can be stored in the cloud server 400.
[0203] When the length and width in the size data are greater than the preset safety distance threshold, the coordinate position value of the central point on the roof of the residential building is given; otherwise, an alarm is issued to indicate that the safety distance of the residential building roof is insufficient. During the flight of the drone, image acquisition is performed iteratively and continuously. The rural residential identification controller 100 in the field of view of the drone performs size prediction processing based on the acquired images. When the length and width of the residential building in the field of view are greater than the preset safety distance threshold, a prompt is issued that the residential building has a sufficient safety distance. Preferably, the image is updated and processed according to the flight speed and frame rate so that the residential buildings in the field of view are identified one by one.
[0204] After finding a site with a sufficient safety distance, the safety distance of the site can be verified by collecting residential images from multiple angles and predicting their size. When flying toward the target residential area, the rural scene image in the new field of view is identified and processed based on the rural scene target classification model, and then the rural residential area anchor frame is updated. According to the position of the residential anchor frame in the field of view, flight control is performed to make the residential area approach the center area of the field of view.
[0205] In the heuristic search for boundary lines at the edge of residential buildings, some residential buildings have multi-layered steps, and the corresponding parallel edges will appear in multiple pairs. In this case, the candidate straight line groups may or may exceed 2 groups. At this time, in the candidate straight line groups, the minimum length or width corresponding to the boundary formed by the pair is used as the safety distance; and for height, the maximum height is used as the safety distance.
[0206] Preferably, if the flat ground is taken as the landing point, then in one of the candidate straight line groups, the second pair of edges and the third pair of edges are formed by two relative edge lines of two candidate straight lines on the ground rather than the roof of a residential building with a pair of other straight line segments sandwiched in between, so as to further obtain the geometric features of the flat ground.
[0207] Example 4
[0208] Different from the above embodiments, this embodiment provides another method for obtaining the dimensions of the long side and the wide side of a residential building.
[0209] See also Figure 1 , and combined with Fig. 7EAs shown, in the boundary of the three-dimensional residential building, in addition to the horizontal parallel line segments, there are also vertical side edges, which can also sandwich the long side or wide side of the residential building. Therefore, this embodiment forms the second pair of side lines and the third pair of side lines based on the acquired residential side edges, using two residential side edges sandwiching a pair of other straight line segments as two opposite side lines.
[0210] At this time, preferably, in the input of the neural network, the slope of the boundary line, i.e. the side edge, is replaced by the angle rotated counterclockwise from 0 degrees to the boundary line, so as to prevent errors caused by using the slope when the angle is close to 90 degrees.
[0211] The present invention is used to identify the size of rural residential buildings in the image space under the vision of drones. After residential area segmentation, residential edge extraction and boundary line determination, the slope k of a pair of boundary lines corresponding to the length, width and height of each side is used. 1 , k 2 And the perpendicular distance D from the center point of the image to the boundary line 1 , D 2 A total of four geometric features were input into the established neural network-based residential size prediction model. The model can accurately predict the actual size of rural residential buildings in three dimensions: length, width and height, thereby providing a decision-making basis for the docking and flight of drones and improving the safety and convenience of drone operations.
[0212] Several embodiments of the present invention have been described above, but these embodiments and their modifications are intended to fall within the scope and spirit of the invention and are also intended to fall within the scope of the invention described in the claims and their equivalents.
Claims
1. The method for identifying rural residential buildings under the vision of drones includes the following steps: S1, initialization, establishing a neural network for predicting rural residential size; S2, determine whether the current state is offline network training or online size prediction, if it is the latter, go to S9, otherwise go to S3; S3, using oblique photography to collect images of rural dwellings of different sizes from the drone's field of view at a certain height, and record the actual length, width and height of the dwellings; S4, selecting a residential area from the rural residential image, removing the part outside the residential area to form a residential target image, performing edge extraction on the residential target image to obtain a residential edge image; S5, performing Hough line transform on the residential edge image, and obtaining two relative edge lines of the residential building that sandwich the long edge, the wide edge, or the high edge as the target edge and are approximately perpendicular to the target edge based on the surfaces of the residential building, respectively, to form a residential edge line image and record the corresponding relationship between the target edge and its two relative edge lines; S6, according to any two relative edge lines obtained in the residential edge line image, calculating the geometric feature quantity of each edge line relative to the image center; recording the geometric feature quantity of the two relative edge lines and the size of the corresponding target edge as a training sample; S7, determine whether the current training sample set needs to be supplemented, if so, go to step S3, otherwise go to step S8; S8. Based on the obtained training sample set, a rural residential size prediction neural network is trained to obtain a trained residential size prediction neural network as a residential size prediction model; S9, in an online state, perform residential size identification in an actual rural scene; detect whether there is an exit instruction, if so, proceed to step S14, otherwise, proceed to the next step S10 after the residential size identification instruction is triggered; S10, acquiring a rural residential image, determining a ROI region containing residential buildings, extracting residential buildings from the ROI region as foreground objects, and setting all non-foreground object regions in the rural residential image to zero to form a residential building target image; S11, extracting the edge of the residential target image to obtain the residential edge image; S12, performing Hough line transform on the residential edge image, and obtaining a residential edge image including three pairs of edge lines in the residential outline edge through heuristic search, wherein the first pair of edge lines of the three pairs of edge lines are two opposite edge lines of the top and bottom of the residential building with a pair of side edges in between, and the second pair of edge lines and the third pair of edge lines of the three pairs of edge lines are two opposite edge lines with a pair of horizontal long sides and horizontal wide sides in between, respectively; S13, from the residential boundary image, sequentially obtain geometric features of two opposite boundary lines relative to the image center for any pair of the three boundary lines, input the geometric features into the trained residential dimension prediction model, obtain a residential dimension prediction value, and output the dimension prediction value; And go to step S9; S14. End the rural residential size prediction.
2. The method for identifying rural dwellings in the field of vision of a drone according to claim 1, characterized in that: In the step S3, rectangular residential models of various specifications are built, and residential image samples are obtained by obliquely photographing the models from different directions; In the step S4, after retaining only the residential area portion of the acquired residential image sample, grayscale and binarization processing is performed, and the edge of the binarized residential target image is extracted to obtain a residential edge image.
3. The method for identifying rural dwellings in the field of vision of a drone according to claim 1, characterized in that: In the step S6, the slope of each sideline and the vertical distance from the center of the image to the sideline are calculated as the geometric feature quantity of each sideline relative to the center of the image; and the slopes k1 and k2 of the two relative sidelines and the vertical distances D1 and D2 from the center of the image to the two relative sidelines, a total of four geometric features, are used as the input quantity of the training sample, and the size of the target side corresponding to the two relative sidelines is used as the output quantity of the training sample; In the step S13, four geometric features, namely the slopes k1 and k2 of the two relative sidelines and the vertical distances D1 and D2 from the center of the image to the two relative sidelines, are obtained as geometric features relative to the center of the image and input into the trained residential size prediction model to obtain a predicted value of the residential size. If the two relative sidelines correspond to the first, second or third pair of sidelines, the residential size is the height, length or width of the residential size accordingly.
4. The method for identifying rural dwellings in the field of vision of a drone according to claim 1, characterized in that: In step S6, the method of calculating and extracting geometric features for any pair of edge lines among the three pairs of edge lines includes: Assuming that the coordinates of the two endpoints of a residential boundary line are (x1, y1) and (x2, y2), then its slope is k = y1-y2 / x1-x2; The vertical distance from the center of the image to the edge of the residential building is Where (x0, y0) represents the coordinates of the center of the image, A and B are the coefficients of the general formula of the straight line equation, C is a constant term and: A = y2-y1, B = x1-x2, C = -x1(y2-y1)+y1(x2-x1); According to the calculation formulas of k and d, the slopes k1 and k2 of any pair of edge lines of the residential edge and the vertical distances D1 and D2 from the center point of the image to the two edge lines are calculated respectively, a total of four quantities as the geometric features of the residential buildings in the rural residential image.
5. The method for identifying rural dwellings in the field of vision of a drone according to claim 1, characterized in that: In step S8, training the neural network for predicting the size of rural houses includes: T1, initializing the weights and biases of the network; T2, reading the network parameters and the training sample set recorded in the document; T3, normalizing the sample data; T4, calculating the error between the predicted output value and the expected value of each sample; T5, correcting the network weights and biases by reverse calculation based on the error; T6, if the conditions for ending the network training are met, ending the training, otherwise going to T4 to continue the training; T7, analyzing and outputting the results.
6. The method for identifying rural dwellings in the field of vision of a drone according to claim 1, characterized in that: The step S12 comprises: after performing Hough line transformation on the residential edge image, according to the polar coordinate space parameter characteristics in the Hough line transformation, respectively selecting the straight line segments with the longest length in the vertical and horizontal directions as candidate straight line segments, and according to the orientation characteristics of the candidate straight line segments, respectively determining two relative edge lines sandwiching a pair of long sides, wide sides and high sides as a pair of edge lines of the length, width and height of the residential building, and together forming three pairs of edge lines of the residential building; For each sideline of any pair of the three pairs of sidelines, in combination with the corresponding residential edge image, the coordinates of the two end points of the straight line are used to describe it, and the slope of the straight line and the perpendicular distance from the center point of the image to the straight line are respectively calculated based on the two end points.
7. The method for identifying rural dwellings in the field of vision of a drone according to claim 1, characterized in that: The step S12 comprises: when performing Hough line transform on each edge pixel point in the residential edge image, accumulating and obtaining a ρ-θ plane matrix corresponding to the image, wherein ρ represents the distance of the vertical segment of the straight line from the origin, and θ is the angle of the vertical line to the horizontal axis; Based on the accumulated ρ-θ plane matrix, search for the two or three longest edges in a range close to the vertical direction as the side edges of the residential building; In the range of the non-side edge direction, the plane matrix element values, i.e., the lengths of the straight line segments, are sorted from large to small, and two secondary straight line segments close to the main straight line segment in direction are searched in sequence from the main straight line segment with the largest length, and the main straight line segment and the corresponding two secondary straight line segments form a candidate straight line group; For any straight line segment in the candidate straight line group, calculate its distance from the upper and lower end points of the side edge of the residential building. If the minimum value of the distance corresponds to the distance between the straight line segment and the upper end point of a side edge, then the straight line segment is identified as a roof candidate straight line, otherwise it is identified as a ground candidate straight line.
8. The method for identifying rural dwellings in the field of vision of a drone according to claim 7, characterized in that: The step S12 further comprises: in one of the candidate straight line groups, forming the first pair of side lines with two opposite side lines, namely, a residential roof candidate straight line and a ground candidate straight line sandwiching a pair of side edges, In one of the candidate straight line groups, the second pair of sidelines and the third pair of sidelines are formed by two opposite sidelines of two residential roof candidate straight lines sandwiching a pair of other straight line segments.
9. The method for identifying rural dwellings in the field of vision of a drone according to claim 1, characterized in that: In step S1, a target classification model based on the ASFF-YOLOv5s network is also established. After collecting rural scene images with a bird's-eye view camera, the network is offline trained using an image sample set consisting of rural scene images with rural residential areas labeled to obtain a rural scene target classification model; In step S10, the rural scene target classification model performs recognition processing on the rural scene image including residential houses, i.e., the rural residential house image, to obtain a rural residential area anchor frame, and uses the area within the anchor frame as the ROI area of the rural residential houses. Preferably, the ASFF-YOLOv5s network adds three layers 24, 25, and 26 between the 23rd and the next layer of the YOLOv5s network. These three layers receive the outputs of the 17th, 20th, and 23th layers as input, respectively, and after the ASFF adaptive spatial feature fusion module performs feature fusion of different scales, they replace the original 17th, 20th, and 23th layers in turn and input them into the 24th layer of the original output layer, i.e., the 27th layer of the new network.
10. The method for identifying rural dwellings in the field of vision of a drone according to claim 1, characterized in that: The step S12 comprises: when performing the Hough line transform on each edge pixel point in the residential edge image, accumulating and obtaining the ρ-θ plane matrix corresponding to the image, wherein ρ represents the distance of the vertical segment of the straight line from the origin, and θ is the angle of the vertical line to the horizontal axis; in the accumulating, the foreground pixel point (u, v) in the image and a straight line element l i,j (ρ i,j ,θ i,j ) The corresponding judgment method is: Take a point H(ρ i,j ·cosθ i,j , ρ i,j sinθ i,j ), recorded as H(u′, v′), calculate J = |vv′-(uu′)·tanθ i,j |, if |J|≤ε, then the pixel point is determined to be on the straight line and accumulated, where ε is a preset small positive number.
