Country dwelling identification device and system under view of unmanned aerial vehicle

Through neural network and Hoff line transformation technology, the applicability and accuracy of drones' measurement of residential dimensions in rural transportation is solved, and the accurate prediction of the length, width and height of residential buildings in the drone's vision is achieved, which improves the Transportation safety and convenience.

CN118918464AActive Publication Date: 2024-11-08CHINA JILIANG UNIV
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202410948045.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-08
Publication Date
2024-11-08
Estimated Expiration
2044-07-08

AI Technical Summary

Technical Problem

The prior art is difficult to accurately measure the length, width and height dimensions of houses in rural drone transportation, especially when the drone flight attitude and house orientation are unpredictable, which limits the applicability and accuracy of the measurement.

Method used

Using a neural network-based drone house identification device under rural village houses, the edge features of the house are extracted through image processing and Hoff line transformation technology, and the slope and offset are used as input features to establish a house size prediction model to achieve the length of the house. Online prediction of width and height dimensions.

Benefits of technology

It realizes accurate prediction of three-direction sizes of houses of various specifications under unfixed heading, with wide applicability, high prediction accuracy, and strong robust model, which simplifies operation steps and improves real-time.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918464A_ABST
    Figure CN118918464A_ABST
Patent Text Reader

Abstract

The invention discloses a device and a system for identifying country dwellings under the view of an unmanned aerial vehicle, a host unit obtains a country dwelling image, extracts a dwelling area from the country dwelling image, carries out edge detection, and obtains three pairs of side lines corresponding to the two sides of the side where the length, width and height are located through Hough line transformation and heuristic search; and calculating the slope of each pair of side lines and the vertical distance between the center of the image and the side lines, inputting the slope and the vertical distance into a dwelling size prediction model based on a neural network as constraints, respectively performing size prediction on the length, the width and the height of the dwelling, and outputting from a user interface unit or a handheld terminal. The device can automatically obtain the horizontal edges and the vertical side edges of the ground and the roof of the dwelling, and the established dwelling size prediction model extracts features from a straight line without depending on accurate pixel starting and ending points, so that the prediction precision is high, the generalization performance is good, and the method is suitable for large-scale popularization and application. And the sizes of various specifications of dwellings can be accurately estimated according to the rural dwelling images acquired from different directions, and the application range is wide.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned aerial vehicle (UAV) rural transportation application, and specifically relates to a rural residential identification device and system under the field of view of an unmanned aerial vehicle (UAV). Background Art

[0002] The application of drones in rural transportation has broad potential and practical value. In rural areas where medical resources are scarce, drones can quickly transport emergency medicines, vaccines, blood and medical equipment. They can also provide telemedicine services to remote areas and quickly transport medical samples and equipment. With the integration of special accessories, drones can quickly collect soil, water and crop samples in farmland to support precision agriculture and crop monitoring. In natural disasters or emergencies, drones can quickly transport and deliver relief supplies, fly to disaster areas quickly, conduct disaster assessments and filming, and help formulate rescue plans. In terms of environmental monitoring and protection, drones are used to monitor ecological environments such as forests, rivers, and wetlands, and collect data such as vegetation coverage and water quality changes; at the same time, they can monitor illegal logging, poaching and other activities to protect natural resources and the ecological environment.

[0003] In terms of infrastructure inspection and maintenance, drones are used to inspect power lines, communication base stations, bridges and other infrastructure to promptly detect and address problems. They are also used to conduct regular inspections of rural houses and public facilities to ensure safety and maintenance needs.

[0004] In terms of e-commerce logistics, drones can provide fast delivery services for small parcels, letters and express deliveries, improve rural logistics efficiency, and deliver fresh food and daily necessities to farmers and residents, improving the convenience of life. In the future, it is expected to support the logistics and distribution of rural e-commerce platforms, shorten the circulation time of agricultural products and daily necessities, and provide reverse logistics transportation, provide farmers with convenient agricultural product sales channels, and quickly transport agricultural products to urban markets.

[0005] Drone transportation has the advantages of high efficiency and wide coverage. Drones can reach the destination quickly and save time, especially in rural areas with inconvenient transportation. Drones can also provide a full range of services because they can cover inconvenient or remote rural areas.

[0006] In rural transportation, whether it is a normal landing or an emergency landing, drone docking needs to consider safety, convenience and efficiency. In order to avoid rural houses or choose a suitable docking point in the target residential environment during drone rural transportation, accurate house length, width and height measurement data is required as the basis for planning navigation. On the one hand, during the flight, when setting the minimum flight altitude of the drone, it is necessary to judge the height of rural houses; on the other hand, when flying around houses or landing on roofs, it is necessary to measure the size and length of the houses to ensure a safe lateral and longitudinal distance.

[0007] In the prior art, there are several solutions for measuring buildings. For example, the invention patent application with application number 201811029714.X discloses a spatial measurement method in the fields of intelligent buildings, surveying and mapping, in which a laser emits a light pulse to the measured target, and the distance is obtained by the flight time required for the measured object to reflect. Then, a camera is used to obtain a measurement image to measure the distance between two points in the image; this method requires that the camera optical axis is facing the target plane, and can only measure the distance between two points on the single-orientation target plane. The invention patent application with application number 202310233220.8 discloses a method for identifying, obtaining and calculating the ratio of the length unit to the pixel value of a CAD drawing, using an AI method to automatically obtain the pixel value on the CAD drawing and the number of pixels corresponding to it on the drawing, thereby calculating the ratio of the length of the pixel to the actual space, and automatically calculating the length of the actual space represented by the pixels of the architectural CAD image; obviously, this method requires a priori ratio of the length of the pixel to the actual space, that is, an actual length needs to be measured or marked in advance. The invention patent application with application number 202310416333.1 discloses a field survey method for photovoltaic power stations. It uses any variety of tools such as infrared level, tape measure, tape measure, and laser rangefinder to measure the actual area and length and width of the roof and cross-check, so that users can judge whether a photovoltaic power station needs to be installed. Through on-site photography, on-site measurement and background system review, it is judged whether the roof is suitable for installing a photovoltaic power station based on the surrounding obstacles and the condition of the roof. It can be seen that this application also needs to use actual measurement methods to obtain the size of the roof.

[0008] The invention patent application with application number 202110395821.X discloses a method for monitoring building height based on high-resolution optical remote sensing satellite images based on corner points. It uses spatial distribution to pair corner points, and then uses the geometric model of shadow imaging to calculate the height of the building. It uses the geometric relationship between the total length of the building's shadow on the remote sensing image, the solar altitude angle, the solar azimuth angle and the building height.

[0009] From the above technical solutions, it can be seen that the current image-based measurement of house size is either based on prior data measured on the spot or directly calculated based on pixel ratio. Among them, the latter relies on an accurate proportional relationship, that is, the pixel width value of the measured line is required to correspond to a priori determined ratio. Obviously, when this solution is applied to the drone system, it requires that the drone video screen is fixed at the same relative azimuth angle to be comparable, which undoubtedly greatly limits the measurability of residential size.

[0010] Therefore, there is a need for a device and system for identifying the length, width and height of rural houses with strong generalization performance and wide applicability. Summary of the invention

[0011] The purpose of the present invention is to provide a rural dwelling identification device, controller and system under the field of view of a drone, which is used to identify the characteristics of rural dwellings based on images in the field of view of the drone, and then measure their height, length and width of the roof, etc., without limiting the drone to maintain a certain precise orientation when sampling images.

[0012] In order to solve the above technical problems, the present invention provides a rural residential identification device under the field of view of a drone, which includes a host unit and a user interface unit. The host unit includes an input and output module, a processor and a storage module and is configured as follows:

[0013] Acquire a rural residential image, determine a ROI region containing residential buildings in the rural residential image, extract residential buildings from the ROI region as a residential building target image; obtain a residential building edge image by edge detection for the residential building target image; perform Hough line transform on the residential building edge image, and obtain a residential building edge image including three pairs of edge lines in the residential building contour edge by heuristic search, wherein the first pair of edge lines of the three pairs of edge lines are two opposite edge lines of the top and bottom of the residential building with a pair of side edges in the middle, and the second pair of edge lines and the third pair of edge lines of the three pairs of edge lines are two opposite edge lines of the residential building with a pair of horizontal long sides and horizontal wide sides in the middle, respectively; obtain geometric features of the two opposite edge lines relative to the image center for any pair of edge lines of the three pairs of edge lines in the residential building edge image in turn, input the geometric features into a trained residential building size prediction model based on a neural network, obtain a residential building size prediction value, and output the size prediction value.

[0014] Preferably, an operation panel is provided in the user interface unit, and the host unit is further configured to: extract the residential area from the ROI area as the foreground object, use the non-foreground object area in the rural residential image as the background object, and form the residential target image after setting all background objects in the rural residential image to zero.

[0015] Preferably, the processor includes a general processing module, an acquisition and preprocessing module, a feature extraction module, an iterative learning module and a residential size prediction module.

[0016] The acquisition and preprocessing module acquires the rural residential image from the image acquisition unit, and after filtering, identifies the position of the residential in the image as the ROI region by a trained target classification model according to the deep learning features; removes the portion outside the ROI region in the original rural residential image to form the residential target image, and then forms a binary residential edge image through edge detection; after performing Hough line transformation on the binary residential edge image, the straight line segments with the longest length in the vertical and horizontal directions are respectively selected as candidate straight line segments according to the polar coordinate space parameter characteristics in the Hough line transformation, and two relative sidelines with a pair of long sides, wide sides and high sides in the middle are respectively determined according to the orientation characteristics of the candidate straight line segments as a pair of sidelines of the length, width and height of the residential, and together form three pairs of sidelines of the residential;

[0017] For each edge line of any pair of edge lines among the three pairs of edge lines, the feature extraction module combines the corresponding binary residential edge image, describes it with the coordinates of the two end points of the straight line, and respectively obtains the slope of the straight line and the perpendicular distance from the center point of the image to the straight line based on the two end points; the slope of the straight line on both sides of the residential building and the perpendicular distance from the center point of the image to the straight line are used as the geometric features of the residential building in the rural residential building image;

[0018] The iterative learning module trains the neural network of the residential building size prediction model in the residential building size prediction module based on the collected training samples;

[0019] The general processing module responds to the message, starts identifying residential buildings in the captured image, schedules the identification processing flow, and outputs the size data identified and predicted by the residential building size prediction module.

[0020] Preferably, a target detection unit is provided in the acquisition and preprocessing module, in which a target classification model based on the ASFF-YOLOv5s network is established. After the bird's-eye view camera in the image acquisition unit is used to acquire the rural scene image, the network is offline trained with an image sample set consisting of the rural scene images with the rural residential areas marked, so as to obtain a rural scene target classification model. When running online, the acquisition and preprocessing module uses the rural scene target classification model to identify and process the rural scene image including residential buildings to be tested, i.e., the rural residential images, to obtain an anchor frame of the rural residential area, and uses the area within the anchor frame as the ROI area of ​​the rural residential buildings. The ASFF-YOLOv5s network adds three layers 24, 25, and 26 between the 23rd and the next layer of the YOLOv5s network, and these three layers receive the outputs of the three layers 17, 20, and 23 as inputs, respectively, and after the ASFF adaptive spatial feature fusion module performs feature fusion of different scales, the three layers replace the original three layers 17, 20, and 23 in turn and input them into the 24th layer of the original output layer, i.e., the 27th layer of the new network.

[0021] As a preferred method, for residential images taken based on residential models, the collected images are filtered and grayed, and then binarized to form residential target images; for rural residential images collected online, the target classification model based on the ASFF-YOLOv5s deep learning network uses an anchor frame to locate the residential target ROI area, and the pixels outside the obtained ROI area are set to zero to form the residential target image. The edge extraction can be directly performed on the binary residential target image, and for the color or grayscale residential target image, the color image is first converted into a single-channel image, and then the edge detection operators such as Sobel, Canny, and LOG are used to detect the edge.

[0022] Preferably, the residential size prediction model adopts an artificial neural network, whose input layer receives four geometric feature inputs, namely the slope k1, k2 of a pair of sidelines among the three pairs of sidelines corresponding to the length, width, or height of the residential house and the vertical distances D1, D2 from the center point of the image to the pair of sidelines, from the general processing module, and the output of the output layer is the length, width, or height of the residential house corresponding to the pair of sidelines.

[0023] Preferably, the host unit is further configured to: when performing the Hough line transform on each edge pixel point in the residential edge image, accumulate and obtain the ρ-θ plane matrix corresponding to the image, where ρ represents the vertical segment distance of the straight line from the origin, and θ is the angle of the vertical line to the horizontal axis; in the accumulation, the foreground pixel point (u, v) in the image and a straight line element l i,j (ρ i,j ,θ i,j ) The corresponding judgment method is:

[0024] Take a point H(ρ i,j ·cosθ i,j , ρ i,j sinθ i,j ), recorded as H(u′, v′), calculate J = |vv′-(uu′)·tanθ i,j |, if |J|≤ε, then the pixel point is determined to be on the straight line and accumulated, where ε is a preset small positive number.

[0025] Preferably, the host unit is further configured to: when performing the Hough line transform on each edge pixel point in the residential edge image, accumulate and obtain a ρ-θ plane matrix corresponding to the image, wherein ρ represents the distance of the vertical segment of the straight line from the origin, and θ is the angle of the vertical line to the horizontal axis;

[0026] Based on the accumulated ρ-θ plane matrix, search for the two or three longest edges in a range close to the vertical direction as the side edges of the residential building;

[0027] In the range of non-side edge direction, the plane matrix element values, i.e. the length of the straight line segments, are sorted from large to small, starting from the main straight line segment with the longest length and searching for two secondary straight line segments close to its direction in sequence, and the main straight line segment and the corresponding two secondary straight line segments form a candidate straight line group; for any straight line segment in the candidate straight line group, the distance between it and the upper and lower endpoints of the side edge of the residential building is calculated, and if the minimum value of the distance corresponds to the distance between the straight line segment and the upper endpoint of a side edge, the straight line segment is identified as a roof candidate straight line, otherwise it is identified as a ground candidate straight line. Multiple groups of candidate straight line groups are obtained from straight lines in different directions in a loop.

[0028] Preferably, the host unit is further configured to: in one of the candidate straight line groups, the first pair of sidelines are formed by two relative sidelines, namely, the residential roof candidate straight line and the ground candidate straight line, which sandwich a pair of side edges in between. In one of the candidate straight line groups, the second pair of sidelines and the third pair of sidelines are formed by two relative sidelines, namely, the two residential roof candidate straight lines, which sandwich a pair of other straight line segments in between; or, in the side edges of the residential buildings, the second pair of sidelines and the third pair of sidelines are formed by two relative sidelines, namely, the two residential side edges, which sandwich a pair of other straight line segments in between.

[0029] Preferably, the host unit is further configured to: calculate and extract geometric features for any pair of edge lines among the three pairs of edge lines,

[0030] Assuming that the coordinates of the two endpoints of a residential boundary line are (x1, y1) and (x2, y2), then its slope is k = y1-y2 / x1-x2;

[0031] The vertical distance from the center of the image to the edge of the residential building is

[0032] Where (x0, y0) represents the coordinates of the center of the image, A and B are the coefficients of the general formula of the straight line equation, C is a constant term and: A = y2-y1, B = x1-x2, C = -x1(y2-y1)+y1(x2-x1);

[0033] According to the calculation formulas of k and d, the slopes k1 and k2 of any pair of edge lines of the residential edge and the vertical distances D1 and D2 from the center point of the image to the two edge lines are calculated respectively, a total of four quantities as the geometric features of the residential buildings in the rural residential image.

[0034] Preferably, the BP neural network used in the residential size prediction model has a three-layer structure including a hidden layer, wherein:

[0035] The output of the jth node in the hidden layer is

[0036] The output of the output layer node is

[0037] Among them, the f() function is taken as the tansig function, wij and vj are the connection weights from the input layer to the hidden layer and the connection weights from the hidden layer to the output layer respectively, aj and b are the hidden layer and output layer thresholds respectively, m=4 is the number of input variables, k is the number of hidden layer nodes, and the gradient descent method is used for network training.

[0038] Preferably, the host unit is further configured as follows: before online application, firstly, in an offline state, a drone is used to collect images of the manufactured residential model and form a sample set, and the residential size prediction model is trained with the sample set; the residential model adopts a rectangular parallelepiped, and the size of the rectangular parallelepiped is a first preset multiple S1 of the actual residential size, and the multiple S1 is set according to the rural cruising flight altitude of the drone and the camera resolution; the different rectangular parallelepipeds adopted by the residential model have a size range corresponding to S1 times the common size range of rural residential houses, and the size difference between different rectangular parallelepipeds corresponds to S1 times the actual residential size difference SD.

[0039] Preferably, the first preset multiple S1 is between 1 / 50 and 1 / 20, and the actual residential size difference SD is between 0.5 and 3 meters; the length, width and height of each rectangular block used as a residential model can preferably form an arithmetic progression with S1 times SD as the tolerance.

[0040] Preferably, during the collection of the sample set, the residential models are arranged in different areas of the ground field of view, so that the residential models in the sample set are distributed in each square of the nine-square grid formed by the top, middle, bottom, left, middle and right in the image plane.

[0041] Preferably, the residential size prediction model established in the residential size prediction module receives the geometric features of the residential houses in the rural residential image from the general processing module at its input layer, and transmits the features to the iterative learning module and the general processing module through the first connection matrix and the second connection matrix respectively at its output layer; when training the residential size prediction model offline, the iterative learning module adjusts the connection weights of the network according to the actual values ​​of the residential sizes input into the general processing module and the residential size prediction model through the first connection matrix respectively and the output values ​​of the neural network in the model; in the field environment, the first connection matrix is ​​disconnected, and the residential size prediction model neural network predicts the size of the residential houses in the current rural residential image and outputs it to the general processing module through the second connection matrix, which is processed and analyzed by the general processing module and then output through the output part in the input-output module.

[0042] Preferably, the host unit is further configured to: obtain the coordinate information and orientation information of the current residence through the positioning unit, and after detecting the size of the residence, output the residence size and corresponding coordinate and orientation information through the input and output unit.

[0043] Preferably, after identifying the size of the dwellings, the size of the dwellings is also marked on the corresponding extracted edge segment of the rural dwelling image, such as by using an arrow to mark the corresponding direction segment; preferably, the acquired coordinates and position information, as well as the size of the dwellings, are also stored in a list of a storage module, and the list information can be stored in a cloud server.

[0044] When the length and width in the size data are both greater than the preset safety distance threshold, the coordinate position value of the central point on the roof of the residential building is given; otherwise, an alarm is issued to indicate that the safety distance of the residential roof is insufficient. During the flight of the drone, image acquisition is performed iteratively and continuously, and size prediction processing is performed based on the acquired images. When the length and width appear within the field of view that are greater than the preset safety distance threshold, a prompt is issued that there is a residential building with sufficient safety distance. Preferably, the image is updated and processed according to the flight speed and frame rate so that the residential buildings in the field of view are identified one by one. After finding a venue with sufficient safety distance, the safety distance of the venue is verified by acquiring residential images from multiple angles and performing size prediction.

[0045] Preferably, a plurality of the neural network-based dwelling size prediction models are created in the host unit, each dwelling size prediction model corresponds to a flight altitude and a bird's-eye view angle of an image acquisition unit; when performing online dwelling size prediction, the geometric features obtained after the current acquisition process are input into the dwelling size prediction model corresponding to the flight altitude and the bird's-eye view angle during acquisition, and the model output value is obtained as the dwelling size prediction value.

[0046] Preferably, in the BP neural network of the residential size prediction model, the input layer receives six input quantities, namely, the slopes k1 and k2 of the straight lines on both sides of the residential houses, the vertical distances D1 and D2 from the center point of the image to the straight lines on both sides of the residential houses, and the flight altitude and the bird's-eye view angle during collection, from the general processing module respectively, and the output of the output layer is the residential size.

[0047] In another embodiment of the present invention, a rural dwelling identification system under the field of view of a drone is provided, which includes: an image acquisition unit for performing image sensing acquisition of rural scenes, a user interface unit for parameter input and operation and rural dwelling information display, a server for data management and information transmission, a handheld terminal for remote operation and rural dwelling information interaction, and a host unit connected to the user interface unit, the server, and the image acquisition unit; the host unit includes an input and output module, a processor, and a storage module, and is configured as follows:

[0048] Acquire a rural residential image, determine a ROI region containing residential buildings in the rural residential image, extract residential buildings from the ROI region as a residential building target image; obtain a residential building edge image by edge detection for the residential building target image; perform Hough line transform on the residential building edge image, and obtain a residential building edge image including three pairs of edge lines in the residential building contour edge by heuristic search, wherein the first pair of edge lines of the three pairs of edge lines are two opposite edge lines of the top and bottom of the residential building with a pair of side edges in the middle, and the second pair of edge lines and the third pair of edge lines of the three pairs of edge lines are two opposite edge lines of the residential building with a pair of horizontal long sides and horizontal wide sides in the middle, respectively; obtain geometric features of the two opposite edge lines relative to the image center for any pair of edge lines of the three pairs of edge lines in the residential building edge image in turn, input the geometric features into a trained residential building size prediction model based on a neural network, obtain a residential building size prediction value, and output the size prediction value.

[0049] Preferably, the rural residential identification system under the vision of the drone may not include the server and the handheld terminal, but the rural residential identification system under the vision of the drone is connected to the server and the handheld terminal based on a communication network during application. A positioning unit for obtaining coordinates may also be provided in the rural residential identification system under the vision of the drone.

[0050] Preferably, the rural residential identification system under the field of view of the drone also includes a flight control platform. In response to the user's operation on the operation panel or the instruction of the host unit, the flight control platform executes the instruction and flies towards the designated residential location.

[0051] The rural residential identification system under the field of view of the UAV continuously detects multiple frames of rural residential images and predicts the size of the residential buildings. When the safety distance of the residential buildings meets the requirements, the sizes of multiple residential buildings are compared, and the building with the largest safety distance is determined as the target landing point. The coordinate position information of the target landing point is output to the flight control platform through the output unit, and the flight control platform controls the power unit to make the UAV fly to the target landing point.

[0052] Preferably, when drones are used in group systems such as logistics and transportation, the system may have multiple handheld terminals and drone platforms, and subscription and publishing services may be provided for this purpose; the host unit is also configured to: subscribe to the acquisition information topic of the handheld terminal, and in response to the topic message, transmit the detected residential size and corresponding coordinates and location information to the handheld terminal through the server; or when it is detected that the residential size reaches a set safety distance threshold, transmit the residential size and corresponding coordinates and location information to the server in the form of a size event topic message.

[0053] Preferably, the user interface unit is provided with an operation panel for parameter input, and is also provided with a display screen for assisting parameter input and displaying the residential size recognition results in the form of a list; the host unit also displays the recognized residential size in segments in the form of a curve on the display screen, and marks the coordinates and positions corresponding to the recognized size on the curve.

[0054] Preferably, the positioning unit in the system also obtains heading information, and also records and identifies the direction identified based on the heading, such as the north direction, and the direction corresponding to the axis of the roof of the residential building in the output position information.

[0055] Compared with the prior art, the scheme of the present invention has the following advantages: the present invention aims at the unpredictable relative orientation relationship between the posture of the drone when collecting images and the orientation of the houses, analyzes the characteristic representation of the size of the houses in the rural house images in the image space, decomposes the size of the houses into three values ​​of length, width and height in three-dimensional directions, and proposes to use two relative sidelines respectively sandwiching a pair of long sides, wide sides and high sides to describe the size of the houses in three directions respectively, and uses the four geometric features of the slope of the two relative sidelines of the rural houses in any direction and the vertical distance from the center point of the image to the two sidelines as constraints to establish an artificial neural network to predict the size of the houses. Samples are collected to identify and train the nonlinear mapping relationship between the size of rural dwellings and the four proposed geometric features. Each sample can select rural dwellings by classifying and identifying targets in aerial images. The three pairs of edge lines corresponding to the length, width and height of the edge of the dwelling outline are obtained by Hough line transform screening. The four geometric features are extracted for two edge lines in any pair of extracted edge lines, and the results are input into the trained dwelling size prediction model to make online predictions of the length, width and height of rural dwellings. The residential size prediction model based on neural network established by the present invention has good generalization performance, can accurately predict the three-dimensional sizes of residential houses of various specifications according to the rural residential images collected from the uncertain navigation direction, and has a wide range of applications. Since the features are extracted from the straight line and do not rely on the precise pixel start and end points, the prediction accuracy is high; at the same time, since the established model is highly robust, when extracting the residential edge lines, the step size in the accumulation matrix of the Hough line transform does not need to be set too small, and the correspondence between the edge points and the straight line elements in the matrix simplifies the judgment, reduces the amount of calculation, speeds up the search speed, and further improves the real-time recognition of rural residential houses; and based on the heuristic search, the horizontal edges and vertical side edges of the ground and roof of the residential houses can be automatically found and marked, simplifying the operation confirmation steps.

[0056] It should be understood that all combinations of the aforementioned concepts and the additional concepts discussed in more detail below (provided that such concepts are not mutually inconsistent) can be considered as part of the inventive subject matter disclosed herein. In particular, all combinations of the claimed subject matter appearing in the disclosed solution can be considered as part of the inventive subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a schematic diagram of a rural residential scene;

[0058] Figure 2 It is a schematic diagram of the structure of the rural residential identification device and system under the vision of the drone;

[0059] Figure 3A This is a structural diagram of the host unit / controller in the rural residential identification device under the vision of the drone. Figure 3B To obtain and preprocess the module structure diagram, Figure 3C This is the structural diagram of the feature extraction module;

[0060] Figure 4A This is a schematic diagram of the neural network model structure of the residential size prediction module. Figure 4B This is a schematic diagram of the partial structure of the host unit / controller in the rural residential identification device under the vision of the drone;

[0061] Figure 5A This is a schematic diagram of the drone's posture during sample collection. Figure 5B This is a schematic diagram for comparing residential models of different sizes;

[0062] Fig. 6A , Figure 6B This is a schematic diagram of residential models photographed at different angles during sample collection. Figure 6C This is a schematic diagram of the single-sided outline of a residential building. Fig.6D The residential model image after Hough line transformation straight line detection;

[0063] Fig. 7A , Fig. 7E The schematic diagram of the proposed residential model edge constraints and geometric features is shown in Figure 2. Figure 7B , Figure 7C , Fig.7D The diagrams of length, width and height constraints and geometric features of rural houses are as follows;

[0064] Figure 8 Fitting curve between actual and predicted values ​​of residential model width in the test set;

[0065] Fig. 9 Fitting curve between actual and predicted values ​​of residential model length in test set;

[0066] Fig.10Fitting curve between actual value and predicted value of residential model height in test set;

[0067] Fig.11 Schematic diagram of target annotation for rural drone field of view images;

[0068] Fig.12 This is a schematic diagram of small, medium and large-scale targets in rural scenes;

[0069] Fig.13A Local diagram of YOLOv5s network configuration file, Fig. 13B This is a schematic diagram of the feature fusion of ASFF-3 and learning of features at different scales. Fig. 13C , Fig.13D They are the partial diagram of the ASFF-YOLOv5s network configuration file and the schematic diagram of the network structure;

[0070] Fig.14A , Fig. 14B A schematic diagram comparing the loss curves of the training set and validation set before and after improving the network;

[0071] Fig.15 Schematic diagram of the residential target anchor frame and residential size identification in the drone field of view image.

[0072] Among them: 1000 rural residential identification systems under the vision of drones, 900 rural residential identification devices under the vision of drones, 100 host units / controllers, 200 user interface units, 300 handheld terminals, 400 servers, 500 image acquisition units, 600 flight control platforms, 700 positioning units, 800 rural residential environments / scenes,

[0073] 110 processor, 120 storage module, 130 input and output module,

[0074] 111 general processing module, 112 acquisition and preprocessing module, 113 feature extraction module, 114 iterative learning module, 115 residential size prediction module,

[0075] 1121 image acquisition unit, 1122 target detection unit, 1123 residential area extraction unit,

[0076] 1131 edge detection unit, 1132 straight line extraction (screening) unit, 1133 slope feature extraction unit, 1134 distance feature extraction unit, 1135 side edge extraction unit, 1136 top and bottom edge extraction unit, 1137 roof parallel line extraction unit,

[0077] 1141 first connection array, 1142 second connection array,

[0078] 1101 neural network model, 131 input unit, 132 output unit,

[0079] 210 display screen, 220 operation panel. DETAILED DESCRIPTION

[0080] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, but the present invention is not limited to these embodiments. The present invention covers any substitution, modification, equivalent method and scheme made within the spirit and scope of the present invention.

[0081] Embodiment 1:

[0082] When drones are performing missions in rural areas, they need to measure the height of residential buildings to set the minimum flight altitude when flying over villages, and they need to measure the length and width of residential buildings when landing on rooftops or flying over houses. Therefore, identifying and measuring the size of rural residential buildings in the field of vision of drones is the basis of rural application of drones.

[0083] like Figure 1 As shown in the figure, in rural scenes, there are mostly farmland and vegetable fields with residential houses scattered among them. Drones provide transportation for rural areas and can provide landing sites on the roofs of residential houses or on the flat ground around residential houses. To ensure the safety of drones flying over and landing, it is necessary to understand the size of rural residential houses to make flight and landing decisions.

[0084] Since rural houses are often not of the same specifications, their sizes vary; and the property rights also belong to different entities, it is difficult to collect size data a priori. Therefore, the present invention provides a rural house identification device under the vision of a drone, which does not rely on prior three-dimensional size measurement and can estimate the size of rural houses by collecting images of rural houses under a non-fixed heading during flight.

[0085] like Figure 2As shown, the present invention provides a rural residential identification device 900 under the field of view of a drone, which includes a host unit 100 and a user interface unit 200; wherein the host unit 100 includes a processor 110, a storage module 120 and an input and output module 130; the host unit 110 is configured to: obtain a rural residential image, determine an ROI area containing residential buildings in the rural residential image, extract a residential area from the ROI area as a residential target image; obtain a residential edge image by edge detection of the residential target image; perform Hough line transformation on the residential edge image, and obtain three pairs of edges in the residential outline edge by heuristic search. The residential boundary line image is a residential boundary line image of three pairs of lines, wherein the first pair of line lines among the three pairs of line lines are two opposite line lines of the top and bottom of the residential boundary with a pair of side edges in between, and the second pair of line lines and the third pair of line lines among the three pairs of line lines are two opposite line lines with a pair of horizontal long sides and horizontal wide sides in between respectively; from the residential boundary line image, for any pair of line lines among the three pairs of line lines in turn, geometric features of the two opposite line lines relative to the image center are obtained, the geometric features are input into a residential boundary size prediction model based on a trained neural network to obtain a residential boundary size prediction value, and the size prediction value is output by the output unit 132 in the input-output module 130.

[0086] See also Figure 1 As shown, in the images collected in the rural scenes, the residential houses are approximately displayed as a rectangular shape; and generally, when the drone collects images, the lateral direction of the imaging surface is parallel to the horizontal plane, and the longitudinal axis in the middle of the imaging surface is approximately parallel to the image formed by the plumb line. However, due to the perspective transformation effect, in the image, any two parallel edges or side edges of the actual residential houses appear to converge at a vanishing point at infinity. On the one hand, it is obvious that when measuring the distance from the pixel points at both ends of the edge segment of the residential house in the image and then converting it back to the actual width according to the proportional relationship, the proportional relationship is difficult to accurately determine, and the ratio will be directly related to the front-back or left-right position of the edge in the image. On the other hand, referring to Figure 1 In actual applications, the scenes shown in the left and right figures cannot force the drone to face a precise direction of a certain surface of the residential building for proportional measurement, that is, to shoot directly at the residential building for image collection. Otherwise, the applicability of the identification will be greatly limited.

[0087] Although the physical size of rural houses in the field of view is certain, when it is sampled from different headings, its edge is uncertain in both pixel width and direction in the image space. This certain physical size and uncertain image features are in a contradictory relationship that is difficult to unify. Therefore, this contradiction needs to be decoupled and modeled. The heading of the drone is not fixed during flight, and the orientation of the houses varies greatly; therefore, the relative relationship between the heading of the drone and the orientation of the houses is unpredictable. In order to improve the versatility and applicability of the size identification of houses, it is necessary to study how to characterize the relationship between the two in various relative orientations in the image space, that is, what kind of features in the image can determine the size of the houses?

[0088] After observing the length, width and height edges of various residential buildings in different directions in the image, it is found that the key lies in the two residential edge lines on both sides of the length, width or height edge; but the residential edge itself is not the only way to determine the size of the residential building, and the angle of the line itself can only reflect the relative angle difference between it and the heading. After a large number of experiments and tests, it is also found that the values ​​of length, width or height, that is, the relative distance between the two residential edge lines on both sides, will affect the imaging effect of the two parallel sides in the image. So, how to characterize the two sides? Through further research, hypothesis and reasoning, it is proposed to use the slopes k1 and k2 of the two parallel lines at different spacings and the vertical distances D1 and D2 from the center point of the image to the two sides of the line as the constraint conditions for the spacing between the two lines, that is, the size of the edge of one of the orientations of the residential buildings in the image space. Further, how to correspond this constraint condition to the size of the residential building? Since there is a nonlinear and coupled mapping relationship between the two, inspired by the fitting application of neural networks in system nonlinear modeling, the present invention proposes to establish a residential building size prediction model using artificial neural networks.

[0089] Neural network is a machine learning model that imitates biological neural systems and is particularly suitable for nonlinear mapping modeling. It learns and represents complex functional relationships through a multi-layer artificial neuron structure. Neural network consists of an input layer, a hidden layer, and an output layer; each layer contains a number of neurons, and each neuron is connected to the neurons in the previous layer through weights. Among them, the input layer receives input data, the hidden layer performs nonlinear transformation and feature extraction, and can have multiple layers, and the output layer outputs the final prediction result.

[0090] like Figure 4AAs shown, the residential size prediction model of the present invention adopts a three-layer BP neural network with one hidden layer, and its input layer receives four geometric feature inputs, namely, the slope k1, k2 of a pair of sidelines corresponding to the length, width, or height of the residential building among the three pairs of sidelines and the vertical distances D1, D2 from the center point of the image to the pair of sidelines, from the general processing module, and the output of the output layer is the length, width or height of the residential building corresponding to the pair of sidelines.

[0091] To extract the four input quantities or geometric features of rural houses in the image space, it is necessary to first obtain the two straight lines corresponding to the length, width or height of the houses from the image, that is, the two edge lines. Figure 3A As shown, in the processor 110 of the host unit or controller 100, multiple processing modules are set, namely, a general processing module 111, an acquisition and preprocessing module 112, a feature extraction module 113, an iterative learning module 114 and a residential size prediction module 115. Among them, the general processing module 111 responds to the message and the operation of the user interface unit, and starts the identification of residential houses in the collected image, schedules the identification processing flow, inputs the rural residential images collected in real time into the acquisition and preprocessing module 112, and outputs the size data identified and predicted by the residential size prediction model in the residential size prediction module 115.

[0092] The acquisition and preprocessing module 112 acquires the rural residential image taken by the image acquisition unit 500, removes high-frequency noise through filtering, and then determines the ROI area including the residential area according to deep learning features or color texture and other features; removes the part outside the ROI area in the original rural residential image to form a residential target image, and then forms a binary residential edge image through edge detection; after performing Hough line transformation on the binary residential edge image, the straight line segments with the largest length in the vertical and horizontal directions are respectively selected as candidate straight line segments according to the polar coordinate space parameter features in the Hough line transformation, and According to the orientation features of the candidate straight line segments, two relative sidelines with a pair of long sides, wide sides and high sides in between are determined respectively as a pair of sidelines of the length, width and height of the residential buildings, and together form three pairs of sidelines of the residential buildings; for each sideline of any pair of sidelines in the three pairs of sidelines, the feature extraction module combines the corresponding binary residential edge image, describes it with the coordinates of the two end points of the straight line, and obtains the slope of the straight line and the vertical distance from the center of the image to the straight line based on the two end points; the slope of the straight line on both sides of the residential building and the vertical distance from the center of the image to the straight line are used as the geometric features of the residential buildings in the rural residential building image. Preferably, the slope of the straight line can be obtained from the angle corresponding to the straight line in the Hough line transform accumulation, i.e., the accumulator array.

[0093] like Figure 3BAs shown, in the acquisition and preprocessing module 112, it is further divided into an image acquisition unit 1121, a target detection unit 1122, and a residential area extraction unit 1123. The image acquisition unit 1121 acquires the rural residential image acquired in real time by the external image acquisition unit 500 through the general processing module 111, or acquires the pre-stored residential image from the storage module 120 in an offline state. The residential area extraction unit 1123 preliminarily determines the ROI area including the residential area in the image; preferably, a target classification model based on the YOLOv5s network can be established in the target detection unit 1122, and after the rural scene image is acquired by the overhead camera in the image acquisition unit, the network is offline trained with an image sample set composed of the rural scene images labeled with various target areas such as rural residential areas to obtain a rural scene target classification model; when running online, refer to Fig.15 As shown, the residential area extraction unit 1123 uses the rural scene target classification model to identify and process the rural scene image including residential houses, i.e., the rural residential image, to obtain a rural residential area anchor frame, and uses the area within the anchor frame as the ROI area of ​​the rural residential houses.

[0094] In the YOLOv5 model series, the YOLOv5s model has the smallest size and the least training parameters. Comparative experiments show that YOLOv5s can balance recognition accuracy and image processing frame rate, and is suitable for deployment on mobile devices. The classic YOLOv5s network structure is divided into three parts. The first part is Backbone, which is called the backbone network part, used to extract features from input images; the second part is Neck, which is used to extract features of different scales from network structures of different depths in Backbone for feature fusion, so that feature information of different depths can be retained to the maximum extent, which is more conducive to the learning of network model parameters during training; the third part is Output, that is, output prediction, which mainly predicts the input image with tensors of different scales after feature fusion.

[0095] In rural transportation scenes, there are often some targets with rural characteristics, such as high-voltage towers, puddles, small rivers, small bridges, signal towers, houses and greenhouses. Under non-inclement weather conditions, a large number of target images in rural scenes are obtained through drone aerial photography. In order to improve the richness of the data set, the images are taken in sunny weather and cloudy weather, so that different lighting conditions are also taken into account. After the data set is collected, the targets to be inspected in the image are annotated so that the images can be supervised when they are sent to the neural network training. Fig.11It shows the marking of the target object as a residential building; after marking all the target objects, click the Save button to complete the labeling of a picture and save it. The labeled file is saved in txt format, and the number of label files corresponds to the number of picture files. Each line in the label file represents a target position in the picture, and the saved information is from left to right, including the target type, the x-coordinate of the target center point, the y-coordinate of the target center point, the width of the target, and the height of the target; the data uses normalized values.

[0096] The data set is randomly divided into a training set and a validation set according to a certain ratio. After training the training set, the validation set is used to validate the network. The results show that the AP of residential buildings on the validation set is not high, only 0.669. Therefore, after analyzing the characteristics of the sample and the characteristics of the deep network, it is decided to start with the characteristics of the sample to improve the performance of the classification network.

[0097] In rural scenes, the size of objects varies significantly. Fig.12 As shown, the high-voltage tower, residential buildings and small river correspond to small targets, medium targets and large targets in the image respectively. Based on this feature, the present invention proposes to introduce the ASFF adaptive spatial feature fusion module into the deep learning network to enhance its prediction and recognition capabilities for features of different scales.

[0098] like Fig. 13B As shown in Figure 2, the process of learning weights of feature maps of different scales by the ASFF module is as follows: 1) Upsampling the feature maps of Level 1 and Level 2 to the same scale as Level 3, and the transformation process is recorded as X 1→3 , X 2→3 and X 3→3 ; 2) Use a 1*1 convolution kernel to compress the number of channels at three scales to a uniform length; 3) Concatenate the compressed number of channels; 4) Use a 1*1 convolution kernel to further compress the number of channels to 3; 5) Obtain the weight information of the three different scale spaces through the Softmax classification function, i.e., α 3 , β 3 and γ 3 For the i-th layer, the corresponding ASFF fusion feature formula is:

[0099] ASFF i =X 1→i α i +X 2→i β i +X 3→2 γ i

[0100] A partial picture of the network configuration file after adding ASFF is shown below Fig. 13C As shown. Fig.13A , Fig. 13CAs shown, the ASFF-YOLOv5s network established by the rural scene target classification model of the present invention adds three layers 24, 25, and 26 between the 23rd and the next layer of the YOLOv5s network. These three layers receive the outputs of the 17th, 20th, and 23th layers as inputs, respectively, and after the features of different scales are fused by the ASFF adaptive spatial feature fusion module, they replace the original 17th, 20th, and 23th layers in turn and are input to the 24th layer of the original output layer, i.e., the 27th layer of the new network. Fig.13D Schematic diagram of the ASFF-YOLOv5s network structure.

[0101] After the ASFF-YOLOv5s model is trained, its performance is compared with the classic YOLOv5s. Fig.14A , Fig. 14B The training set loss curve and validation set loss curve of the two are compared. It can be seen that the loss value of ASFF-YOLOv5s is significantly lower than that of the YOLOv5s benchmark model, and the convergence speed is also significantly faster. When the training round is 100 epochs, the train_loss of ASFF-YOLOv5s and YOLOv5s are 0.1437 and 0.1140 respectively, and the val_loss are 0.0951 and 0.0915 respectively.

[0102] Compared with the decrease in the loss value of the network model, we are more concerned about the detection accuracy of ASFF-YOLOv5s on the validation set for each category. The comparison of AP of each category before and after the improvement of the network shows that the AP value of the ASFF-YOLOv5s model on the validation set is improved compared with the classic YOLOv5s benchmark model, among which the high-voltage tower is improved by 3.9%, the puddle is improved by 9.9%, the small river is improved by 5%, the small bridge is improved by 7.8%, the residential house is improved by 8%, the signal tower is improved by 8%, the greenhouse is improved by 3.8%, and the mAP is improved by 6.5% overall; it can be seen that the improved network model has significantly improved the detection accuracy of targets of various sizes.

[0103] After the target classification model based on the ASFF-YOLOv5s network locates the residential target with an anchor frame to obtain the ROI area of ​​the residential target, the residential area is extracted from the ROI area as the foreground object, and the non-foreground object area in the rural residential image is used as the background object. After all the background objects in the rural residential image are set to zero, the residential target image is formed, which facilitates the subsequent extraction of the residential edge straight line.

[0104] like Figure 3CAs shown, in the feature extraction module 113, an edge detection unit 1131, a straight line extraction and screening unit 1132, a side edge extraction unit 1135, a top and bottom edge extraction unit 1136, a roof parallel line extraction unit 1137, a slope feature extraction unit 1133, and a distance feature extraction unit 1134 are also set.

[0105] For residential target images, the edge detection unit 1131 performs edge extraction on the residential target images. Edge extraction can be performed directly on the binary residential target images, and for color or grayscale residential target images, the color images are first converted into single-channel images, and then the Canny edge detection operator is used to detect edges.

[0106] The straight line extraction and screening unit 1132 extracts the edge straight lines of the straight boundaries of the residential buildings, including the long sides, wide sides and high sides, from the binary residential edge image. First, analyze how to reflect the long sides, wide sides and high sides of the residential buildings? If you directly look for the corresponding line segments, then if the end point position of the line segment moves, it will cause obvious errors; more importantly, even if a boundary line segment is obtained, how do you know its actual length? After modeling analysis, the present invention describes the boundary edge with two relative edge lines on both sides of any boundary edge of length, width and height, perpendicular to the boundary edge and sandwiching the boundary edge. Secondly, analyze how to extract such relative edge lines. The present invention adopts Hough line transform.

[0107] The Hough line transform maps points in the image space to curves in the parameter space, such as the polar coordinate space, and then finds the intersection of the curves in the parameter space to detect the straight line. In the image space, the equation of the straight line can be expressed as: y = kx + b; using polar coordinate notation, the equation of the straight line can be expressed as: ρ = xcosθ + ysinθ, where ρ is the vertical distance from the straight line to the origin of the coordinates, and θ is the angle between the perpendicular line of the straight line and the positive direction of the x-axis. The essence of the Hough line transform is to establish a mapping relationship between the image space and the parameter space. On the basis of making full use of the global characteristics of the image, the point-line duality characteristics are used to map the pixel coordinates to the straight line represented by the elements in the parameter space, and then the analysis is performed in the parameter space. The detection task is completed by cumulative statistics in the parameter space.

[0108] In order to detect straight lines, a plane matrix, namely an accumulator array (ρ, θ), is created in the ρ-θ plane based on the Hough line transform, and the parameter space is divided into discrete grids; then, for each edge point (x, y) in the image space, the ρ value of ρ = xcosθ + ysinθ under different θ is calculated, and the corresponding position in the accumulator array is voted; after voting, peak detection is performed, and the element with a larger value in the accumulator array is the detected straight line parameter (ρ, θ). Simultaneously, a heuristic search is performed to obtain a residential edge image including three pairs of edge lines in the residential contour edge.

[0109] Specifically, refer to Figure 1 As shown, the drone is in a level flight state when collecting images; therefore, generally, the wall edge of the plumb bob on the ground appears to be approximately parallel to the longitudinal axis in the image. According to the heuristic rule, the side edge extraction unit 1135 searches for the two or three longest side lines as the side edges of the residential building within the range of the straight line close to the vertical direction based on the accumulator array (ρ, θ); wherein the vertical line close to the vertical direction has θ within the vicinity of 0 degrees. Generally, three or more side edges can be found in residential buildings, and only when the drone is facing the side wall of the residential building, there are only two side edges.

[0110] After excluding the straight lines in the side edge direction, find the two straight lines with the longest length and close directions in the set of straight lines obtained by voting to form a candidate straight line pair. According to the degree of proximity of the θ values, several subsets of straight lines with close directions can be clustered. For each subset, 3 to 5 candidate straight lines can be screened. In the range of non-side edge directions, the lengths of the straight line segments are sorted from large to small according to the values ​​of the accumulator array elements, and the two secondary straight line segments close to the main straight line segment with the longest length are searched in sequence. The main straight line segment and the corresponding two secondary straight line segments form a candidate straight line group. Preferably, 2 to 3 candidate straight line groups are obtained; some residential buildings have multi-story stepped shapes, and multiple pairs of parallel edges will appear.

[0111] Combination Figure 1 As shown, it can be found that the horizontal boundaries on the roofs and ground of the houses on the left and right sides have straight edges with nearly parallel directions in the field of view. To this end, all side edge groups are recorded in array form, and the upper and lower endpoints of each side edge are obtained by comparing the pixel point set corresponding to the same (ρ, θ) recorded in the Hough line transform accumulation process with the minimum and maximum values ​​of the ordinate. For any straight line segment in the obtained candidate straight line group, calculate its distance from the upper and lower endpoints of the side edge of the residential building. If the element with the smallest value in the distance set corresponds to the distance between the straight line segment and the upper endpoint of a side edge, then the straight line segment is identified as a roof candidate straight line. Otherwise, if the element with the smallest value in the distance set corresponds to the distance between the straight line segment and the lower endpoint of a side edge, it is identified as a ground candidate straight line.

[0112] According to the directions corresponding to different candidate straight line groups and the identifiers of the candidate straight lines in the groups, the top and bottom edge extraction unit 1136 and the roof parallel line extraction unit 1137 respectively determine the three pairs of sidelines corresponding to the length, width and height of the residential outline edge. The candidate straight line groups are sorted according to the length of the main straight line segments in their respective groups, and the two candidate straight line groups with the largest corresponding lengths are respectively recorded as the first candidate straight line group and the second candidate straight line group. First, the top and bottom edge extraction unit 1136 forms the first pair of sidelines with two relative sidelines of the residential roof candidate straight line and the ground candidate straight line sandwiched between a pair of side edges in one of the candidate straight line groups. Then, the roof parallel line extraction unit 1137 forms the second pair of sidelines with two relative sidelines of the two residential roof candidate straight lines sandwiched between a pair of candidate straight lines in the first candidate straight line group in the second candidate straight line group; similarly, in the first candidate straight line group, the third pair of sidelines are formed with two relative sidelines of the two residential roof candidate straight lines sandwiched between a pair of candidate straight lines in the second candidate straight line group.

[0113] The first pair of side lines sandwich a pair of side edges or high edges of the residential building. The vertical coordinates of the upper and lower endpoints of a side edge can be accommodated between the vertical coordinates of each corresponding endpoint of the first pair of side lines, i.e., the roof candidate straight line and the ground candidate straight line; that is, the interval covered by the vertical coordinates of each corresponding endpoint of the roof candidate straight line and the ground candidate straight line basically approaches or includes the vertical coordinates of the upper and lower endpoints of the side edge.

[0114] In the Hough line transform, the ρ distance resolution is usually 1 pixel, while the θ angle resolution is usually π / 180. Since it is found that the boundary straight line features in rural dwellings have relatively obvious characteristics, the Hough line transform processing process can be optimized. As a preference, when traversing and accumulating the element values ​​in the ρ-θ plane matrix, the spacing between elements in the value interval can take a larger value according to a priori testing, that is, the spacing between distances and angles is increased accordingly, and the number of rows and columns of the matrix is ​​reduced, thereby greatly reducing the amount of calculation in the calculation judgment and shortening the calculation time.

[0115] Preferably, before peak detection, the elements in the accumulator array (ρ, θ) are merged. When the values ​​of several adjacent elements in the array are above a preset pixel threshold, i.e., the minimum length of a straight line, the centroid of the polygon enclosed by the coordinates of the adjacent elements is obtained, and the (ρ, θ) corresponding to the obtained centroid is used as the residential boundary line.

[0116] For any pair of residential sidelines found, the slope feature extraction unit 1133 and the distance feature extraction unit 1134 respectively calculate the slope of the line and the vertical distance from the center point of the image to the line. Fig. 7AThe schematic diagram of the residential model edge constraints and geometric features shown in the figure shows a pair of two sides of the residential building. Among them, the angles and slopes of the straight lines contained in the two side lines on the left are α(k1) and β(k2), respectively, and the perpendicular distances from the center point of the image to the two side lines of the residential building on this side are D1 and D2 respectively; correspondingly, the angles and slopes of the straight lines contained in the two side lines on the right are α′(k1′) and β′(k2′), respectively, and the perpendicular distances from the center point of the image to the two side lines of the residential building on this side are D1′ and D2′ respectively. In the oblique photography image plane, the two parallel lines will intersect at a point O at the far end, and point O is called the vanishing point.

[0117] The algebraic equation of a line can be expressed in three common forms, namely the general form, the two-point form, and the slope-intercept form. The general form is: Ax+By+C=0, where A and B are not zero at the same time, and the slope of the line k=-A / B. In the Cartesian rectangular coordinate system, the intercept of the line on the y-axis b=-C / B, and the general form applies to all line equations.

[0118] The two-point formula is: The two-point equation is the equation of the straight line passing through the two points when the coordinates of the two points are known. The slope of the straight line is k = (y2-y1) / (x2-x1). It can be seen from the two-point equation that this equation is suitable for describing straight lines that are not perpendicular to the x-axis and y-axis.

[0119] The slope-intercept form is: y=kx+b. The slope-intercept form explicitly expresses the slope k and intercept b of the straight line, and is suitable for representing straight lines that are not perpendicular to the x-axis.

[0120] The straight line detected by Hough line transform in the image is described by the coordinates of the two end points of the straight line. The coordinates of the two end points are marked as (x1, y1) and (x2, y2). Substituting the coordinates of the two points into the two-point equation, and then transforming the two-point equation into the slope-intercept equation form, we get: The corresponding slope is k=y1-y2 / x1-x2. The only difference is that the calculation of the slope of the straight line is completed in the image coordinate system (the y-axis is downward in the positive direction), so the slope of the straight line in the image is equal to that in the Cartesian coordinate system, but has the opposite sign.

[0121] Combining the two-point equation, the perpendicular distance from the center of the image to the edge of the residential building can be calculated as: Where (x0, y0) represents the coordinates of the center of the image, A and B are the coefficients of the general formula of the straight line equation, C is the constant term and: A = y2-y1, B = x1-x2, C = -x1(y2-y1)+y1( x2 -x1). According to the calculation formulas of k and d, the slopes k1 and k2 of the straight lines on both sides of the residential buildings and the vertical distances D1 and D2 from the center point of the image to the two straight lines are calculated respectively, and a total of four quantities are obtained as the geometric features of the residential buildings in the rural residential images.

[0122] Combination Figure 1 The actual residential scene in Fig. 7A The concept of Figure 7B , 7C , 7D respectively illustrate the constraints and geometric characteristics of the length, width, and height of rural dwellings; among them, Fig.7D The two middle side lines are the first pair of side lines with a pair of side edges or high sides in the middle. Figure 7B , Figure 7C The two side lines in are the second pair of side lines and the third pair of side lines with a pair of long sides and wide sides in the middle. See also Fig. 7E As shown in the figure, for the determined residential boundary line, a vertical line is drawn from the center point of the image, represented by a dotted line. Fig. 7E In the figure, since the foot of the perpendicular is outside the range of the roof edge, the roof edge is extended to the foot of the perpendicular.

[0123] As a preference, a human-computer interaction interface is provided in the user interface unit. For residential buildings with complex outline structures, after automatic edge screening, the operator confirms and marks three pairs of edge lines through the human-computer interaction interface, obtains geometric features of the marked residential edge lines, and then predicts the size of the residential building. Figure 1 Taking the residential building shown in the figure as an example, there is a structure on the roof of the residential building on the left. In the human-computer interaction interface, the edge lines corresponding to the length and width of the structure can be further marked, and the geometric features of the marked structure can be calculated to predict its size. Finally, when calculating the length and width of the roof plane, the length and width obtained after subtracting the size of the structure from the overall length and width are used as the basis for determining the safe landing distance of the drone. Figure 1 For the residential house on the middle right, the roof edge is almost unobstructed, so it can be automatically screened first and then simply confirmed by the operator. The roof size can be directly obtained from the residential house size prediction model, which greatly simplifies the measurement operation and realizes the automatic measurement of the size of rural residential houses.

[0124] In order to reveal the nonlinear mapping structure of the residential size prediction model of the present invention, a rural residential scale model is established and then the model is tilted to obtain training image samples. Figure 5A As shown in FIG. 1 , when the drone is shooting the ground in the air, the angle between the optical axis of the onboard lens and the horizontal plane is α, which can be selected from a common working angle such as 45 degrees. Since the actual rural houses have different sizes, in order to facilitate modeling, the present invention uses a rectangular parallelepiped to make residential models of various specifications, such as Figure 5B shown.

[0125] In an offline state, a drone is used to collect images of the residential model and form a sample set, and the residential size prediction model is trained with the sample set. The residential model is in the shape of a cuboid, and the size of the cuboid is a first preset multiple S1 of the actual residential size. The multiple S1 is set according to the rural cruising flight altitude of the drone and the camera resolution; the different cuboids used in the residential model have a size range corresponding to S1 times the common size range of rural residential buildings, and the size difference between different cuboids corresponds to S1 times the actual residential size difference SD. For example, when the flight altitude is 100 meters, the sampling height of the corresponding model is 2 meters, so as to ensure that the residential model can occupy a significant image space during imaging, so as to coordinate the image resolution and the recognition accuracy of the residential size prediction model, and make it unnecessary for the drone to lower its altitude specifically for predicting the residential size, thereby increasing the scope of application of the prediction.

[0126] Preferably, the first preset multiple S1 is between 1 / 50 and 1 / 20, and the actual residential size difference SD is between 0.5 and 3 meters; the length, width and height of each rectangular block used as a residential model can preferably form an arithmetic progression with S1 times SD as the tolerance.

[0127] Using the residential model, three different actual drone cruising altitudes were simulated under laboratory conditions, and the camera was tilted at a 45° angle to shoot. real Compared with the aerial photography height h under laboratory conditions model The corresponding relationship is: when h real When the distance is 30m, 40m, or 50m, h model 667mm, 889mm, 1111mm, the scaling ratio is:

[0128]

[0129] refer to Fig. 6A As shown, due to the projective transformation, the straight lines that are parallel in the original three-dimensional cube become non-parallel in the two-dimensional image. The residential model is a three-dimensional model. The long side, wide side and high side of the residential building belong to different spatial positions and are combined into different planes in pairs. The present invention uses two opposite sides that are perpendicular to and sandwich the target side on a plane on a certain plane to express geometric feature constraints. For this purpose, feature extraction is performed from the edge lines on different planes.

[0130] Since the image sampling of the residential model is an oblique image, in order to ensure the diversity of the features representing the residential model in the image as much as possible and improve the generalization ability of the network model for predicting the size of the residential building, the residential model is photographed from 0°-180°. At the same time, in order to highlight the distance relationship between the drone and the actual location of the residential building, the residential models are arranged in different areas of the ground field of view, so that the residential models in the sample set are distributed in each square of the nine-square grid formed by the top, middle, bottom, left, middle, and right in the image plane. To simplify the explanation, Figure 6B Indicated Fig. 6A The top surface of the middle cuboid is at different positions in the image space when the sample set is collected. To improve the accuracy of the model, the image is shot with a black ground as the background. For the convenience of display, the image is inverted.

[0131] The present invention subdivides the size of residential models and forms an interval sequence, and shoots residential images from various relative angles during sampling, thereby ensuring the sufficiency of samples, thereby improving the generalization ability of the neural network and reducing the prediction error of the residential size.

[0132] For the rural residential images based on residential models, the collected images are filtered and grayed, then binarized to form residential target images, and then edge detection is performed to obtain residential edge images. Taking the top plane image of a residential model at a certain location as an example, the preprocessing results of each stage are as follows: Figure 6C As shown in the figure, from left to right they correspond to the processing results of grayscale, binarization, and edge detection. The Hough line transform is performed on the residential edge image after Canny edge detection. The transformed image is shown in Fig.6D As shown in the figure, the left and right figures are the Hough line transformation results of a pair of edge lines on both sides of the wide side. Afterwards, a heuristic search is performed on the straight lines obtained after the transformation to select the corresponding two side straight lines.

[0133] To ensure the correct detection and identification of residential boundary lines, a manual confirmation step can be added to the boundary search. Specifically, in the process of collecting straight line features representing residential buildings and establishing a training sample library, the interference of non-residential boundary straight lines is eliminated through the human-computer interface. With the help of the HighGUI module provided by openCV, a graphical user interface is established and the mouse callback function setMouseCallback() is used. In the callback function, a left mouse button click event and a right mouse button click event are created. Each time the left mouse button is clicked, a straight line detected by Hough line transform is drawn in the image, and it is confirmed after being compared with the original residential image.

[0134] Refer to the above Fig. 7AAs described in the description, for each model residential image, the four geometric features of the edge lines on both sides of a boundary are calculated, and the corresponding actual boundary size, i.e., length, width, or height, is recorded as the real value of the residential size. The geometric features and the real value of the residential size are formed into samples, and after the samples are collected and recorded to form a sample set, they are input into the residential size prediction model based on the neural network for offline training. Preferably, the sample set can be stored in the form of a data table composed of input and output real values ​​as a record.

[0135] The BP neural network used in the residential size prediction model in this embodiment is as follows: Figure 4A As shown, the model is a three-layer structure including a hidden layer, in which

[0136] The output of the jth node in the hidden layer is

[0137] The output of the output layer node is

[0138] In the formula, the f() function is taken as the tansig function, and w ij and v j are the connection weights from the input layer to the hidden layer and from the hidden layer to the output layer, respectively. j and b are the hidden layer and output layer thresholds respectively, m=4 is the number of input variables, k is the number of hidden layer nodes, and the gradient descent method is used for network training.

[0139] When predicting the size of residential buildings online, a pair of edge lines corresponding to the length, width and height of the residential buildings are obtained from the residential building model image. The length, width and height of the residential building model are predicted based on the four feature quantities of each pair of edge lines calculated. The prediction model also maps the model back to the actual residential building size as the actual output value based on the ratio of the model to the actual size.

[0140] Preferably, a plurality of the neural network-based residential size prediction models are created in the host unit, each residential size prediction model corresponds to a flight altitude and a bird's-eye view angle of an image acquisition unit; when performing residential size prediction online, the geometric features obtained after the current acquisition process are input into the residential size prediction model corresponding to the flight altitude and the bird's-eye view angle during acquisition, and the model output value is obtained as the residential size prediction value.

[0141] The sample data sets at three heights are randomly divided into training sets and test sets, both of which cover various residential models. The geometric features of the test set are input into the trained network model as input to obtain the predicted value. The fitting curves of the actual model value and the predicted value of the residential width at different heights are shown in Figure 1. Figure 8 As shown; accordingly, the fitting curve of the model actual value and predicted value of residential length is shown as Fig. 9As shown; the fitting curve of the model actual value and predicted value of residential height is shown in Fig.10 shown.

[0142] Analyze the regression results at three different laboratory heights, and the coefficient of determination R in the width dimension at three different heights 2 The R values ​​of the three heights are 0.99815, 0.99863 and 0.99649 respectively. The closer the coefficient of determination is to 1, the better the regression effect is, indicating that the model size prediction models at the three heights have high reliability. The root mean square error (RMSE) at the three different heights is 2.6277, 1.632 and 1.8552 respectively, indicating that the predicted value is close to the true value and the network model has good generalization ability. 2 Both the accuracy and RMSE indicate that the model has high reliability and good generalization ability.

[0143] Example 2

[0144] Based on the previous embodiment, Figure 2 As shown, this embodiment provides a rural dwelling identification system 1000 under the field of view of a drone, which includes: an image acquisition unit 500 for image sensing and acquisition of rural scenes, a user interface unit 200 for parameter input and operation and rural dwelling information display, a server 400 for data management and information transmission, a handheld terminal 300 for remote operation and rural dwelling information interaction, and a host unit 100 connected to the user interface unit 200, the server 400, and the image acquisition unit 500. The host unit 100 includes an input and output module 130, a processor 110 and a storage module 120, and is configured to: obtain a rural residential image, determine a ROI region containing residential buildings in the rural residential image, and extract a residential area from the ROI region as a residential target image; obtain a residential edge image by edge detection for the residential target image; perform Hough line transform on the residential edge image, and obtain a residential edge image including three pairs of edge lines in the residential contour edge by heuristic search, wherein the first pair of edge lines of the three pairs of edge lines are two opposite edge lines of the top and bottom of the residential building with a pair of side edges in the middle, and the second pair of edge lines and the third pair of edge lines of the three pairs of edge lines are two opposite edge lines with a pair of horizontal long sides and horizontal wide sides in the middle, respectively; obtain geometric features of the two opposite edge lines relative to the image center for any pair of the three pairs of edge lines in the residential edge image in turn, input the geometric features into a trained residential size prediction model based on a neural network, obtain a residential size prediction value, and output the size prediction value.

[0145] Preferably, the rural residential identification system 1000 under the drone vision may not include the server 400 and the handheld terminal 300, but the rural residential identification system under the drone vision is connected to the server 400 and the handheld terminal 300 based on a communication network during application. A positioning unit 700 for acquiring coordinates may also be provided in the rural residential identification system 1000 under the drone vision.

[0146] Preferably, the user interface unit 200 is provided with a display screen 210 and an operation panel 220. The rural residential identification system 1000 under the vision of the drone further includes a flight control platform 600. In response to the user's operation on the operation panel 220 or the instruction of the host unit 100, the flight control platform 600 executes the instruction and flies toward the designated residential site.

[0147] The rural residential identification system 1000 under the field of view of the drone continuously detects multiple frames of rural residential images and predicts the size of the residential buildings. When the safety distance of the residential buildings meets the requirements, the sizes of multiple residential buildings are compared, and the building with the largest safety distance is determined as the target landing point. The coordinate position information of the target landing point is output to the flight control platform 600 through the output unit, and the flight control platform controls the power unit to make the drone fly to the target landing point.

[0148] Preferably, when the drone is used in a group system such as logistics and transportation, the system may have multiple handheld terminals and drone platforms accordingly. Through the server, the host unit and the remote client deployed at the handheld terminal can send messages of residential size. Preferably, an MQTT server can be deployed, and the server pushes messages to the client subscribed to the topic. Multiple devices can subscribe to multiple topics at the same time, and when publishing messages, those who subscribe to the same topic can receive the same message. The publish-subscribe mode is a messaging mode that decouples the client that sends the message, i.e., the publisher, from the client that receives the message, i.e., the subscriber, so that the two do not need to establish a direct connection or know each other's existence.

[0149] The server provides subscription and publishing services, and the host unit is also configured to: subscribe to the topic of obtaining residential information of the handheld terminal, and transmit the detected residential size and corresponding coordinates and location information to the handheld terminal through the server in response to the topic message; or when it is detected that the residential size reaches a set safety distance threshold, transmit the residential size and corresponding coordinates and location information to the server in the form of a residential size event topic message.

[0150] Preferably, the user interface unit is provided with an operation panel for parameter input, and is also provided with a display screen for assisting parameter input and displaying the residential size recognition results in the form of a list; the host unit also displays the recognized residential size in segments in the form of a curve on the display screen, and marks the coordinates and positions corresponding to the recognized size on the curve.

[0151] Preferably, the positioning unit 700 in the system also obtains heading information, and also records and identifies the direction identified based on the heading, such as the north direction, and the direction corresponding to the axis of the roof of the residential building in the output position information.

[0152] Preferably, the host unit may also communicate directly with the handheld terminal via a wireless network.

[0153] The host unit can be a local control unit deployed separately on the ground, or a control unit deployed on a drone platform. When deployed on a drone platform, the host unit on the drone platform performs edge computing for online residential size prediction. Due to the limited resources of the edge computing unit, the processing calculation of the host unit is optimized.

[0154] In the general Hough line transform, for each edge point (x, y) in the image space, the ρ value of ρ = xcosθ + ysinθ under different θ is calculated, and the corresponding position in the accumulator array is voted. This process of calculating ρ requires two floating-point multiplication operations.

[0155] To this end, based on the obvious boundary straight line features in rural residential buildings, the pixel check accumulation process of Hough line transform is optimized. Specifically, when the Hough line transform is performed on each foreground pixel point (u, v) in the residential edge image, in the accumulation statistics, it is determined whether the pixel point is adjacent to a straight line element l i,j (ρ i,j ,θ i,j ) The corresponding judgment method is: take a point H(ρ i,j ·cosθ i,j , ρ i,j sinθ i,j ), recorded as H(u′, v′), calculate J = |vv′-(uu′)·tanθ i,j |, if |J|≤ε, then the pixel point is determined to be on the straight line and accumulated, where i and j are the row and column labels in the ρ-θ plane matrix, and ε is a preset small positive number.

[0156] By setting the fulcrum, a tangent operation is performed once before traversal judgment, so that each time the foreground pixel is judged, only two subtraction operations and one multiplication operation are required to perform judgment accumulation, thereby avoiding two time-consuming floating-point operations for each pixel, further saving computing costs, and thus suitable for edge computing deployed on drone platforms.

[0157] Preferably, in the heuristic search for obtaining the edge lines of residential houses, when the angle is expressed as a slope in a rectangular coordinate system, when the absolute value of the difference in the edge line slopes is less than a very small first slope threshold, the two edge lines are merged; at the same time, if the angle tangent value or the slope is greater than a very large second slope threshold, the edge line is determined to be a vertical line perpendicular to the horizontal axis, thereby preventing the edge line from being judged as different straight lines due to slight differences when the slope is very large, and avoiding the false detection interference of short edge lines on the edge lines of residential houses.

[0158] like Figure 4A As shown, as a preferred embodiment, in the sample collection, the present embodiment can group and record samples from the two sets of relative edges of the front side and the upper surface of the cube, and the output layer corresponds to the height, length and width respectively. Based on the subdivided sample set, different neural networks can be established for the height, length and width of the residential buildings respectively, and trained based on the corresponding sample sets. As a preferred embodiment, the width and length on the horizontal plane can also be combined for sampling and recording, without distinguishing between length and width.

[0159] Different from Example 1, in this embodiment, in the artificial neural network of the residential size prediction model, the input layer receives six input quantities, namely, the slopes k1 and k2 of the straight lines on both sides of the residential house, the vertical distances D1 and D2 from the center point of the image to the straight lines on both sides of the residential house, and the flight altitude and bird's-eye view angle during acquisition, from the general processing module, and the output quantity of the output layer is the residential house size. After establishing a neural network with this structure, the input quantity of each sample in the training set samples is supplemented, and after supplementing enough samples in different heights and bird's-eye view angles, it is trained with a new training sample set. When applied online, the new model can further simplify the image sampling requirements for rural residential prediction; it uses the generalization performance of the neural network to reduce the number of models required for rural residential size prediction, and also simplifies the system structure, so that one model can cover different flight sampling scenarios, further improving the applicability of the system.

[0160] Example 3

[0161] Based on the above embodiments, this embodiment provides a controller for identifying rural houses in the field of vision of a drone. Figure 2 , Figure 3A , Figure 4BAs shown, the rural residential identification controller 100 under the drone's field of view includes a processor 110, a storage module 120 and an input and output module 130. The processor 110 includes a general processing module 111, an acquisition and preprocessing module 112, a feature extraction module 113, an iterative learning module 114 and a residential size prediction module 115. Among them, the general processing module 111 responds to the message and the operation of the user interface unit, and starts the identification of the residential buildings in the collected image, schedules the identification processing flow, inputs the rural residential images collected in real time into the acquisition and preprocessing module 112, and outputs the size data identified and predicted by the residential size prediction module 115.

[0162] Combination Figure 3A , Figure 4B As shown, the rural residential image is input into the controller by the input unit 131 in the input-output module 130; the residential size prediction model established in the residential size prediction module 115 has an input layer receiving the geometric features of the residential buildings in the rural residential image extracted by the feature extraction module 113 from the general processing module 111, and an output layer transmitting the geometric features to the iterative learning module 114 and the general processing module 111 through the first connection matrix 1141 and the second connection matrix 1142 respectively.

[0163] When the residential size prediction model is trained offline, the iterative learning module 114 adjusts the connection weights of the network according to the actual values ​​of the residential size and the output values ​​of the neural network in the model respectively input by the general processing module 111 and the residential size prediction model through the first connection matrix 1141; in the field environment, the first connection matrix 1141 is disconnected, and the residential size prediction model neural network predicts the size of the residential buildings in the current rural residential image and outputs it to the general processing module 111 through the second connection matrix 1142, and is processed and analyzed by the general processing module 111 and output through the output unit 132 in the input-output module 130.

[0164] The steps of iterative training include: STEP1, initialize the weights and biases of the network; STEP2, read the network parameters and the training sample set written in the document above; STEP3, normalize the sample data; STEP4, calculate the error between the predicted output value and the expected value of each sample; STEP5, reversely calculate and correct the network weights and biases; STEP6: if the conditions for the end of network training are met, end the training, otherwise go to STEP4 to continue training; STEP7, analyze and output the results.

[0165] The general processing module 111 in the controller also obtains the coordinate information and orientation information of the current location of the residential building through the positioning unit 700, and after determining the size of the residential building, outputs the size of the residential building and the corresponding coordinate and orientation information through the output unit 132.

[0166] like Fig.15 As shown, after the residential size is identified, the residential size is also marked in the residential edge segment corresponding to the extracted residential edge segment in the rural residential image, such as by using an arrow to mark the corresponding direction segment. Preferably, the acquired coordinates and location information, as well as the residential size, are also stored in a list of the storage module 120, and the list information can be stored in the cloud server 400.

[0167] When the length and width in the size data are greater than the preset safety distance threshold, the coordinate position value of the central point on the roof of the residential building is given; otherwise, an alarm is issued to indicate that the safety distance of the residential building roof is insufficient. During the flight of the drone, image acquisition is performed iteratively and continuously. The rural residential identification controller 100 in the field of view of the drone performs size prediction processing based on the acquired images. When the length and width of the residential building in the field of view are greater than the preset safety distance threshold, a prompt is issued that the residential building has a sufficient safety distance. Preferably, the image is updated and processed according to the flight speed and frame rate so that the residential buildings in the field of view are identified one by one.

[0168] After finding a site with a sufficient safety distance, the safety distance of the site can be verified by collecting residential images from multiple angles and predicting their size. When flying toward the target residential area, the rural scene image in the new field of view is identified and processed based on the rural scene target classification model, and then the rural residential area anchor frame is updated. According to the position of the residential anchor frame in the field of view, flight control is performed to make the residential area approach the center area of ​​the field of view.

[0169] In the heuristic search for boundary lines at the edge of residential buildings, some residential buildings have multi-layered steps, and the corresponding parallel edges will appear in multiple pairs. In this case, the candidate straight line groups may or may exceed 2 groups. At this time, in the candidate straight line groups, the minimum length or width corresponding to the boundary formed by the pair is used as the safety distance; and for height, the maximum height is used as the safety distance.

[0170] Preferably, if the flat ground is taken as the landing point, then in one of the candidate straight line groups, the second pair of edges and the third pair of edges are formed by two relative edge lines of two candidate straight lines on the ground rather than the roof of a residential building with a pair of other straight line segments sandwiched in between, so as to further obtain the geometric features of the flat ground.

[0171] Example 4

[0172] Different from the above embodiments, this embodiment provides another method for obtaining the dimensions of the long side and the wide side of a residential building.

[0173] See also Figure 1 , and combined with Fig. 7EAs shown, in the boundary of the three-dimensional residential building, in addition to the horizontal parallel line segments, there are also vertical side edges, which can also sandwich the long side or wide side of the residential building. Therefore, this embodiment forms the second pair of side lines and the third pair of side lines based on the acquired residential side edges, using two residential side edges sandwiching a pair of other straight line segments as two opposite side lines.

[0174] At this time, preferably, in the input of the neural network, the slope of the boundary line, i.e. the side edge, is replaced by the angle rotated counterclockwise from 0 degrees to the boundary line, so as to prevent errors caused by using the slope when the angle is close to 90 degrees.

[0175] The present invention is applied to identify the size of rural houses in the image space under the field of view of the drone. After the residential area is segmented, the residential edge is extracted and the boundary line is determined, four geometric features, namely the slopes k1 and k2 of a pair of boundary lines corresponding to the length, width and height respectively, and the vertical distances D1 and D2 from the image center point to the boundary lines, are input into the established residential size prediction model based on the neural network. The model can accurately predict the actual size of the rural houses in the three dimensions of length, width and height, thereby providing a decision-making basis for the docking and flight of the drone, and improving the safety and convenience of the drone operation.

[0176] Several embodiments of the present invention are described above, but these embodiments are presented as examples and do not limit the scope of the invention. These embodiments can be implemented in various other ways, and various omissions, substitutions, and changes can be made without departing from the scope of the main purpose of the invention. These embodiments or their modifications are included in the scope or main purpose of the invention, and are also included in the invention described in the claims and their equivalents.

Claims

1. A device for identifying rural residential buildings in the field of vision of a drone, comprising a host unit and a user interface unit, wherein the host unit comprises an input and output module, a processor and a storage module and is configured as follows: Acquire a rural residential image, determine a ROI region containing residential buildings in the rural residential image, extract residential buildings from the ROI region as a residential building target image; obtain a residential building edge image by edge detection on the residential building target image; perform Hough line transformation on the residential building edge image, and obtain a residential building edge image including three pairs of edge lines in the residential building outline edge by heuristic search, wherein: The first pair of side lines of the three pairs of side lines are two opposite side lines of the top and bottom of the residential building with a pair of side edges in between, and the second pair of side lines and the third pair of side lines of the three pairs of side lines are two opposite side lines with a pair of horizontal long sides and horizontal wide sides in between respectively; From the residential boundary image, for any pair of the three pairs of boundary lines, obtain geometric features of the two relative boundary lines relative to the image center, input the geometric features into a trained residential size prediction model based on a neural network, obtain a residential size prediction value, and output the size prediction value.

2. The rural residential identification device under the vision of a drone according to claim 1 is characterized in that: The processor includes a general processing module, an acquisition and preprocessing module, a feature extraction module, an iterative learning module and a residential size prediction module. The acquisition and preprocessing module acquires the rural residential image from the image acquisition unit, and after filtering, identifies the location of the residential image in the image as the ROI region according to the deep learning features by a trained target classification model; In the original rural residential image, the portion outside the ROI area is removed to form a residential target image, and then a binary residential edge image is formed through edge detection; after the binary residential edge image is subjected to Hough line transformation, the straight line segments with the longest lengths in the vertical and horizontal directions are respectively selected as candidate straight line segments according to the polar coordinate space parameter characteristics in the Hough line transformation, and two relative sidelines with a pair of long sides, wide sides and high sides in the middle are respectively determined according to the orientation characteristics of the candidate straight line segments as a pair of sidelines of the length, width and height of the residential building, and together form three pairs of sidelines of the residential building; For each edge line of any pair of edge lines among the three pairs of edge lines, the feature extraction module combines the corresponding binary residential edge image, describes it with the coordinates of the two end points of the straight line, and respectively obtains the slope of the straight line and the perpendicular distance from the center point of the image to the straight line based on the two end points; the slope of the straight line on both sides of the residential building and the perpendicular distance from the center point of the image to the straight line are used as the geometric features of the residential building in the rural residential building image; The iterative learning module trains the neural network of the residential building size prediction model in the residential building size prediction module based on the collected training samples; The general processing module responds to the message, starts identifying residential buildings in the captured image, schedules the identification processing flow, and outputs the size data identified and predicted by the residential building size prediction module.

3. The rural residential identification device under the vision of a drone according to claim 2 is characterized in that: The acquisition and preprocessing module is provided with a target detection unit, in which a target classification model based on the ASFF-YOLOv5s network is established. After the rural scene image is acquired by the bird's-eye view camera in the image acquisition unit, the network is offline trained with an image sample set consisting of the rural scene image with the rural residential area marked, so as to obtain a rural scene target classification model. When running online, the acquisition and preprocessing module uses the rural scene target classification model to perform recognition processing on the rural scene image including residential houses to be tested, i.e., the rural residential image, to obtain an anchor frame of the rural residential area, and uses the area within the anchor frame as the ROI area of ​​the rural residential area. The ASFF-YOLOv5s network adds three layers 24, 25, and 26 between the 23rd and the next layer of the YOLOv5s network. These three layers receive the outputs of the 17th, 20th, and 23th layers as inputs, respectively, and after the ASFF adaptive spatial feature fusion module performs feature fusion of different scales, they replace the original 17th, 20th, and 23th layers in turn and input them into the 24th layer of the original output layer, i.e., the 27th layer of the new network.

4. The rural residential identification device under the vision of a drone according to claim 1 is characterized in that: The residential size prediction model adopts an artificial neural network, and its input layer receives four geometric feature inputs, namely the slopes k1 and k2 of a pair of sidelines corresponding to the length, width and height of the residential building among the three pairs of sidelines and the vertical distances D1 and D2 from the center point of the image to the pair of sidelines, from the general processing module respectively. The output of the output layer is the length, width and height of the residential building corresponding to the pair of sidelines.

5. The rural residential identification device under the vision of a drone according to claim 1, characterized in that: The host unit is also configured to: when performing the Hough line transform on each edge pixel point in the residential edge image, accumulate and obtain the ρ-θ plane matrix corresponding to the image, where ρ represents the vertical segment distance of the straight line from the origin, and θ is the angle of the vertical line to the horizontal axis; in the accumulation, the foreground pixel point (u, v) in the image and a straight line element l i,j (ρ i,j ,θ i,j ) The corresponding judgment method is: Take a point H(ρ i,j ·cosθ i,j , ρ i,j sinθ i,j ), recorded as H(u′, v′), calculate J = |vv′-(uu′)·tanθ i,j |, if |J|≤ε, then the pixel point is determined to be on the straight line and accumulated, where ε is a preset small positive number.

6. The rural residential identification device under the vision of a drone according to claim 1, characterized in that: The host unit is further configured to: when performing the Hough line transform on each edge pixel point in the residential edge image, accumulate and obtain a ρ-θ plane matrix corresponding to the image, where ρ represents the distance of the vertical segment of the straight line from the origin, and θ is the angle of the vertical line to the horizontal axis; Based on the accumulated ρ-θ plane matrix, search for the two or three longest edges in a range close to the vertical direction as the side edges of the residential building; In the range of the non-side edge direction, the plane matrix element values, i.e., the lengths of the straight line segments, are sorted from large to small, and two secondary straight line segments close to the main straight line segment in direction are searched in sequence from the main straight line segment with the largest length, and the main straight line segment and the corresponding two secondary straight line segments form a candidate straight line group; For any straight line segment in the candidate straight line group, calculate its distance from the upper and lower end points of the side edge of the residential building. If the minimum value of the distance corresponds to the distance between the straight line segment and the upper end point of a side edge, then the straight line segment is identified as a roof candidate straight line, otherwise it is identified as a ground candidate straight line.

7. The rural residential identification device under the vision of a drone according to claim 6, characterized in that: The host unit is further configured to: in one of the candidate straight line groups, form the first pair of side lines with two opposite side lines, namely, a residential roof candidate straight line and a ground candidate straight line with a pair of side edges in between; In one of the candidate straight line groups, the second pair of sidelines and the third pair of sidelines are formed by two opposite sidelines of two residential roof candidate straight lines sandwiching a pair of other straight line segments.

8. The rural residential identification device under the vision of a drone according to claim 1, characterized in that: The host unit is further configured to: calculate and extract geometric features for any pair of edge lines among the three pairs of edge lines, Assuming that the coordinates of the two endpoints of a residential boundary line are (x1, y1) and (x2, y2), then its slope is k = y1-y2 / x1-x2; The vertical distance from the center of the image to the edge of the residential building is Where (x0, y0) represents the coordinates of the center of the image, A and B are the coefficients of the general formula of the straight line equation, C is a constant term and: A = y2-y1, B = x1-x2, C = -x1(y2-y1)+y1(x2-x1); According to the calculation formulas of k and d, the slopes k1 and k2 of any pair of edge lines of the residential edge and the vertical distances D1 and D2 from the center point of the image to the two edge lines are calculated respectively, a total of four quantities as the geometric features of the residential buildings in the rural residential image.

9. The rural residential identification device under the vision of a drone according to claim 1, characterized in that: The BP neural network used in the residential size prediction model is a three-layer structure including a hidden layer, wherein: The output of the jth node in the hidden layer is The output of the output layer node is Among them, the f() function is taken as the tansig function, w ij and v j are the connection weights from the input layer to the hidden layer and from the hidden layer to the output layer, respectively. j and b are the hidden layer and output layer thresholds respectively, m=4 is the number of input variables, k is the number of hidden layer nodes, and the gradient descent method is used for network training.

10. Rural residential identification system under drone vision, including: An image acquisition unit for sensing and acquiring images of rural scenes, a user interface unit for parameter input and operation and rural residential information display, a server for data management and information transmission, a handheld terminal for remote operation and rural residential information interaction, and a host unit connected to the user interface unit, the server, and the image acquisition unit, The host unit includes an input / output module, a processor and a storage module, and is configured as follows: Acquire a rural residential image, determine a ROI region containing residential buildings in the rural residential image, extract residential building regions from the ROI region as residential building target images; obtain residential building edge images by edge detection of the residential building target images; perform Hough line transformation on the residential building edge images, and obtain residential building edge images including three pairs of edge lines in the residential building contour edges by heuristic search, wherein a first pair of edge lines of the three pairs of edge lines are two opposite edge lines of the top and bottom of the residential building with a pair of side edges sandwiched in the middle, and a second pair of edge lines and a third pair of edge lines of the three pairs of edge lines are two opposite edge lines of the residential building with a pair of horizontal long sides and horizontal wide sides sandwiched in the middle, respectively; From the residential boundary image, for any pair of the three pairs of boundary lines, obtain geometric features of the two relative boundary lines relative to the image center, input the geometric features into a trained residential size prediction model based on a neural network, obtain a residential size prediction value, and output the size prediction value.

Citation Information

Patent Citations

  • Space measuring and calculating method based on laser and images

    CN110879057A

  • High-resolution optical remote sensing satellite image building height monitoring method based on angular points

    CN113139994A

  • Method for identifying, acquiring and calculating ratio of length unit to pixel value of CAD (Computer Aided Design) drawing

    CN116343253A

  • On-site survey method for photovoltaic power station

    CN116659469A

  • An edge connectivity algorithm that can be implemented in parallel starting from a breakpoint

    CN102270299A