An indoor unmanned aerial vehicle positioning method, device, equipment and storage medium

Abstract

The application discloses an indoor unmanned aerial vehicle positioning method and device, equipment and storage medium, through acquiring source point cloud data and source panoramic image collected by an unmanned aerial vehicle in a flight process, fusing semantic information of the source panoramic image with the source point cloud data to obtain first fusion data; extracting target point cloud data and target panoramic image of a plurality of position points from a preset database, fusing semantic information of the target panoramic image with the target point cloud data to obtain second fusion data of the position points; matching the first fusion data with the second fusion data of the plurality of position points to determine position information of the unmanned aerial vehicle, fusing semantic segmentation data of an indoor complex scene with features of point cloud data, solving the problem that deviation of unmanned aerial vehicle positioning is prone to occurring due to structural complexity of an indoor environment, realizing indoor accurate positioning of the unmanned aerial vehicle, and guaranteeing flight safety of the unmanned aerial vehicle in the indoor environment.

Description

Technical Field

[0001] This invention relates to the field of drone inspection technology, and in particular to an indoor drone positioning method, device, equipment, and storage medium. Background Technology

[0002] Drones are widely used in industrial inspections, enabling them to inspect and monitor the condition of the environment and facilities, and to promptly identify and address problems. However, GPS signals are often blocked or attenuated in underground tunnels, remote factories far from signal base stations, and indoor locations such as underground parking lots, making reliable positioning information unavailable.

[0003] Existing technology involves equipping a robot with both LiDAR and a camera to collect LiDAR point clouds and depth camera point clouds, manually calibrating the LiDAR point cloud that falls on the camera image, converting the depth camera point cloud information into LiDAR lines, fusing the positioning information obtained from the LiDAR and depth camera, registering the point cloud with an indoor 3D model, and estimating the robot's current position.

[0004] However, existing methods rely on manual calibration of point clouds falling on the checkerboard pattern in lidar point cloud frames, which increases the time and complexity of the calibration process, resulting in long processing times and low efficiency. Summary of the Invention

[0005] This invention provides an indoor drone positioning method, apparatus, device, and storage medium to solve the problem that the structural complexity of indoor environments leads to positioning errors in drones, thereby achieving accurate indoor positioning of drones and ensuring the flight safety of drones indoors.

[0006] According to one aspect of the present invention, an indoor unmanned aerial vehicle (UAV) positioning method is provided, comprising:

[0007] The source point cloud data and source panoramic image collected by the UAV during flight are acquired, and the semantic information of the source panoramic image is fused with the source point cloud data to obtain the first fused data.

[0008] Target point cloud data and target panoramic images of multiple location points are extracted from a preset database. The semantic information of the target panoramic image is fused with the target point cloud data to obtain the second fused data of the location points.

[0009] The first fused data is matched with the second fused data from multiple location points to determine the location information of the drone.

[0010] According to another aspect of the present invention, an indoor unmanned aerial vehicle (UAV) positioning device is provided, comprising:

[0011] The first fusion data acquisition module is used to acquire source point cloud data and source panoramic images collected by the UAV during flight, and to fuse the semantic information of the source panoramic image with the source point cloud data to obtain the first fusion data.

[0012] The second fusion data acquisition module is used to extract target point cloud data and target panoramic images of multiple location points from a preset database, and fuse the semantic information of the target panoramic image with the target point cloud data to obtain the second fusion data of the location points.

[0013] The location information determination module is used to match the first fused data with the second fused data of multiple location points to determine the location information of the UAV.

[0014] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0015] At least one processor; and

[0016] A memory that is communicatively connected to at least one processor; wherein,

[0017] The memory stores a computer program that can be executed by at least one processor, such that the at least one processor is able to perform the indoor drone positioning method of any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores computer instructions for causing a processor to execute and implement the indoor unmanned aerial vehicle positioning method of any embodiment of the present invention.

[0019] The technical solution of this invention acquires source point cloud data and source panoramic images collected by the UAV during flight, fuses the semantic information of the source panoramic images with the source point cloud data to obtain first fused data; extracts target point cloud data and target panoramic images of multiple location points from a preset database, fuses the semantic information of the target panoramic images with the target point cloud data to obtain second fused data of the location points; matches the first fused data with the second fused data of multiple location points to determine the UAV's location information. By fusing semantic segmentation data of complex indoor scenes with features of point cloud data, this invention solves the problem that the structural complexity of indoor environments can easily lead to UAV positioning errors, achieving accurate indoor positioning of the UAV and ensuring the flight safety of the UAV indoors.

[0020] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of an indoor drone positioning method provided in Embodiment 1 of the present invention;

[0023] Figure 2 This is a flowchart of an indoor drone positioning method provided in Embodiment 2 of the present invention;

[0024] Figure 3 This is a schematic diagram of the structure of an indoor drone positioning device provided in Embodiment 3 of the present invention;

[0025] Figure 4 This is a schematic diagram of the structure of an electronic device that implements the indoor drone positioning method of this invention. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present invention, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0028] Example 1

[0029] Figure 1This is a flowchart of an indoor drone positioning method provided in Embodiment 1 of the present invention. This embodiment is applicable to indoor drone inspection scenarios. The method can be executed by an indoor drone positioning device, which can be implemented in hardware and / or software and can be configured in electronic devices such as computers or servers. Figure 1 As shown, the method includes:

[0030] S110. Acquire source point cloud data and source panoramic images collected by the UAV during flight, and fuse the semantic information of the source panoramic images with the source point cloud data to obtain the first fused data.

[0031] In this embodiment, source point cloud data It is point cloud data of the environment around the current location point collected by lidar during real-time inspection by the drone, where i = 1, 2, ..., N S N S This refers to the number of points in the source point cloud data. (Source panoramic image) It is a panoramic image of the surrounding environment at the current location point, captured by the onboard camera during real-time inspection by the drone, where j = 1, 2, ..., P S P S =W S ×H S W S and H S These are the width and height of the source panoramic image. Semantic information refers to the semantic category information of each pixel in the panoramic image, which can be obtained through semantic analysis of the source panoramic image. The source point cloud data is fused with the classification information of the pixels in the corresponding source panoramic image, so that the points in the source point cloud data possess semantic information in addition to their original positional information. The first fused data is the source point cloud data X that incorporates the semantic information of the source panoramic image. S .

[0032] Specifically, the point cloud data and panoramic images of the surrounding environment at the current location point are collected using the lidar and airborne camera equipped on the drone, respectively, and used as the source point cloud data X. S Heyuan Panoramic Image M S Identify the semantic information of the source panoramic image and compare it with the points in the source point cloud data. The semantic information in the corresponding source panoramic image is used to assign points in the source point cloud data. Source point cloud data X, which incorporates semantic information from the source panoramic image, S As the first fused data.

[0033] Taking indoor drone inspection of a substation as an example, the drone flies along a flight path and collects surrounding point cloud information in real time using lidar, which serves as the source point cloud data X.S Panoramic photos are captured in real time using an airborne camera, serving as the source panoramic image M. S The pixel-level classification information of the GIS (Gas Insulated Substation) devices in the source panoramic image is assigned to the corresponding points in the source point cloud data. Source point cloud data X, which incorporates semantic information from the source panoramic image, S As the first fused data.

[0034] Optionally, semantic segmentation is performed on the source panoramic image to determine its semantic information; the source point cloud data and the source panoramic image are aligned, and the semantic information of the source panoramic image is fused with the source point cloud data to obtain the first fused data.

[0035] In this embodiment, semantic segmentation is performed on the source panoramic image using deep learning methods to classify each pixel in the source panoramic image. Alignment processing is achieved by using projection transformation to spatially align all points in the point cloud data with their corresponding pixels in the panoramic image; each point in the point cloud data corresponds to a pixel in the panoramic image.

[0036] Specifically, a semantic segmentation model is used to perform semantic segmentation on the source panoramic image to obtain the classification information of each pixel, and the source point cloud data X is then processed. S Projected onto the source panoramic image M S Align the source point cloud data and the source panoramic image to obtain the points in the source point cloud data. The projection point fr on the source panoramic image i S Clustering algorithms, such as the nearest neighbor algorithm, are used to find the points that are closest to the projected point fr. i S The corresponding pixel in the panoramic image is used to assign the classification information of the pixel to the projection point fr. i S Points in the corresponding source point cloud data Source point cloud data X, which incorporates semantic information S As the first fused data.

[0037] For example, the encoder is used to extract features from the source panoramic image to obtain a pixel-level image feature sequence. The pixel feature sequence is processed and segmented using a decoder including a self-attention module. The source point cloud data is projected onto the source panoramic image for alignment. The nearest neighbor algorithm is used to assign semantic information of the source panoramic image to the source point cloud data to obtain the first fused data.

[0038] S120. Extract target point cloud data and target panoramic image from multiple location points from the preset database, and fuse the semantic information of the target panoramic image with the target point cloud data to obtain the second fused data of the location points.

[0039] In this embodiment, the preset database is a database composed of collected information from multiple location points. The collected information includes point cloud data and panoramic images of the surrounding environment of each location point. Target point cloud data. It is the point cloud data of the surrounding environment of a certain location point in the preset database, where l = 1, 2, ..., N T N T N is the number of points in the target point cloud data. T It can be equal to N S It can also be different from N. S This embodiment does not impose any limitations on this. Target panoramic image These are panoramic images of the surrounding environment of the same location points in a pre-defined database, where j = 1, 2, ..., P. T P T =W T ×H T W T and H T It is the width and height of the target panoramic image, W T =W S And H T =H S The second fused data is the target point cloud data X, which incorporates semantic information from the target panoramic image. T .

[0040] Specifically, the drone's position is estimated by extracting target point cloud data and panoramic images of multiple locations from a pre-set database. For the same location point, the target point cloud data and panoramic image are compared with the target point cloud data X. T Points in The corresponding panoramic image of the target M T The semantic information of the pixels in the data is used to assign points to points. X, target point cloud data that incorporates semantic information from the target panoramic image. T As the second fused data.

[0041] Taking indoor drone inspection of a substation as an example, Kalman filtering is used to estimate the drone's flight position. Target point cloud data and panoramic images of multiple locations close to the estimated drone position are extracted from a pre-set database. The pixel-level classification information of the GIS (Gas Insulated Substation) equipment in the panoramic image is then assigned to the corresponding points in the target point cloud data. X, target point cloud data that incorporates semantic information from the target panoramic image.T As the second fused data.

[0042] Optionally, the preset database can be constructed by controlling the drone to fly along a preset flight path and acquiring target point cloud data and target panoramic images when the drone is at a preset location point, and creating a preset database based on the target point cloud data and target panoramic images at the preset location point.

[0043] In this embodiment, the preset flight path is a drone inspection path established based on the point cloud distribution of the indoor environment, which meets the inspection requirements. The drone's flight position and attitude are controlled manually or automatically. For example, flight control commands are sent to the drone via a control handle, or the preset flight path is sent to the drone so that the drone flies based on the preset flight path. Alternatively, the controller can communicate with the drone, loading the preset flight path and sending control commands to the drone after parsing it. Preset position points are key locations within the preset flight path; multiple preset position points exist within a single preset flight path.

[0044] Specifically, the drone can be controlled manually or automatically to fly along a preset flight path. When the drone reaches a preset location, it hovers at that location and uses a lidar to scan the surrounding environment, collecting point cloud data. It also uses an onboard camera to capture panoramic images of the surrounding environment. The system acquires point cloud data and panoramic images from all preset locations and uses the point cloud data and panoramic images from each preset location as the target point cloud data and target panoramic image for that current preset location. The drone then transmits the collected target point cloud data and target panoramic images from the preset locations back to the user. In this embodiment, the electronic device receives the target point cloud data and target panoramic images from the preset locations, or exports them from the drone's memory, and uses this data to create a preset database.

[0045] Taking indoor drone inspection of a substation as an example, the drone is manually or computer-controlled to perform inspection tasks according to a preset flight path. Multiple key locations are selected along the inspection route and designated as preset locations. The drone is manually or automatically controlled to hover at each preset location to collect detailed point cloud data of the surrounding area and to capture panoramic images of the surrounding electrical equipment. This yields point cloud data and panoramic images for each preset location, which are then used as the target point cloud data and target panoramic image for that location. A preset database is created using the target point cloud data and target panoramic images from all preset locations.

[0046] Optionally, the system acquires point cloud data and panoramic images collected by the UAV at multiple error location points centered on a preset location point; determines the average point cloud data of the multiple error location points as the target point cloud data of the preset location point; determines the error location point with the smallest error relative to the preset location point, and determines the panoramic image of the error location point with the smallest error as the target panoramic image of the preset location point.

[0047] In this embodiment, the error location point is the location point of the collected point cloud data and panoramic image set to address the positional errors that may occur during the flight of the UAV.

[0048] Specifically, the drone, centered on each preset location point, adjusts its position every K centimeters within a range of L centimeters in height, forward / backward movement, and left / right movement, obtaining multiple error location points for each preset location point. For each preset location point, point cloud data and panoramic images are acquired from the multiple error location points centered on that preset location point. The average of all point cloud data is calculated, and the average point cloud data is used as the target point cloud data for that preset location point. The panoramic image acquired from the error location point corresponding to the point cloud data with the smallest error compared to the average point cloud data is used as the target panoramic image for that preset location point. Taking a substation indoor drone inspection as an example, the drone's height is controlled to fluctuate by 2 centimeters vertically, 2 centimeters forward / backward, and 2 centimeters left / right from the preset location point, acquiring point cloud data and panoramic images every 1 centimeter, resulting in point cloud data and panoramic images for 125 error location points.

[0049] The technical solution of this embodiment obtains the target point cloud data and target panoramic image of the preset position point by collecting point cloud data and panoramic images from multiple error position points centered on the preset position point, thus avoiding inaccurate target point cloud data and target panoramic images caused by position errors during the flight of the UAV.

[0050] Based on the above embodiments, after obtaining the first fused data and the second fused data, smoothing processing is performed on the first fused data and the second fused data. Smoothing is a point cloud smoothing operation performed on the first fused data and the second fused data. By smoothing the point cloud data in the first fused data or the second fused data, intra-class coordinate smoothing is performed to reduce the offset of the point cloud data.

[0051] Optionally, the semantic information of the source panoramic image or the target panoramic image includes protrusion semantic information, which is used to provide protrusion prompts during the smoothing process. In this embodiment, the protrusion is an object that protrudes from the plane it is on, and can serve as an obstacle during the drone's flight, including but not limited to valves. By setting different semantic information for protrusions, it can serve as a prompt during the smoothing process, avoiding the situation where protrusions are mistakenly smoothed.

[0052] Specifically, during the classification of source and target panoramic images, protruding objects are classified separately, and the point cloud data at the location of the protruding objects are assigned corresponding semantic information. This ensures that the corresponding protruding point clouds in the first and second fused data have different semantic information from the surrounding point clouds. Smoothing algorithms such as simple averaging, weighted averaging, or linear moving average are used to smooth the first and second fused data, and intra-class coordinate smoothing is performed on the first and second fused data based on the semantic information.

[0053] For example, semantic segmentation, including the valve, is performed on the source panoramic image and the target panoramic image. Semantic information is obtained from the source point cloud data and the target point cloud data based on the aligned point clouds. The point cloud aligned with the valve pixels is assigned the semantic information of the valve category, resulting in first fused data and second fused data. A simple averaging method is used to smooth the first fused data and the second fused data. Coordinate smoothing is performed on the point clouds of the same category based on the semantic information. Since the pixels at the valve location have a different category than the pixels in the surrounding locations, they will not be incorrectly smoothed.

[0054] The technical solution of this embodiment can improve the accuracy of point cloud positions in the first fused data and the second fused data, and reduce the safety risks of drones passing through gaps caused by the offset of some points in the point cloud data acquired by the lidar.

[0055] S130. Match the first fused data with the second fused data of multiple location points to determine the location information of the UAV.

[0056] In this embodiment, matching involves matching the first fused data with the second fused data based on features, including planes, edges, corners, and semantics.

[0057] Specifically, a point cloud registration algorithm is used to register the first fused data with the second fused data of multiple location points. The location point corresponding to the second fused data with the highest matching degree is taken as the location point of the drone, and the location point of the drone is taken as the location information of the drone indoors.

[0058] Taking indoor drone inspection of substations as an example, traditional registration methods such as ICP algorithm or artificial intelligence point cloud registration algorithms such as PointNetLK can be used to match the first fused data with the second fused data of multiple location points to obtain the matching degree between the first fused data and the second fused data of different location points. The location point corresponding to the second fused data with the highest matching degree with the first fused data is taken as the real-time location point of the drone, thereby determining the drone's location information indoors.

[0059] The technical solution of this embodiment acquires source point cloud data and source panoramic images collected by the UAV during flight, fuses the semantic information of the source panoramic image with the source point cloud data to obtain first fused data; extracts target point cloud data and target panoramic images of multiple location points from a preset database, fuses the semantic information of the target panoramic image with the target point cloud data to obtain second fused data of the location points; matches the first fused data with the second fused data of multiple location points to determine the UAV's location information. By fusing the semantic segmentation data of complex indoor scenes with the features of point cloud data, the solution addresses the problem of UAV positioning errors caused by the structural complexity of indoor environments, achieving accurate indoor positioning of the UAV and ensuring the flight safety of the UAV indoors.

[0060] Example 2

[0061] Figure 2 This is a flowchart of an indoor drone positioning method provided in Embodiment 2 of the present invention. The technical solution of the present invention is further optimized based on any of the above embodiments. Figure 2 As shown, the method includes:

[0062] S210. Acquire source point cloud data and source panoramic images collected by the UAV during flight, and fuse the semantic information of the source panoramic images with the source point cloud data to obtain the first fused data.

[0063] S220. Extract target point cloud data and target panoramic image from multiple location points from the preset database, and fuse the semantic information of the target panoramic image with the target point cloud data to obtain the second fused data of the location points.

[0064] S230. Obtain the feature encoding of the first fused data and perform feature encoding on the second fused data of multiple location points.

[0065] Specifically, a feature extractor is used to extract features from the first fused data and the second fused data at K locations. The extracted features are then encoded to obtain the feature code f of the first fused data. S The feature encoding of the second fused data at the k-th location point, where k = 1, 2, ..., K.

[0066] For example, the PointNet algorithm can be used to extract features from the first and second fused data through a multilayer perceptron (MLP) and max pooling operations. The features of each point in the point cloud data are encoded to learn global and local feature representations of the point cloud data. By inputting point cloud data containing semantic information, the features extracted by PointNet will include detailed device classification information in the indoor environment, resulting in the feature encoding f of the first fused data. S =PointNet(XS The feature encoding of the second fused data at the k-th location point.

[0067] The technical solution of this embodiment extracts features from the first fused data and the second fused data, eliminating the need to calculate the one-to-one correspondence between point clouds, thus accelerating point cloud matching. Furthermore, the extracted features include semantic information, which can significantly improve the accuracy of point cloud matching and UAV positioning in complex indoor scenes.

[0068] S240. Determine the rigid body transformation matrix corresponding to each location point based on the feature encoding of the first fused data and the feature encoding of the second fused data for each location point.

[0069] In this embodiment, the rigid body transformation matrix T k =[R k |t k ] represents the feature code f of the first fused data. S The feature encoding f of the second fused data at the k-th position point is obtained through rotation and translation. k T The transformation matrix, where R k It is a rotation matrix, t k It is a translation vector.

[0070] Specifically, an iterative algorithm is used to minimize the feature encoding of the transformed first fused data to the feature encoding of the second fused data. The differences are used to converge the rigid body transformation matrix, and the converged rigid body transformation matrix is ​​used as the feature encoding f of the first fused data. S Feature encoding of the second fused data up to the k-th location point The rigid body transformation matrix of .

[0071] For example, the feature encoding f of the first fused data is obtained based on the LK algorithm. S Feature encoding of the second fused data up to the k-th location point The initial rigid body transformation matrix T k,0 =[R k,0 |t k,0 And calculate the residual, that is, use the rigid body transformation matrix T. k,b =[R k,b |t k,b The feature encoding of the first fused data after transformation is converted to the feature encoding of the second fused data. The distance is given by , where b = 0, 1, ..., B, and B is the maximum number of iterations.

[0072] Calculate the derivative of the distance residual with respect to the transformation parameters using the Jacobian matrix:

[0073]

[0074] Update the rigid body transformation matrix:

[0075]

[0076] T k,b+1 =ΔT k,b *T k,b ,

[0077] The process iterates repeatedly, minimizing the residuals to converge the rigid body transformation matrix. The iteration process terminates when the convergence condition is met. The convergence condition can be either the number of iterations equals the maximum number of iterations (b = B), or it can be the residual d from multiple consecutive iterations. k,b Less than the threshold, this embodiment does not impose any restrictions on this.

[0078] The convergent rigid body transformation matrix is ​​used as the feature encoding f of the first fused data. S Feature encoding of the second fused data up to the k-th location point The rigid body transformation matrix is ​​obtained. The above process is repeated to obtain the feature encoding f of the first fused data. S Feature encoding of second fused data at multiple location points The rigid body transformation matrix of .

[0079] S250: Determine the position information of the UAV based on the rigid body transformation matrix of multiple position points.

[0080] Specifically, the rigid body transformation matrices of multiple location points are compared, and the location point corresponding to one of the rigid body transformation matrices is taken as the UAV's position information.

[0081] Optionally, determine the minimum rigid body transformation matrix among the rigid body transformation matrices of multiple location points; determine the UAV's position information based on the location points with the minimum rigid body transformation matrix.

[0082] Specifically, the feature codes of the first fused data after being transformed by the rigid body transformation matrix at different locations are compared with the feature codes of the second fused data at the corresponding locations. The distance is calculated, and the location point corresponding to the minimum distance is taken as the real-time location point of the drone, thereby determining the drone's location information indoors.

[0083] For example, suppose the first fused data X S Second fused data corresponding to two location points and First Fusion Data X S Feature encoding f S Second fusion data to the first location point Feature encoding f1 TThe rigid body transformation matrix is ​​T1, and the first fused data X S Feature encoding f S Second fusion data to the first location point Feature encoding The rigid body transformation matrix is ​​T2. Based on the rigid body transformation matrix T1, the feature encoding f of the transformed first fused data is calculated. S Feature encoding f1 of the second fused data to the first location point T The distance d1; based on the rigid body transformation matrix T2, the feature encoding f of the transformed first fused data is calculated. S Feature encoding of the second fused data to the second location point The distance d2. If d1 > d2, then the first fused data X is considered to be... S Second fusion data with the first location point The best match is determined by using the first location point as the drone's real-time location point, thus establishing the drone's indoor location information; if d1 < d2, then the first fused data X is considered the best match. S Second fusion data with the second location point The best match is to use the second location point as the drone's real-time location point, thereby determining the drone's location information indoors.

[0084] The technical solution of this embodiment acquires source point cloud data and source panoramic images collected by the UAV during flight, and fuses the semantic information of the source panoramic images with the source point cloud data to obtain first fused data; extracts target point cloud data and target panoramic images of multiple location points from a preset database, and fuses the semantic information of the target panoramic images with the target point cloud data to obtain second fused data of the location points; acquires feature codes of the first fused data and the second fused data of multiple location points for feature encoding; determines the rigid body transformation matrix corresponding to each location point based on the feature codes of the first fused data and the feature codes of the second fused data of each location point; determines the UAV's position information based on the rigid body transformation matrices of multiple location points. By fusing the semantic segmentation data of complex indoor scenes with the features of point cloud data, this solution addresses the problem of UAV positioning errors caused by the structural complexity of indoor environments, achieving accurate indoor positioning of the UAV and ensuring the flight safety of the UAV indoors.

[0085] Example 3

[0086] Figure 3 This is a structural schematic diagram of an indoor drone positioning device provided in Embodiment 3 of the present invention. Figure 3 As shown, the device includes:

[0087] The first fusion module 310 is used to acquire source point cloud data and source panoramic images collected by the UAV during flight, and to fuse the semantic information of the source panoramic image with the source point cloud data to obtain the first fused data.

[0088] The second fusion module 320 is used to extract target point cloud data and target panoramic image of multiple location points from a preset database, and fuse the semantic information of the target panoramic image with the target point cloud data to obtain the second fused data of the location points.

[0089] The location information determination module 330 is used to match the first fused data with the second fused data of multiple location points to determine the location information of the UAV.

[0090] The technical solution of this embodiment acquires source point cloud data and source panoramic images collected by the UAV during flight, fuses the semantic information of the source panoramic image with the source point cloud data to obtain first fused data; extracts target point cloud data and target panoramic images of multiple location points from a preset database, fuses the semantic information of the target panoramic image with the target point cloud data to obtain second fused data of the location points; matches the first fused data with the second fused data of multiple location points to determine the UAV's location information. By fusing the semantic segmentation data of complex indoor scenes with the features of point cloud data, the solution addresses the problem of UAV positioning errors caused by the structural complexity of indoor environments, achieving accurate indoor positioning of the UAV and ensuring the flight safety of the UAV indoors.

[0091] Optionally, based on the above embodiments, the first fusion module 310 is specifically used for:

[0092] Semantic segmentation is performed on the source panoramic image to determine its semantic information;

[0093] The source point cloud data and the source panoramic image are aligned, and the semantic information of the source panoramic image is fused with the source point cloud data to obtain the first fused data.

[0094] Based on the above embodiments, optionally, the semantic information of the source panoramic image or the semantic information of the target panoramic image includes protrusion semantic information, which is used to provide protrusion prompts during the smoothing process.

[0095] The first fusion module 310 and the second fusion module 320 also include:

[0096] Smoothing is performed on the first and second fused data.

[0097] Based on the above embodiments, optionally, the location information determination module 330 includes:

[0098] The feature encoding unit is used to acquire the feature encoding of the first fused data and perform feature encoding on the second fused data of multiple location points;

[0099] The rigid body transformation matrix determination unit is used to determine the rigid body transformation matrix corresponding to each position point based on the feature encoding of the first fused data and the feature encoding of the second fused data of each position point.

[0100] The position information determination unit is used to determine the position information of the UAV based on the rigid body transformation matrix of multiple position points.

[0101] Based on the above embodiments, optionally, the location information determination unit is specifically used for:

[0102] Determine the minimum rigid body transformation matrix among multiple rigid body transformation matrices at multiple locations;

[0103] The position information of the UAV is determined based on the position point of the minimum rigid body transformation matrix.

[0104] Based on the above embodiments, optionally, the preset database creation method includes:

[0105] Control the drone to fly according to the preset flight path, and acquire target point cloud data and target panoramic images when the drone is at the preset location point. Create a preset database based on the target point cloud data and target panoramic images at the preset location point.

[0106] Based on the above embodiments, optionally, the preset database creation method also includes:

[0107] Acquire point cloud data and panoramic images collected by the drone at multiple error points centered on a preset location point;

[0108] The average point cloud data of multiple error location points is determined as the target point cloud data of the preset location point;

[0109] Determine the error position point with the smallest error relative to the preset position point, and use the panoramic image of the error position point with the smallest error as the target panoramic image of the preset position point.

[0110] The indoor drone positioning device provided in this embodiment of the invention can execute the indoor drone positioning method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0111] Example 4

[0112] Figure 4This is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0113] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0114] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0115] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as indoor drone positioning methods.

[0116] In some embodiments, the indoor drone positioning method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the indoor drone positioning method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the indoor drone positioning method by any other suitable means (e.g., by means of firmware).

[0117] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0118] Computer programs for implementing the indoor drone positioning method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs can be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0119] Example 5

[0120] Embodiment 5 of the present invention also provides a computer-readable storage medium storing computer instructions for causing a processor to execute an indoor drone positioning method, the method comprising:

[0121] The system acquires source point cloud data and source panoramic images collected by the UAV during flight, fuses the semantic information of the source panoramic images with the source point cloud data to obtain first fused data; extracts target point cloud data and target panoramic images of multiple location points from a preset database, fuses the semantic information of the target panoramic images with the target point cloud data to obtain second fused data of the location points; and matches the first fused data with the second fused data of multiple location points to determine the UAV's location information.

[0122] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0123] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0124] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0125] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0126] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0127] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. An indoor drone positioning method, characterized in that, include: Acquire source point cloud data and source panoramic images collected by the UAV during flight, and fuse the semantic information of the source panoramic image with the source point cloud data to obtain first fused data; Target point cloud data and target panoramic images of multiple location points are extracted from a preset database. The semantic information of the target panoramic image is fused with the target point cloud data to obtain the second fused data of the location points. The first fused data is matched with the second fused data of the multiple location points to determine the location information of the UAV; The step of matching the first fused data with the second fused data of the plurality of location points to determine the location information of the UAV includes: The feature codes of the first fused data and the feature codes of the second fused data at multiple location points are obtained and then encoded. The rigid body transformation matrix corresponding to each location point is determined based on the feature encoding of the first fused data and the feature encoding of the second fused data at each location point. The position information of the UAV is determined based on the rigid body transformation matrix of multiple location points; The determination of the UAV's position information based on the rigid body transformation matrix of multiple location points includes: Determine the minimum rigid body transformation matrix among the rigid body transformation matrices of the plurality of position points; The position information of the UAV is determined based on the position point of the minimum rigid body transformation matrix.

2. The method according to claim 1, characterized in that, The step of fusing the semantic information of the source panoramic image with the source point cloud data to obtain the first fused data includes: Semantic segmentation is performed on the source panoramic image to determine its semantic information; The source point cloud data and the source panoramic image are aligned, and the semantic information of the source panoramic image is fused with the source point cloud data to obtain the first fused data.

3. The method according to claim 1, characterized in that, The semantic information of the source panoramic image or the semantic information of the target panoramic image includes protrusion semantic information, which is used to provide protrusion prompts during the smoothing process. The method further includes: The first fused data and the second fused data are smoothed.

4. The method according to claim 1, characterized in that, The methods for creating the preset database include: The drone is controlled to fly along a preset flight path, and target point cloud data and target panoramic images are collected by the drone at preset locations. The preset database is created based on the target point cloud data and target panoramic images at the preset locations.

5. The method according to claim 4, characterized in that, Acquiring target point cloud data and target panoramic images collected by the UAV at a preset location includes: Acquire point cloud data and panoramic images collected by the UAV at multiple error location points centered on the preset location point; The average point cloud data of multiple error location points is determined as the target point cloud data of the preset location point; The error position point with the smallest error relative to the preset position point is determined, and the panoramic image of the error position point with the smallest error is determined as the target panoramic image of the preset position point.

6. An indoor drone positioning device, characterized in that, include: The first fusion module is used to acquire source point cloud data and source panoramic images collected by the UAV during flight, and to fuse the semantic information of the source panoramic image with the source point cloud data to obtain the first fused data. The second fusion module is used to extract target point cloud data and target panoramic images of multiple location points from a preset database, and fuse the semantic information of the target panoramic image with the target point cloud data to obtain the second fused data of the location points. The location information determination module is used to match the first fused data with the second fused data of the plurality of location points to determine the location information of the UAV; The location information determination module includes: The feature encoding unit is used to acquire the feature encoding of the first fused data and perform feature encoding on the second fused data of multiple location points; The rigid body transformation matrix determination unit is used to determine the rigid body transformation matrix corresponding to each position point based on the feature encoding of the first fused data and the feature encoding of the second fused data of each position point. The position information determination unit is used to determine the position information of the UAV based on the rigid body transformation matrix of multiple position points; The location information determination unit is specifically used for: Determine the minimum rigid body transformation matrix among the rigid body transformation matrices of the plurality of position points; The position information of the UAV is determined based on the position point of the minimum rigid body transformation matrix.

7. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the indoor UAV positioning method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the indoor UAV positioning method according to any one of claims 1-5.

Images