Generation of representation data of three-dimensional map data

Abstract

A technique for generating representation data of map data in a movable object environment is disclosed. A method of generating representation data of three-dimensional map data can include receiving three-dimensional map data acquired by a sensor (1402), generating representation data by projecting the three-dimensional map data onto a two-dimensional plane based on a selected perspective (1404), and associating the representation data with the three-dimensional map data (1406).

Description

[0001] This application claims priority to U.S. Patent Application No. 17 / 068,649, filed on October 12, 2020, the entire contents of which are incorporated herein by reference.

[0002] Copyright Notice

[0003] A portion of the disclosure in this application includes copyrighted material. The copyright holder does not object to anyone copying, by fax, patent documents or patent disclosures appearing in the Patent and Trademark Office's patent archives or records, but reserves all copyright rights. Technical Field

[0004] The disclosed embodiments generally relate to techniques for acquiring map data in a movable object environment, and more specifically, but not limited to, techniques for generating representation data of the acquired map data that will be rendered on a client device. Background Technology

[0005] For example, mobile objects such as unmanned aerial vehicles (UAVs) can be used to perform surveillance, reconnaissance, and exploration missions for various applications. These mobile objects can carry payloads including various sensors, enabling them to acquire sensor data during movement. The acquired sensor data can be rendered on client devices, such as mobile devices or terminals, that communicate with the mobile object via a remote controller, remote server, or other computing device. Summary of the Invention

[0006] A technique for generating representation data of map data acquired in a mobile object environment is disclosed. A method for generating representation data of three-dimensional map data may include: receiving three-dimensional map data acquired by sensors of a payload connected to a drone (UAV); generating representation data by projecting the three-dimensional map data onto a two-dimensional plane based on a selected viewpoint; and associating the representation data with the three-dimensional map data. Attached Figure Description

[0007] Figure 1 Examples of movable objects in a movable object environment according to various embodiments are shown.

[0008] Figure 2 Examples of the architecture of movable objects in a movable object environment according to various embodiments are shown.

[0009] Figure 3 Examples of the architectures of computing devices and client devices according to various embodiments are shown.

[0010] Figure 4 Examples of adapter devices in a movable object environment according to various embodiments are shown.

[0011] Figure 5A and Figure 5B Examples of hierarchical data structures according to various embodiments are shown.

[0012] Figure 6A and Figure 6B Examples of removing outliers from map data according to various embodiments are shown.

[0013] Figure 7 Examples of overlaying information onto map data according to various embodiments are shown.

[0014] Figure 8 Examples of selected viewpoints for map data according to various embodiments are shown.

[0015] Figure 9A and Figure 9B Examples of projecting three-dimensional map data onto a two-dimensional plane according to various embodiments are shown.

[0016] Figure 10A and Figure 10B Examples of selected viewpoints for map data in surveying tasks according to various embodiments are shown.

[0017] Figure 11 Examples of representation data generated from map data in surveying tasks according to various embodiments are shown.

[0018] Figure 12 Examples are shown of inserting representation data and metadata into map data based on different file formats according to various embodiments.

[0019] Figure 13 Examples of combined map data and representation data rendered on a client device's display according to various embodiments are shown.

[0020] Figure 14 A flowchart illustrating a method for generating representation data using map data acquired by sensors in a moving object environment, according to various embodiments, is shown.

[0021] Figure 15 A flowchart is shown of a method for associating representation data with map data to generate combined map data according to various embodiments.

[0022] Figure 16 Examples of supporting movable object interfaces in a software development environment are shown according to various embodiments.

[0023] Figure 17 Examples of movable object interfaces according to various embodiments are shown.

[0024] Figure 18Examples of components of a movable object in a software development kit (SDK) according to various embodiments are shown. Detailed Implementation

[0025] The invention is illustrated in the accompanying drawings by way of example rather than limitation, in which similar reference numerals denote similar elements. It should be noted that in this disclosure, references to "a," "an," or "some" embodiments do not necessarily refer to the same embodiment, but rather to at least one embodiment.

[0026] The following description of the present invention describes the generation of map data acquired from a movable object and the generation of representation data of the acquired map data. The movable object 104 may be an unmanned aerial vehicle, an unmanned vehicle, a handheld device, and / or a robot. For ease of illustration, a drone (UAV) is typically used as an example of a movable object. It will be apparent to those skilled in the art that other types of movable objects can be used without limitation.

[0027] LiDAR (Light Detection and Ranging) sensors can be used to generate highly accurate maps of a target environment. However, LiDAR sensors generate vast amounts of data that are not typically readily accessible to humans. Instead, extensive configurations of LiDAR sensors, along with other sensors such as positioning sensors, and significant post-processing of the acquired data are required to generate maps that can be effectively interpreted and / or used by humans for a variety of applications. For example, a LiDAR sensor may acquire map / scan data associated with a LiDAR scanning sensor, and it requires a high-precision inertial navigation system to generate map data that can be transformed into a useful coordinate system, such as a global coordinate system. Therefore, to obtain usable map data, the complexity of the system, the complexity of the post-processing, and the cost of all required components increase rapidly.

[0028] Furthermore, because map data in conventional systems is not as easily interpreted by humans as traditional image data, operators of surveying systems cannot easily identify any areas in the target environment that have not yet been surveyed or have not been fully surveyed. Instead, operators of these conventional systems must wait for the data to undergo post-processing, which can take hours or days. Afterward, if incomplete map data is discovered, operators must perform additional surveying tasks in the target environment to attempt to complete the map data acquisition. Depending on the operator's skill, this process may be repeated several times before the surveying is complete.

[0029] Furthermore, after a surveying task is completed, the large amount of raw map data collected, or post-processed map data (e.g., after georeferencing, outlier removal, downsampling, etc.), is stored in the form of standard 3D map data, such as LAS or PLY files. These files are typically large and require significant processing time and data communication bandwidth to transmit. Moreover, when multiple surveying tasks are performed, multiple map data files are usually stored in a single task folder or on the same memory card, and users can only distinguish these files based on their generation time / name in chronological order. This makes it difficult for users to identify or select map data files of interest for further processing (e.g., downloading for post-processing, or searching for files to view).

[0030] This embodiment provides a method for generating representation data of raw 3D map data, wherein the representation data is a 2D image showing a preview of the 3D map data. The representation data is generated by projecting the raw 3D map data onto a 2D image plane based on a preset viewpoint or a user-selected viewpoint, such as a top-down viewpoint, a side viewpoint, or a viewpoint with the most feature points. The generated representation data can be stored as a separate image file and associated with the raw 3D map data. At, for example, the midpoint of the 3D map data, or a reserved position appended to the tail data block of the 3D map data, the generated representation data can be written into the raw 3D map data using data insertion techniques to generate new composite map data, which has the same standard 3D map data format (e.g., LAS file or PLY file) as the raw 3D map data. Furthermore, metadata for the representation data can be generated and inserted into the composite map data at, for example, a reserved position in the head data block of the map data.

[0031] In traditional systems, users can only visualize or understand map data by the file timestamps or filenames generated chronologically. This makes it difficult for users to identify specific map data from online systems for downloading necessary data for playback or further processing, or to identify specific map data from offline systems to mark the desired map data stored locally for further processing. By generating representational data of 3D map data, users can intuitively visualize 2D image previews and easily mark map data of interest for online download or offline retrieval, without having to meticulously check every file for all surveying tasks. During a surveying task, users can also check image previews using a visualization application on their client device to determine if specific areas of the target object / environment have not yet been scanned, instead of downloading all the large map data files to check the progress of the surveying task.

[0032] Because users can preview the image to determine whether to download the map data file for record saving, checking for missing areas to be scanned, or further processing, embodiments of this disclosure significantly reduce data communication bandwidth. Since the generated representation data is stored in standard image file formats such as JPEG, GIF, PNG, TIFF, BMP, etc., and its file size (e.g., in megabytes) is much smaller than the file size of standard map data such as LAS or PLY files (e.g., in gigabytes), embodiments of this disclosure also significantly reduce file processing time. In some embodiments, the generated representation data can be stored in different files. In such embodiments, when a user needs to generate a preview of the 3D map data, it can be easily retrieved in a very short processing time due to its smaller size. Furthermore, by maintaining the same file format between the combined point cloud data and the original point cloud data, the combined point cloud data can be recognized by any commercial point cloud visualization application / service / browser provided online or offline by any third-party vendor without further modification.

[0033] Figure 1 Examples of movable objects in a movable object environment 100 according to various embodiments are shown. Figure 1 As shown, a client device 110 (e.g., a smartphone, tablet, remote terminal, personal computer, or other mobile device) in the movable object environment 100 can communicate with the movable object 104 via communication link 106. The movable object 104 can be an unmanned aerial vehicle, an unmanned vehicle, a handheld device, and / or a robot. The client device 110 can be a portable personal computing device, a smartphone, a remote control, a wearable computer, a virtual reality / augmented reality system, and / or a personal computer. Furthermore, the client device 110 may include a remote control 111 and a communication system 120A, which handles communication between the client device 110 and the movable object 104 via communication system 120B. For example, communication between the client device 110 and the movable object 104 (e.g., a drone (UAV)) can include uplink communication and downlink communication. Uplink communication can be used to transmit control signals or commands, and downlink communication can be used to transmit media or video streams, map / scan data acquired by scanning sensors, or other sensor data acquired by other sensors.

[0034] According to various embodiments, communication link 106 may be part of a network based on various wireless technologies such as WiFi, Bluetooth, 3G / 4G, and other radio frequency technologies. Furthermore, communication link 106 may be based on other computer network technologies, such as Internet technology, or any other wired or wireless network technology. In some embodiments, communication link 106 may be a non-network technology, including a direct point-to-point connection such as Universal Serial Bus (USB) or Universal Asynchronous Receiver / Transmitter (UART).

[0035] In various embodiments, a movable object 104 in the movable object environment 100 may include an adapter device 122 and a payload 124, the payload 124 including multiple sensors, such as scanning sensors (e.g., LiDAR sensors), one or more cameras, and / or a sensor set in a single payload unit. In various embodiments, the adapter device 122 includes a port for connecting the payload 124 to the movable object 104, which provides power, data communication, and structural support to the payload 124. Although the movable object 104 is generally described as an aircraft, this is not intended to be limiting, and any suitable type of movable object may be used. Those skilled in the art will understand that any embodiment described herein in the context of an aircraft system can be applied to any suitable movable object (e.g., a UAV). In some cases, the payload 124 may be mounted on the movable object 104 without the need for the adapter device 122.

[0036] According to various embodiments, the movable object 104 may include one or more motion mechanisms 116 (e.g., propulsion mechanisms), a sensing system 118, and a communication system 120B. The motion mechanism 116 may include one or more of a rotor, propeller, blade, engine, motor, wheel, axle, magnet, nozzle, animal, or human. For example, the movable object may have one or more propulsion mechanisms. The motion mechanisms may be of the same type or different types. The motion mechanism 116 may be mounted on the movable object 104 using any suitable method, such as a support element (e.g., a drive shaft) (or vice versa). The motion mechanism 116 may be mounted on any suitable portion of the movable object 104, such as the top, bottom, front, rear, side, or a suitable combination thereof.

[0037] In some embodiments, the mobility mechanism 116 enables the movable object 104 to take off vertically from or land vertically on a surface without requiring any horizontal movement of the movable object 104 (e.g., without traveling along a runway). Optionally, the mobility mechanism 116 may be operable to allow the movable object 104 to hover in a designated position and / or orientation. One or more of the mobility mechanisms 116 may be controlled independently of other mobility mechanisms, for example, through a visualization application 128 executed on a client device 110 or other computing device communicating with the mobility mechanism. Alternatively, the mobility mechanisms 116 may be configured to be controlled simultaneously. For example, the movable object 104 may have multiple horizontally oriented rotors that provide lift and / or thrust to the movable object. Multiple horizontally oriented rotors may be driven to provide the movable object 104 with the ability to take off vertically, land vertically, and hover. In some embodiments, one or more of the horizontally oriented rotors may rotate clockwise while simultaneously rotating counterclockwise. For example, the number of clockwise rotating rotors may be equal to the number of counterclockwise rotating rotors. The rotational rate of each horizontally oriented rotor can be varied independently to control the lift and / or thrust generated by each rotor, thereby adjusting the spatial arrangement, velocity, and / or acceleration of the movable object 104 (e.g., with respect to up to three translational degrees and up to three rotational degrees). As further described herein, a controller, such as flight controller 114, can send movement commands to movement mechanism 116 to control the movement of the movable object 104. These movement commands may be based on and / or derived from instructions received from client device 110 or other computing entities.

[0038] The sensing system 118 may include one or more sensors capable of sensing the spatial arrangement, velocity, and / or acceleration of the movable object 104 (e.g., with respect to translational and rotational degrees). These sensors may include any of the following: a GPS sensor, a real-time kinematic (RTK) sensor, a motion sensor, an inertial sensor, a distance sensor, or an image sensor. The sensing data provided by the sensing system 118 can be used (e.g., with appropriate processing units and / or control modules) to control the spatial arrangement, velocity, and / or orientation of the movable object 104. Alternatively, the sensing system 118 may be used to provide data about the environment surrounding the movable object, such as weather conditions, distances to potential obstacles, locations of geographic features, locations of man-made structures, etc.

[0039] Communication system 120B is capable of communicating with client device 110 via communication link 106 and communication system 120A. Communication link 106 may include various wired and / or wireless technologies as described above. Communication system 120A or 120B may include any number of transmitters, receivers, and / or transceivers suitable for wireless communication. Communication may be unidirectional, such that data can only be transmitted in one direction. For example, unidirectional communication may consist only of transmitting data from movable object 104 to client device 110 and vice versa. Data may be transmitted from one or more transmitters of communication system 120B of movable object 104 to one or more receivers of communication system 120A of client device 110, and vice versa. Alternatively, communication may be bidirectional, such that data can be transmitted in both directions between movable object 104 and client device 110. Two-way communication may include transmitting data from one or more transmitters of the communication system 120B of the movable object 104 to one or more receivers of the communication system 120A of the client device 110, and transmitting data from one or more transmitters of the communication system 120A of the client device 110 to one or more receivers of the communication system 120B of the movable object 104.

[0040] In some embodiments, a visualization application 128 executing on a client device 110 or other computing device communicating with the movable object 104 can provide control data to one or more of the movable object 104, adapter device 122, and payload 124, and receive information from one or more of the movable object 104, adapter device 122, and payload 124 (e.g., position and / or motion information of the movable object, adapter device, or payload; payload-sensed data, such as image data acquired by one or more payload cameras or map / scan data acquired by a payload LiDAR sensor; and data generated from image data acquired by payload cameras or LiDAR data generated from map / scan data acquired by a payload LiDAR sensor).

[0041] In some embodiments, control data may (e.g., through control of the moving mechanism 116) cause changes in the position and / or orientation of the movable object, or (e.g., through control of the adapter device 122) cause movement of the payload relative to the movable object. Control data from the visualization display application 128 may control the payload 124, for example, by controlling the operation of the scanning sensor, camera, or other image acquisition device (e.g., taking still or moving photos, zooming in or out, turning on or off, switching imaging modes, changing image resolution, changing focal length, changing depth of field, changing exposure time, changing angle of view or field of view, adding or deleting waypoints, etc.).

[0042] In some cases, communication from the movable object, adapter device, and / or payload may include information obtained from one or more sensors (e.g., sensing system 118 or payload 124 or other payloads) and / or data generated based on the sensing information (e.g., 3D map data or point cloud obtained from a LiDAR sensor). Communication may include sensing information obtained from one or more different types of sensors (e.g., LiDAR sensor, GPS sensor, RTK sensor, motion sensor, inertial sensor, distance sensor, or image sensor). This information may relate to the position (e.g., location, orientation), movement, or acceleration of the movable object, adapter device, and / or payload. This information from the payload may include data acquired by the payload or the sensing status of the payload.

[0043] In some embodiments, the movable object 104 and / or payload 124 may include one or more processors, such as a DSP, CPU, GPU, field-programmable gate array (FPGA), system-on-a-chip (SoC), application-specific integrated circuit (ASIC), or other processors and / or accelerators. As described above, the payload may include various sensors integrated into a single payload, such as a LiDAR sensor, one or more cameras, an inertial navigation system, etc. The payload may acquire sensor data for providing LiDAR-based mapping for various applications such as construction, surveying, and target inspection.

[0044] In various embodiments, during a mapping mission, the visualization application 128 can obtain sensor data from the payload 124, for example, via connection 106 or another connection between the client device 110 and the movable object 104 and / or the payload 124. In some embodiments, one or more other networks and / or systems may act as intermediaries for the connection. For example, the movable object or payload may be connected to a server in a cloud computing system, satellite, or other communication system, which may provide sensor data to the client device 110.

[0045] In some embodiments, the payload enables the raw map data to be stored locally on a storage medium on the payload or UAV for subsequent post-processing. In various embodiments, once the mapping task is complete, sensor data can be obtained from the payload 124 and provided to the computing device 126 for post-processing. For example, the payload 124 or a movable object 104 communicating with the payload 124 via an adapter device 122 may include a removable medium such as a secure digital card (SD card), or other removable medium such as a flash-based memory device. The removable medium may store the sensor data from the mapping task obtained from the payload 124.

[0046] In some embodiments, the computing device 126 may be located outside the movable object 104, such as at a grounding terminal, remote controller 111, client device 110, or other remote terminal. In these embodiments, the computing device 126 may include a data interface 136, such as a card reader, which can read sensor data stored on a movable medium. In other embodiments, the computing device 126 may be located on the movable object 104, such as at the payload 124 or within the movable object 104. In these embodiments, the computing device 126 may include the data interface 136, which can read sensor data from onboard memory of the payload 124 or the movable object 104, or from the movable medium via an onboard card reader.

[0047] In some embodiments, computing device 126 may directly operate on data stored on a removable medium, or store a local copy, for example, from memory 132 to a disk (not shown) or other storage location accessible to computing device 126, such as an attached storage device, network storage location, etc. Computing device 126 may include one or more processors 134, such as a DSP, CPU, GPU, field-programmable gate array (FPGA), system-on-a-chip (SoC), application-specific integrated circuit (ASIC), or other processors and / or accelerators. As shown, memory 132 may include a post-processing application 130 that processes the raw map data to generate post-processed map / scan data. The post-processed scan data may be visualized by a visualization application 128 on client device 110, for example, rendered on a display or touchscreen of client device 110.

[0048] As described above, sensor data may include scan data obtained from a LiDAR sensor or other sensor providing a high-resolution scan of the target environment, pose data indicating the payload's attitude at the time the scan data was obtained (e.g., from an inertial measurement unit), and positioning data from positioning sensors (e.g., a GPS module, an RTK module, or other positioning sensors), wherein all sensors providing sensor data are incorporated into a single payload 124. In some embodiments, the sensors incorporated into the single payload 124 may be pre-calibrated based on external and internal parameters of the sensors, and synchronized based on a reference clock signal shared among the sensors. The reference clock signal may be generated by a time circuit associated with one of the sensors or by separate time circuits connected to the sensors. In some embodiments, positioning data from positioning sensors may be updated based on calibration data received from positioning sensors of the movable object 104, which may be included in functional module 108, sensing system 118, or a separate module connected to the movable object 104 that provides positioning data for the movable object. In some embodiments, the scan data may be georeferenced using the positioning data and may be used to construct a map of the target environment.

[0049] As described below, georeferenced scan data and payload pose data can be provided to post-processing application 130 for post-processing into a human-readable format. In some embodiments, post-processing application 130 can output an optimized map as a LiDAR Data Exchange File (LAS) or Polygonal Array (PLY) file, which can be used by various tools, such as visualization application 128, to render a map of the target environment and / or use the map data for further processing, planning, etc. Metadata embedded in the LAS or PLY output file facilitates map integration with various third-party tools. In various embodiments, depending on user preference, maps can be output in various file formats, such as .las or .ply files.

[0050] In some embodiments, post-processing application 130 may also generate representation data of the raw or post-processed map data or scan data, which provides a preview of the raw or post-processed map data to be visualized via visualization display application 128 on mobile device 110. For example, post-processing application 130 may generate two-dimensional image data for three-dimensional point cloud data obtained from a LiDAR scanning sensor. The raw three-dimensional point cloud data may be raw point cloud data obtained from payload 124 or point cloud data processed by post-processing application 130. In embodiments, representation data may be image file formats such as JPEG, GIF, PNG, TIFF, BMP files, etc., and map data may be point cloud file formats such as LAS or PLY files. In other embodiments, representation data may be other compressed file formats smaller than standard point cloud file formats. For example, representation data may be generated by selecting a portion of the three-dimensional map data or downsampling the three-dimensional map data for compression. The generated representation data may be stored in different files, or the raw point cloud data may be inserted at the beginning to provide a preview of the raw point cloud data. For example, representation data can be inserted at the midpoint of point cloud data or appended to the end of point cloud data, thus generating combined point cloud data (including the original point cloud data and its representation data).

[0051] The representation data is generated to provide faster, more efficient rendering on the visualization application 128 because the file size of the representation data (e.g., in megabytes) is much smaller than that of the map data (e.g., in gigabytes). The representation data generated based on this disclosure ensures that the preview image contains sufficient map data information. Therefore, it allows users to easily identify specific mapping tasks from multiple mapping tasks through image visualization, rather than having to sort through all mapping tasks by opening point cloud files one by one. The method for generating the representation data will be described in detail below.

[0052] In some embodiments, post-processing application 130 may also generate metadata for the inserted representation data. This metadata may include information such as the start position, end position, or length / size of the inserted representation data. In embodiments, metadata may also be inserted into the point cloud data, for example, at the beginning of the point cloud data. During rendering, visualization application 128 may read only the header of the combined point cloud data to obtain the metadata, which directs or guides visualization application 128 to the beginning of the representation data. Reading only the header significantly reduces the processing time for rendering the preview image compared to reading through the entire combined point cloud data to find the representation data used for rendering. Furthermore, by inserting the representation data and / or its metadata into point cloud data with a standard file format (e.g., LAS or PLY files), the data format of the combined point cloud data will be identical to that of the point cloud data before insertion. Therefore, any point cloud visualization application / service / browser provided online or offline by a third-party vendor can recognize the combined point cloud data.

[0053] The following is for reference Figure 2 Further details about the architecture of the movable object are described.

[0054] Figure 2 Example 200 of the architecture of a movable object in a movable object environment according to various embodiments is shown. Figure 2 As shown, the movable object 104 may include a flight controller 114 that communicates with the payload 124 via an adapter device 122. Furthermore, the flight controller can communicate with various functional modules 108 on the movable object. As further described below, the adapter device 122 can facilitate communication between the flight controller and the payload via a high-bandwidth connection such as Ethernet or Universal Serial Bus (USB). The adapter device 122 can also power the payload 124.

[0055] like Figure 2As shown, the payload may include multiple sensors, including a scanning sensor 202, a monocular camera 204, an RGB camera 206, an inertial navigation system 208, one or more processors 214, and one or more storage devices 216, wherein the inertial navigation system 208 may include an inertial measurement unit 210 and a positioning sensor 212. For example, the scanning sensor 202 may include a LiDAR sensor. The LiDAR sensor can provide high-resolution scanning data of the target environment. Various LiDAR sensors with different characteristics can be incorporated into the payload. For example, the LiDAR sensor may have a field of view of approximately 70 degrees and can implement various scanning modes, such as rocker mode, elliptical mode, petal mode, etc. In some embodiments, higher density point clouds require additional processing time, so lower density LiDAR sensors can be used in the payload. In some embodiments, the payload may implement its components on a single embedded board. The payload may also provide thermal management for its components.

[0056] The payload may also include a grayscale monocular camera 204. The monocular camera 204 may include a mechanical shutter synchronized with an inertial navigation system (INS) 208 such that the payload's attitude at a given moment is associated with the image data when the monocular camera acquires an image. This enables the extraction of visual features (walls, corners, points, etc.) from the image data acquired by the monocular camera 204. For example, the extracted visual features may be associated with a pose-timestamp signature generated from the attitude data produced by the INS. Using the pose-timestamp feature data, visual features can be tracked from frame to frame, thereby enabling the generation of a trajectory for the payload (and thus, the movable object). This enables navigation in areas with limited satellite-based positioning sensor signals, such as indoors or when RTK data is weak or unavailable. In some embodiments, the payload may also include an RGB camera 206. The RGB camera can acquire real-time image data that is streamed to the client device 110 while the movable object is in flight. For example, a user may choose whether to view image data acquired by one or more cameras on the movable object or by the RGB camera on the payload through a user interface of the client device 110. Furthermore, color data can be obtained from image data captured by an RGB camera and overlaid on point cloud data acquired by a scanning sensor. This improves the visualization of the point cloud data, making it more closely resemble the actual objects in the scanned target environment.

[0057] like Figure 2As shown, the payload may also include an inertial navigation system 208. The INS 208 may include an inertial measurement unit 210 and optionally a positioning sensor 212. The IMU 210 provides the payload's attitude, which can be associated with scan data acquired by the scanning sensor and image data acquired by the camera, respectively. The positioning sensor 212 may use global navigation satellite services such as GPS, GLOSNASS, Galileo, BeiDou, etc. In some embodiments, an RTK module 218 on a movable object may be used to enhance the positioning data acquired by the positioning sensor 212, thereby enhancing the positioning data acquired by the INS 208. In some embodiments, RTK information may be received wirelessly from one or more base stations. The antenna of the RTK module 218 and the payload are separated by a fixed distance on the movable object to allow the RTK data acquired by the RTK module 218 to be converted into IMU frames for the payload. Alternatively, the payload 124 itself may not include the positioning sensor 212, but may rely on, for example, the RTK module 218 and the positioning sensor included in the functional module 108. For example, positioning data can be obtained from the RTK module 218 of the movable object 104 and combined with IMU data. The positioning data obtained from the RTK module 218 can be transformed based on the known distance between the RTK antenna and the payload.

[0058] like Figure 2 As shown, the payload may include one or more processors 214. These processors may include embedded processors, including a CPU and a DSP as accelerators. In some embodiments, other processors, such as GPUs, FPGAs, etc., may be used. In some embodiments, processor 214 may georeference scan data using INS data. In some embodiments, the georeferenced scan data is downsampled to a lower resolution before being sent to client device 110 for visualization. In some embodiments, payload communication manager 230 may manage downsampling and other data settings for multiple mobile devices connected to the payload. In some embodiments, different mobile devices may be associated with preference data maintained by payload communication manager 230, wherein the preference data indicates how map data will be prepared and / or sent to the mobile device. For example, the preference data includes: communication protocol settings, channel settings, encryption settings, transmission rate, downsampling settings, etc.

[0059] In some embodiments, processor(s) 214 may also manage the storage of sensor data to one or more storage devices 216. The storage devices(s) may include secure digital (SD) cards or other removable media, solid-state drives (SSDs), eMMC, and / or memory. In some embodiments, the processor may also be used to perform visual inertial odometry (VIO) using image data acquired by monocular camera 204. This may be performed in real time to calculate visual features, which are then stored in a storable format (not necessarily images). In some embodiments, log data may be stored in eMMC, and debug data may be stored in SSD. In some embodiments, processor(s) may include a built-in encoder / decoder for processing image data acquired by an RGB camera.

[0060] Flight controller 114 can send data to and receive data from remote controller via communication system 120B. Flight controller 114 can be connected to various functional modules 108, such as RTK module 218, IMU 220, barometer 222, or magnetometer 224. In some embodiments, communication system 120B can be connected to other computing devices besides flight controller 114, or connected to other computing devices in addition to flight controller 114. In some embodiments, sensor data acquired by one or more functional modules 108 can be transmitted from flight controller 114 to payload 124.

[0061] During a mapping mission, the user can receive data from and issue commands to the UAV using the visualization application 128 on the client device 110. The visualization application 128 can display a visualization of the mapping work performed so far. For example, the post-processing application 130 of the computing device 126 (located on the payload 124 or the movable object 104, or outside the client device 110) can process map data as it is received via the data interface 136. This may include georeferencing scanned and image data using positioning data, and then downsampling the resulting georeferencing map data. When the computing device 126 is on the payload 124 or the movable object 104, the downsampled data can be wirelessly transmitted to the visualization application 128 via the flight controller 114 using the communication system 120B. When the computing device 126 is outside the client device 110, the downsampled data can be directly transmitted to the visualization application 128. The real-time visualization application 128 can then display a visual representation of the downsampled data. This allows users to visualize the amount and / or portion of the target environment that has been scanned, in order to identify areas that still need to be scanned, etc.

[0062] Once the mapping mission is complete and the UAV returns, the map data acquired and processed by the payload can be obtained from the payload or a removable storage medium on the UAV. The removable medium can be provided to computing device 126 and read via data interface 136. For example, if the removable medium is an SD card, data interface 136 can be a card reader. Computing device 126 may include a mapping application 128 that visualizes the map data, and a post-processing application 130 that processes the raw map data into a format suitable for visualization. In some embodiments, post-processing application 130 may be optimized to process data from the payload's scanning sensors. Because the payload comprises a single scanning sensor with fixed characteristics, post-processing application 130 can be optimized for these characteristics, such as scan density. Post-processing may include high-precision post-processing, which is performed on georeferenced map data with a higher density compared to online processing. Furthermore, because real-time processing of INS data is not required, the location data can have higher quality. In some embodiments, high-precision post-processing may also utilize more complex, longer-running optimization algorithms, such as G2O.

[0063] In some embodiments, post-processing may include receiving georeferenced point cloud data and payload pose data, and constructing multiple local maps. In some embodiments, the local maps may be constructed using an Iterative Closest Matching (ICP) module or other modules implementing a matching algorithm. In various embodiments, the ICP module may operate directly on the point cloud data, rather than first extracting features from the scan and using those features to match the scan and construct the local map, thereby improving accuracy and reducing processing time. The local map can then be analyzed to identify corresponding points. Corresponding points include spatial points that have been scanned multiple times from multiple poses. Corresponding points can be used to construct a pose map. In some embodiments, the ICP module may use the ICP algorithm to identify corresponding points in the local map. Embodiments of this disclosure directly determine correspondences using ICP without calculating artificially created features (e.g., PFH, FPFH, 3D SIFT, etc.), rather than using the method employed by many point cloud matching techniques that calculates feature points (e.g., Point Feature Histogram (PFH), Fast Point Feature Histogram (FPFH), 3D Scale Invariant Feature Transform (SIFT) feature points, or other feature extraction techniques) and then estimates correspondences. This also avoids introducing potential errors during feature extraction. The pose graph can then be optimized using graph optimization techniques to create optimized point cloud data. The optimized point cloud data can then be viewed on post-processing application 130 or visualization application 128.

[0064] In some embodiments, post-processing application 130 may also generate representation data of the raw or post-processed map data or scan data, which provides a preview of the raw or post-processed map data to be visualized via visualization display application 128 on mobile device 110. For example, post-processing application 130 may generate two-dimensional image data for three-dimensional point cloud data obtained from a LiDAR scanning sensor. The raw three-dimensional point cloud data may be raw point cloud data obtained through data interface 136 or point cloud data processed by post-processing application 130. In embodiments, representation data may be image file formats such as JPEG, GIF, PNG, TIFF, BMP files, etc., and map data may be point cloud file formats such as LAS or PLY files. In other embodiments, representation data may be other compressed file formats smaller than standard point cloud file formats. For example, representation data may be generated by selecting a portion of the three-dimensional map data or downsampling the three-dimensional map data for compression. The generated representation data may be stored in different files, or raw point cloud data may be inserted at the beginning to provide a preview of the raw point cloud data. For example, representation data can be inserted at the midpoint of point cloud data or appended to the end of point cloud data, thus generating combined point cloud data (including the original point cloud data and its representation data).

[0065] The representation data is generated to provide faster, more efficient rendering on the visualization application 128 because the file size of the representation data (e.g., in megabytes) is much smaller than that of the map data (e.g., in gigabytes). The representation data generated based on this disclosure ensures that the preview image contains sufficient map data information. Therefore, it allows users to easily identify specific mapping tasks from multiple mapping tasks through image visualization, rather than having to sort through all mapping tasks by opening point cloud files one by one. The method for generating the representation data will be described in detail below.

[0066] In some embodiments, post-processing application 130 may also generate metadata for the inserted representation data. This metadata may include information such as the start position, end position, or length / size of the inserted representation data. In embodiments, the metadata may also be inserted into the point cloud data, for example, into the header of the point cloud data. During rendering, visualization application 128 may read only the header of the combined point cloud data to obtain the metadata, which directs or guides visualization application 128 to the start position of the representation data. Reading only the header significantly reduces the processing time for rendering the preview image compared to reading through the entire combined point cloud data to find the representation data used for rendering. Furthermore, by inserting the representation data and / or its metadata into point cloud data with a standard file format (e.g., LAS or PLY files), the data format of the combined point cloud data will be identical to that of the point cloud data before insertion. Therefore, any point cloud visualization application / service / browser provided online or offline by a third-party vendor can recognize the combined point cloud data.

[0067] Figure 3 Example 300 illustrates the architecture of a computing device 126 and a mobile device 110 in a movable object environment according to various embodiments. Figure 3 As shown, computing device 126 may execute on one or more processors 302. As described above, in some embodiments, computing device 126 may be located on movable object 104, such as at payload 124 or within movable object 104. In other embodiments, computing device 126 may be located outside movable object 104, such as at a grounding terminal, remote controller 111, client device 110, or other remote terminal. One or more processors 302 may include a CPU, GPU, FPGA, SoC, or other processor, and may be part of a parallel computing architecture implemented by computing device 126. Computing device 126 may include data interface 303, post-processing application 308, and map generator 316.

[0068] Data interface 303 may include scanning sensor interface 304 and positioning sensor interface 306. Data interface 303 may include hardware and / or software interfaces. Scanning sensor interface 304 may receive data from scanning sensors (e.g., LiDAR or other scanning sensors), and positioning sensor interface 306 may receive data from positioning sensors (e.g., GPS sensors, RTK sensors, IMU sensors, and / or other positioning sensors or combinations thereof). In various embodiments, the scanning sensor may generate map data in a point cloud format (e.g., LAS or PLY files). The point cloud of the map data may be a three-dimensional representation of the target environment. In some embodiments, the point cloud of the map data may be converted to a matrix representation. Positioning data may include GPS coordinates of a movable object, and in some embodiments, may include roll, pitch, and yaw values ​​associated with the movable object corresponding to each GPS coordinate. These roll, pitch, and yaw values ​​may be obtained from positioning sensors, such as inertial measurement units (IMUs) or other sensors. As described above, positioning data may be obtained from an RTK module that corrects the GPS coordinates based on correction signals received from a base station. In some embodiments, the RTK module can generate a variance value associated with each output coordinate. The variance value can represent the accuracy of the corresponding positioning data. For example, if the movable object is moving violently, the variance value will increase, indicating that the acquired positioning data is less accurate. The variance value can also vary due to atmospheric conditions, causing the accuracy of the movable object measurement to differ depending on the specific conditions under which the data was acquired.

[0069] The positioning sensor and the scanning sensor can be synchronized using a time synchronization mechanism. For example, the positioning sensor and the scanning sensor can share a clock circuit. In some embodiments, the positioning sensor may include a clock circuit and output a clock signal to the scanning sensor. In some embodiments, a separate clock circuit may output a clock signal to both the scanning sensor and the positioning sensor. In this case, a shared clock signal can be used to timestamp the positioning data and map data.

[0070] In some embodiments, the positioning sensor and the scanning sensor may output data with different delays. For example, the positioning sensor and the scanning sensor may not start generating data simultaneously. This allows for buffering of the positioning data and / or map data, creating a delay. In some embodiments, the buffer size may be selected based on the delay between the outputs of the individual sensors. In some embodiments, the post-processing application 308 may receive data from the positioning sensor and the scanning sensor and generate synchronization data using timestamps shared by the sensor data relative to a shared clock signal. This allows the positioning data and map data to be synchronized for further processing.

[0071] Furthermore, the frequencies of the data obtained from the various sensors can differ. For example, a scanning sensor can generate data in the range of hundreds of kHz, while a positioning sensor can generate data in the range of hundreds of Hz. Therefore, to ensure that each point in the map data has corresponding positioning data, the upsampling module 310 can interpolate lower-frequency data to match higher-frequency data. For example, assuming the positioning data is generated by the positioning sensor at a frequency of 100 Hz, and the map data is generated by the scanning sensor (e.g., a LiDAR sensor) at a frequency of 100 kHz, the positioning data can be upsampled from 100 Hz to 100 kHz. Various upsampling techniques can be used to upsample the positioning data. For example, a linear fitting algorithm such as least squares can be used. In some embodiments, a nonlinear fitting algorithm can be used to upsample the positioning data. Additionally, the roll, pitch, and yaw values ​​of the positioning data can be interpolated to match the frequency of the map data. In some embodiments, the roll, pitch, and yaw values ​​can be spherical linear interpolation (SLERP) to match the number of points in the map data. Similarly, timestamps can be interpolated to match the interpolated positioning data.

[0072] Once the location data has been upsampled by the upsampling module 310 and synchronized with the map data, the georeference module 312 can convert the matrix representation of the map data from its source reference system (or reference coordinate system) (e.g., scanner reference system or scanner reference coordinate system) to the desired reference system (or desired reference coordinate system). For example, the location data can be converted from a scanner reference system to a North-Eastern (NED) reference system (or NED coordinate system). The reference system to which the location data is converted can vary depending on the application of the generated map. For example, if the map is used for surveying, it can be converted to an NED reference system. As another example, if the map is used for rendering motion such as in flight simulations, it can be converted to a FlightGear coordinate system. Other applications of the map can influence the conversion of the location data to different reference systems or different reference coordinate systems.

[0073] Each point in the point cloud of map data is associated with a position in a scanner reference frame determined relative to the scanning sensor. This position in the scanner reference frame can then be transformed to an output reference frame in the world coordinate system, such as the GPS coordinate system, using positioning data of the movable object generated by the positioning sensor. For example, the position of the scanning sensor in the world coordinate system is known based on the positioning data. In some embodiments, the positioning sensor and the scanning module may be offset (e.g., due to different positions on the movable object). In these embodiments, another correction factor in this offset can be used to transform from the scanner reference frame to the output reference frame (e.g., each measured position in the positioning data can be corrected using the offset between the positioning sensor and the scanning sensor). For each point in the point cloud of map data, a timestamp can be used to identify the corresponding positioning data. This point can then be transformed to a new reference frame. In some embodiments, the scanner reference frame can be transformed to a horizontal reference frame using interpolated roll, pitch, and yaw values ​​from the positioning data. Once the map data has been transformed to a horizontal reference frame, it can be further transformed to a Cartesian coordinate system or another output reference frame. The result of the transformation of each point is a georeferenced point cloud, where each point is now referenced to the world coordinate system. In some embodiments, the georeferenced point cloud can be provided to the map generator 316 before outlier removal to remove outlier data from the georeferenced point cloud.

[0074] After a georeferenced point cloud has been generated, outlier removal module 314 can remove outlier data from the georeferenced point cloud. In some embodiments, the georeferenced point cloud can be downsampled to reduce the number of outliers in the data. This downsampling can be done using voxels. In some embodiments, points in each voxel can be averaged, and each voxel can output one or more average points. Thus, outliers are removed from the dataset during the process of averaging points in each voxel. In various embodiments, the resolution of the voxels (e.g., the size of each voxel) can be arbitrarily defined. This allows for the generation of sparse and dense downsampled point clouds. This resolution can be determined by the user or map manager based on, for example, available computing resources, user preferences, default values, or other application-specific information. For example, a lower resolution (e.g., larger voxels) can be used to generate a sparse downsampled point cloud for visualization on a client device or mobile device. Alternatively or additionally, outliers can be removed statistically. For example, the distance from each point to its nearest neighbor can be determined and statistically analyzed. If the distance from a point to its nearest neighbor is greater than a threshold (e.g., the standard deviation of the nearest neighbor distance in the point cloud), that point can be removed from the point cloud. In some embodiments, the outlier removal technique can be selected by the user or automatically by the post-processing application. In some embodiments, outlier removal can be disabled.

[0075] As mentioned above, point cloud data can be a three-dimensional representation of the target environment. This three-dimensional representation can be divided into voxels (e.g., three-dimensional pixels).

[0076] After statistically removing outliers, the resulting point cloud data can be provided to map generator 316. In some embodiments, map generator 316 may include dense map generator 318 and / or sparse map generator 320. In these embodiments, dense map generator 318 can generate a high-density map from point cloud data received before outlier removal, while sparse map generator 320 can generate a low-density map from sparse downsampled point cloud data received after outlier removal. In other embodiments, dense map generator 318 and sparse map generator 320 can generate high-density and low-density maps, respectively, from point clouds received after outlier removal. In these embodiments, each map generator can use the same process to generate the output map, but the voxel size can be changed to generate a high-density or low-density map. In some embodiments, client device 110 or mobile device can use a low-density map to provide a visualization of the map data. High-density maps can be output as LIDAR Data Exchange Files (LAS) or other file types (such as PLY files) for use with various mapping, planning, analysis or other tools, or rendered on mobile devices 110 via visualization application 128.

[0077] Map generator 316 can use point cloud data to perform probability estimation of the location of points in the map. For example, the map generator can use a 3D map library such as OctoMap to generate an output map. The map generator can divide the point cloud data into voxels. For each voxel, the map generator can determine the number of points in the voxel and, based on the number of points and the variance associated with each point, determine the probability that a point is located in that voxel. This probability can be compared to an occupancy threshold; if the probability is greater than the occupancy threshold, then the point can be represented as being located in that voxel in the output map. In some embodiments, the probability that a given voxel is occupied can be expressed as:

[0078]

[0079] The probability that node n is occupied is P(n|z). 1:t ) represents the current measured value z1, the prior probability P(n), and the previous estimated value P(n|z1). 1:t-1 The function of ). Furthermore, P(n|z t ) represents a given measured value z t The probability that voxel n is occupied. This probability can be enhanced to include the variance of each point as measured by the positioning sensor, as shown in the following formula:

[0080]

[0081]

[0082] In the above formula, P(n) represents the total probability that voxel n is occupied. The use of 1 / 2 in the formula is implementation-specific, so that the probability maps to the range of 1 / 2-1. This range will vary depending on the specific implementation used. In the above formula, the total probability is the product of the probabilities calculated over the x, y, and z dimensions. The probability in each dimension can be based on the mean μ of the points in that dimension and the variance σ of the measurements in a given dimension. 2 Let x, y, and z correspond to the coordinates of a given point. A large number of points within a given voxel that are close to the average point increases the probability, while a more dispersed set of points within the voxel decreases the probability. Similarly, a larger variance associated with the data (e.g., indicating that lower-precision location data has been collected) decreases the probability, while a lower variance increases the probability. P(n,μ,σ) 2 ) represents the Gaussian distribution of a voxel with respect to the mean and variance of points in that voxel.

[0083] If the total probability of a voxel being occupied is greater than an occupancy threshold, a point can be added to that voxel. In some embodiments, the average coordinates of all points in the voxel can be used as the point's location within that voxel. This improves the accuracy of the resulting map compared to alternative methods, such as using the center point of an occupied voxel as the point, which can lead to skewed results depending on the voxel's resolution. In various embodiments, the occupancy threshold can be set based on the amount of available processing resources and / or the acceptable amount of noise in the data for a given application. For example, the occupancy threshold can be set to a default value of 70%. Higher thresholds can also be set to reduce noise. Furthermore, the occupancy threshold can be set based on the quality of the acquired data. For example, high-quality (e.g., low variance) data acquired under a set of conditions may have a lower occupancy threshold, while lower-quality data may require a higher occupancy threshold.

[0084] Subsequently, the resulting map data, with one point in each occupied voxel, can be output as a LAS file or other file format (such as a PLY file). In some embodiments, georeferenced point cloud data can be output without additional processing (e.g., outlier removal). In some embodiments, each point in the point cloud data can also be associated with an intensity value. This intensity value can represent features of the scanned object, such as its height above a reference plane, material composition, etc. The intensity value of each point in the output map can be the average of the intensity values ​​measured for each point in the map data acquired by a scanning sensor (e.g., a LiDAR sensor).

[0085] In an embodiment, post-processing application 308 can generate two-dimensional representation data of high-density or (downsampled after outlier removal) low-density 3D map data. This two-dimensional representation data provides a preview image of the 3D map data. In some embodiments, representation data is generated by projecting the 3D map data onto a two-dimensional projection plane based on a viewpoint of the 3D map data. The viewpoint used for projection can be a top-down view, a side view, or a viewpoint containing the most feature points that captures the most map features. After generating the representation data, post-processing application 308 can also insert the representation data into the original 3D map data to obtain combined map data without changing the standard file format of the 3D map data. For example, the representation data can be appended to the end or middle of the original map data. Data insertion can be performed based on a specific file format of the 3D map data (e.g., LAS or PLY files).

[0086] The post-processing application 308 can also generate metadata for the inserted representation data. The metadata provides additional information about the inserted representation data added to the 3D map data, such as the start and end positions and / or length or size of the inserted representation data. After generating the metadata for the representation data, it can also be inserted into the composite map data, for example, by inserting it into the header of the original 3D map data. In an embodiment, the composite map data, including the inserted representation data and the inserted metadata, can be communicated with the mobile device 110 for display via the visualization application 128. The representation data provides a small-sized preview image (typically in megabytes) of the large-size 3D map data (typically in gigabytes), which can be easily viewed by the user of the mobile device 110. The metadata can be used as a pointer to the exact location of the representation data within the composite map data to be visualized via the visualization application 128 on the mobile device 110.

[0087] Figure 4 Examples of adapter devices in a movable object environment according to various embodiments are shown. Figure 4 As shown, adapter device 122 enables payload 124 to be connected to movable object 104. In some embodiments, adapter device 122 is a payload software development kit (SDK) adapter board, adapter ring, etc. Payload 124 can be connected to adapter device 122, and adapter device can be connected to the body of movable object 104. In some embodiments, adapter device may include a quick-release connector to which payload can be attached / detached.

[0088] When the payload 124 is connected to the movable object 104 via the adapter device 122, the payload 124 can also be controlled by the client device 110 via the remote controller 111. Figure 4As shown, the remote controller 111 and / or the visualization display application 128 can send control commands via a command channel between the communication system of the movable object 104 and the remote controller. These control commands can be transmitted to control the movable object 104 and / or the payload 124. For example, the control commands can be used to control the attitude of the payload to selectively view real-time data (e.g., real-time low-density map data, image data, etc.) acquired by the payload on the mobile device.

[0089] like Figure 4 As shown, after the communication system of the movable object 104 receives the control command, it sends the control command to the adapter device 122. The communication protocol between the adapter device and the communication system of the movable object can be called an internal protocol, and the communication protocol between the adapter device and the payload 124 can be called an external protocol. In this embodiment, the internal protocol between the adapter device 122 and the communication system of the movable object 104 is used as an internal communication protocol, and the external protocol between the adapter device 122 and the payload 124 is used as an external communication protocol. After the communication system of the movable object receives the control command, it uses the internal communication protocol to send the control command to the adapter device through the command channel between the communication system and the adapter device.

[0090] When the adapter device receives control commands sent by the movable object using an internal communication protocol, the internal protocol between the adapter device and the movable object's communication system is converted into an external protocol between the adapter device and the payload 124. In some embodiments, the adapter device can convert the internal protocol into an external protocol by adding a header conforming to the external protocol to the outer layer of the internal protocol message, thereby converting the internal protocol message into an external protocol message.

[0091] like Figure 4 As shown, the communication interface between the adapter device and the payload 124 may include a Controller Area Network (CAN) interface or a Universal Asynchronous Receiver / Transmitter (UART) interface. After the adapter device translates the internal protocol between the adapter device and the communication system of the movable object into an external protocol between the adapter device and the payload 124, control commands are sent to the payload 124 via the CAN interface or UART interface using the external protocol.

[0092] As described above, payload 124 can acquire sensor data from multiple sensors incorporated into the payload, such as LiDAR sensors, one or more cameras, INS, etc. Payload 124 can transmit sensor data to the adapter device via a network port between payload 124 and the adapter device. Alternatively, payload 124 can also transmit sensor data via a CAN interface or UART interface between payload 124 and the adapter device. Optionally, payload 124 can use an external communication protocol to transmit sensor data to the adapter device via a network port, CAN interface, or UART interface.

[0093] After the adapter device receives sensor data from the payload 124, it converts the external protocol between itself and the payload 124 into an internal protocol between the communication system of the movable object 104 and the adapter device. In some embodiments, the adapter device uses the internal protocol to transmit the sensor data to the communication system of the movable object via a data channel between the adapter device and the movable object. Further, the communication system transmits the sensor data to the remote controller 111 via a data channel between the movable object and the remote controller 111, and the remote controller 111 then forwards the sensor data to the client device 110.

[0094] After the adapter device receives the sensor data sent by the payload 124, it can encrypt the sensor data to obtain encrypted data. Further, the adapter device uses an internal protocol to send the encrypted data to the communication system of the movable object via a data channel between the adapter device and the movable object. The communication system then sends the encrypted data to the remote controller 111 via a data channel between the movable object and the remote controller 111. The remote controller 111 then forwards the encrypted data to the client device 110.

[0095] In some embodiments, the payload 124 can be mounted on a movable object via the adapter device 122. When the adapter device 122 receives a control command sent by the movable object 104 for controlling the payload 124, it converts the internal protocol between the movable object and the adapter device into an external protocol between the adapter device 122 and the payload 124, and sends the control command to the payload 124 using the external protocol. This allows third-party devices manufactured by third-party manufacturers to communicate normally with the movable object via the external protocol, thereby enabling the movable object to support third-party devices and expanding the application range of the movable object.

[0096] In some embodiments, to facilitate communication with the payload, the adapter device 122 sends a handshake instruction to the payload 124. The handshake instruction is used to detect whether the adapter device 122 and the payload 124 are in a normal communication connection. In some embodiments, the adapter device 122 may also send the handshake instruction to the payload 124 periodically or at any time. If the payload 124 does not respond, or if the response message from the payload 124 is incorrect, the adapter device 122 may disconnect the communication connection with the payload 124, or the adapter device 122 may restrict the functions available to the payload.

[0097] The adapter device 122 may also include a power interface for supplying power to the payload 124. For example... Figure 4 As shown, the movable object 104 can supply power to the adapter device 122. Furthermore, the adapter device 122 can supply power to the payload 124. The adapter device may include a power interface through which the adapter device 122 supplies power to the payload 124. In various embodiments, the communication interface between the movable object 104 and the adapter device 122 may include a Universal Serial Bus (USB) interface.

[0098] like Figure 4 As shown, a USB interface can be used to establish a data channel between the communication system of the adapter device 122 and the movable object 104. In some embodiments, the adapter device 122 can convert the USB interface into a network port, such as an Ethernet port. The payload 124 can transmit data with the adapter device 122 through this network port, so that the payload 124 can easily communicate with the adapter device 122 via the transmission control protocol without a USB driver.

[0099] In some embodiments, the interfaces output by the movable object 104 include a CAN port, a USB port, and a 12V 4A power port. The CAN port, USB port, and 12V 4A power port are respectively connected to the adapter device 122. The CAN port, USB port, and 12V 4A power port undergo protocol conversion through the adapter device 122, and can generate a pair of external interfaces.

[0100] Figure 5A and Figure 5B Examples of hierarchical data structures according to various embodiments are shown. As described above, such as Figure 5A As shown, the data representing a 500-dimensional environment can be divided into multiple voxels. For example... Figure 5AAs shown, the target environment can be divided into eight voxels, each voxel is further divided into eight sub-voxels, and each sub-voxel is further divided into eight smaller sub-voxels. Each voxel can represent a different volumetric part of the 3D environment. Voxels can be subdivided until the smallest voxel size is reached. The resulting 3D environment can be represented as a hierarchical data structure 502, where the root of the data structure represents the entire 3D environment, and each child node represents a different voxel at a different level within the 3D environment.

[0101] Figure 6A and Figure 6B Examples of removing outliers from map data according to various embodiments are shown. Figure 6A As shown, when scanning a target object, it can be represented as multiple points clustered on different parts of the object, including surfaces (e.g., surface 601), edges (e.g., edge 603), and other parts of the target object in the target environment. For ease of description, these surfaces, edges, etc., are shown as solid. Additional outliers exist in the various regions 600A–600F of the data. This is most noticeable in the blank areas, such as… Figure 6A As shown. These points are scattered compared to the denser clusters of points on the surface and edges of the target object. Outlier removal can be used to eliminate or reduce the number of these points in the data. As mentioned above, georeferenced point cloud data can be downsampled to reduce the number of outliers in the data. Alternatively, outliers can also be removed statistically. For example, the distance from each point to its nearest neighbor can be determined and statistically analyzed. If the distance from a point to its nearest neighbor is greater than a threshold (e.g., the standard deviation of the nearest neighbor distance in the point cloud), that point can be removed from the point cloud. Figure 6B As shown, the area 602A to 602F of the point cloud data shrinks as outliers are removed, providing a clearer 3D map.

[0102] Figure 7 Example 700 of overlaying data values ​​into map data according to various embodiments is shown. For example... Figure 7 As shown, overlay information 702 can be obtained from an RGB camera or other sensor incorporated into the payload. For example, in some embodiments, the overlay data may include color data, which may include pixel values ​​in various color schemes (e.g., 16-bit, 32-bit, etc.). While the scanning sensor acquires point cloud data, color data can be extracted from one or more images acquired by the RGB camera, and these color values ​​can be overlaid on a visualization of the point cloud data. Although in Figure 7The overlay data is depicted as grayscale, but depending on the color values ​​of the image data acquired by the RGB camera, the color data can include a variety of color values. In some embodiments, the overlay data can include the height above a reference plane. For example, a color value can be assigned to a point based on its height above a reference plane. The height value can be a relative height value relative to a reference plane, or an absolute height value (e.g., relative to sea level). The reference plane can correspond to the ground, a floor, or any plane selected by the user. These values ​​can change monochromaticly with changes in height, or they can change color with changes in height. In some embodiments, the overlay data can represent intensity values. Intensity values ​​can correspond to the return intensity of the laser beam received by the LiDAR sensor. Intensity values ​​can indicate the material composition or properties of objects in the target environment. For example, based on the reflectivity of a material, the properties of the material (e.g., the type of material, such as metal, wood, concrete, etc.) can be inferred, and the overlay information can indicate these features by assigning different color values ​​to different features. Additionally or alternatively, in some embodiments, point cloud data can be overlaid on a map of the scanned target area. For example, point cloud data can be overlaid on a two-dimensional or three-dimensional map provided by a mapping service.

[0103] As described above, map data can be generated by using a movable object (e.g., a UAV) to perform mapping tasks, acquired by a scanning sensor connected to the movable object. For example, point cloud data can be acquired and generated using a LiDAR scanning sensor on a payload connected to a UAV. The generated point cloud data is a three-dimensional representation of the target object or target environment. As described above, the raw three-dimensional point cloud data or the post-processed point cloud data processed by the post-processing application 308 and generated by the map generator 316 can have standard three-dimensional map data file formats, such as LiDAR Data Exchange File (LAS) or Polygon (PLY) files, which can be used by various tools such as the visualization application 128 to render a map of the target object or target environment and / or use the three-dimensional map data for further processing, planning, playback, etc.

[0104] This invention also discloses techniques for generating representation data of 3D map data, such as generating a 2D image preview of 3D point cloud data. In embodiments, the representation data may be image file formats such as JPEG, GIF, PNG, TIFF, BMP, etc., while the original map data or post-processed map data may be point cloud file formats such as LAS or PLY. In other embodiments, the representation data may be other compressed file formats smaller than the standard point cloud file format. For example, representation data is generated by selecting a portion of the 3D map data or downsampling the 3D map data for compression. Taking the generation of representation data by projection as an example, a 2D image preview is generated by projecting the 3D point cloud data onto a 2D plane using a selective perspective based on the 3D point cloud data. Perspective projection or orthographic projection can be used to project the 3D point cloud data onto the 2D plane. The perspective used to project the 3D point cloud data can be selected as a top-down or side-view perspective of the target object or target environment. The perspective can also be selected to include the most feature points to obtain the most features of the target object or target environment, thereby most helpful in providing a preview of the object or environment. In this embodiment, the two-dimensional plane can be a plane perpendicular to the central axis of the selected viewpoint, or any plane intersecting the central axis of the selected viewpoint at a certain angle. In this embodiment, the three-dimensional map data may not be projected onto the two-dimensional plane, but may be projected onto a two-dimensional curved surface or other two-dimensional surfaces including uniform or non-uniform planes.

[0105] In another embodiment, ray tracing technology can be used to generate a 2D image preview. In this embodiment, pixels in the 2D image preview are generated based on tracing the path of virtual light and simulating the effect of light encountering a target object or environment. In another embodiment, the user can select other 2D image data as custom representation data, such as photographic images of the target object / environment, image data acquired by imaging sensors (e.g., RGB or grayscale cameras) in the payload of a UAV during past flight missions, or user-drawn sketches. In this embodiment, similar to representation data generated using 3D point cloud data, user-defined representation data can be used in a similar manner and with similar techniques described herein (e.g., associating 2D representation data with 3D map data to generate metadata for the 2D representation data, generating combined map data including 3D map data, 2D representation data, and metadata through data interpolation, or storing representation data in a separate file, etc.).

[0106] Figure 8 Examples of selected viewpoints based on map data according to various embodiments are shown. Figure 8As shown, the 3D point cloud data of the target object 800 can be projected from the top of the point cloud data (e.g., top view 802 of the point cloud data). Selecting the top view projects the 3D point cloud data onto a two-dimensional plane relative to a reference plane, such as horizontal relative to the ground. This reference plane can correspond to the ground, floor, sea level, or any plane selected by the user. The 3D point cloud data of the target object 800 can also be projected from one side of the point cloud data (e.g., side view 804 of the point cloud data). Selecting the side view projects the 3D point cloud data onto a two-dimensional plane relative to a reference plane, such as vertical relative to the ground. This reference plane can correspond to the ground, floor, sea level, or any plane selected by the user. The 3D point cloud data of the target object 800 can also be projected from a viewpoint that includes the most feature points (e.g., point cloud data viewpoint 806 with the most feature points). Selecting the viewpoint with the most feature points in the 3D point cloud data reflects the most features of the 3D point cloud data, ensuring that the generated representation data captures the most unique features of the target object 800.

[0107] Figure 9A and Figure 9B Examples of projecting 3D map data onto a 2D plane according to various embodiments are shown. As described above, 3D point cloud data can be projected onto a 2D plane using perspective projection or orthographic projection. Figure 9A An example is shown of perspective projecting 3D point cloud data of a target object 900 onto a 2D projection plane 902 to generate representational data, such as a 2D image preview 904. Perspective projection, or perspective transformation, is a linear projection where the projection line 906 originates from a single point (e.g., the projection center 908). When the viewer's eye is at the center of projection 908, a projected image 904 is seen on the projection plane 902, and this projection line corresponds to the path of light originating from the target object 900. The effect of perspective projection is that objects at a greater distance appear smaller than objects at a greater distance.

[0108] In the embodiment, when selecting the perspective of the 3D point cloud data (e.g.) Figure 8After the top-down viewpoint 802, side viewpoint 804, or viewpoint 804 with the most feature points shown, the points of the 3D point cloud are first transformed to a coordinate system associated with the selected viewpoint. Then, all points in the point cloud data are scanned based on perspective projection and projected onto cells of the projection plane 902. The top, bottom, left, and right boundaries of the points in each cell of the projection plane 902 are calculated based on the projected points located in each cell plane. Subsequently, pixel blocks can be generated by discretizing each cell of the projection plane 902 according to a specific resolution value. During this process, each pixel block can be updated to record information (e.g., intensity or color value) of the point closest to the projection center to form a final two-dimensional image preview of the representation data. In embodiments, the generated representation data can be image file formats such as JPEG, GIF, PNG, TIFF, BMP, etc., while the map data can be point cloud file formats such as LAS or PLY. In other embodiments, the representation data can be other compressed file formats smaller than the standard point cloud file format. For example, representation data can be generated by selecting a portion of the 3D map data or downsampling the 3D map data for compression. The generated representation data can be stored in different files, or inserted into the original point cloud data via data insertion, as detailed below.

[0109] Figure 9B An example is shown of orthogonally projecting 3D point cloud data of a target object 900 onto a 2D projection plane 902 to generate representation data, such as a 2D image preview 904. Orthogonal projection, or orthogonal transformation, is a parallel projection, where the projection lines 906 are parallel. The projected image 904 on the projection plane 902 is formed by extending the parallel projection lines 906 from each vertex of the target object 900 until they intersect the projection plane 902. The effect of orthogonal projection is that each object line that was originally parallel will remain parallel after projection.

[0110] In the embodiment, when selecting the perspective of the 3D point cloud data (e.g.) Figure 8After viewing from a top-down perspective 802, a side-view perspective 804, or a perspective with the most feature points 804, the points of the 3D point cloud are first transformed to a coordinate system associated with the selected perspective. Then, all points in the point cloud data are scanned based on orthogonal projection and projected onto cells of the projection plane 902. The top, bottom, left, and right boundaries of the points in each cell of the projection plane 902 are calculated based on the projected points located in each cell plane. Subsequently, pixel blocks can be generated by discretizing each cell of the projection plane 902 according to a specific resolution value. During this process, each pixel block can be updated to record information (e.g., intensity or color value) of the point farthest from the projection center to form a final two-dimensional image preview of the representation data. In embodiments, the generated representation data can be image file formats such as JPEG, GIF, PNG, TIFF, BMP, etc., while the map data can be point cloud file formats such as LAS or PLY. In other embodiments, the representation data can be other compressed file formats smaller than the standard point cloud file format. For example, representation data can be generated by selecting a portion of 3D map data or by downsampling and compressing the 3D map data. The generated representation data can be stored in different files or inserted into the original point cloud data via data interpolation, as detailed below.

[0111] Figure 10A and Figure 10B Examples of selected viewpoints for map data in surveying tasks according to various embodiments are shown. Figure 10A An example of point cloud data generated in a mapping task for scanning a target environment is shown at a selected top-down view of 1000. Figure 10B An example of a selected side view 1002 of point cloud data generated in a mapping task for scanning a target environment is shown. In an embodiment, as... Figure 10A and Figure 10B The point cloud data shown can be in a standard point cloud file format such as LAS or PLY files.

[0112] In this embodiment, the point cloud data is colored based on the overlay information of points projected from a selected viewpoint (i.e., projection points). For example... Figure 7 As described above, overlay information 702 can be obtained from an RGB camera or other sensor incorporated into the payload. For example, in some embodiments, the overlay data may include color data of the projected points, which may include pixel values ​​of various color schemes (e.g., 16-bit, 32-bit, etc.). While the scanning sensor acquires point cloud data, color data can be extracted from one or more images acquired by the RGB camera, and these color values ​​can be overlaid on a visualization of the projected points in the point cloud data.

[0113] Despite Figure 7The image is shown in grayscale, but the color data can include a variety of color values ​​depending on the color values ​​of the image data acquired by the RGB camera. In some embodiments, the overlay data can include the height above a reference plane. For example, color values ​​can be assigned to projection points based on their height above the reference plane. The height value can be a relative height value relative to the reference plane or an absolute height value (e.g., relative to sea level). The reference plane can correspond to the ground, a floor, or any plane selected by the user. These values ​​can change monochromatic with height or can change color with height. In some embodiments, the overlay data can represent the intensity value of a point projected from a selective viewing angle. This intensity value can correspond to the return intensity of a laser beam received by a LiDAR sensor. This intensity value can indicate the material composition or properties of objects in the target environment. For example, based on the reflectivity of a material, an image of the material (e.g., the type of material, such as metal, wood, concrete, etc.) can be inferred, and the overlay information can indicate these properties by assigning different color values ​​to different features. Alternatively or additionally, in some embodiments, point cloud data can be overlaid on a map of the target environment scanned from the same viewing angle. For example, points in point cloud data projected from a top-down view can be overlaid on a two-dimensional map provided by a mapping service from the same top-down view.

[0114] Figure 11 Examples of representation data generated from map data in surveying tasks according to various embodiments are shown. For example... Figure 11 As shown, by from Figure 10A The selected top-down viewpoint 1000 of the associated point cloud data projects the points onto a two-dimensional image plane to generate an image preview 1100. In this embodiment, the image preview 1100 can be a standard image file format such as JPEG, GIF, PNG, TIFF, BMP, etc., with a file size (typically in megabytes) much smaller than... Figure 10A The file size of the point cloud data shown is typically in gigabytes. In this embodiment, each pixel of the generated representation data 1100 can also be colored based on some coloring drawing logic to reflect the color value, intensity value, or height value of the associated point cloud data. For example, such as Figure 7 As shown, the overlay information 702 obtained from an RGB camera or other sensor incorporated into the payload can be used to represent a shading plot of the data. For example, in some embodiments, the overlay data may include color data of the projected points, which may include pixel values ​​in various color schemes (e.g., 16-bit, 32-bit, etc.). Color data can be extracted from one or more images acquired by the RGB camera while the scanning sensor acquires point cloud data. These color values ​​can be overlaid on a visualization of the projected points in the point cloud data, or on pixels of the projected image 1100 corresponding to the associated point cloud data.

[0115] Despite Figure 7 The overlay data is depicted as grayscale, but depending on the color values ​​of the image data acquired by the RGB camera, the color data can include a variety of color values. In some embodiments, the overlay data can include the height above a reference plane. For example, color values ​​can be assigned to the projection points based on their height above the reference plane, and color values ​​can be assigned to each pixel of the projected image 1100 associated with each projection point. The height values ​​can be relative to the reference plane or absolute height values ​​(e.g., relative to sea level). The reference plane can correspond to the ground, a floor, or any plane selected by the user. These values ​​can vary monochromaticly with height, or they can change color with height. In some embodiments, the overlay data can represent the intensity value of a point projected from a selective viewing angle. This intensity value can correspond to the return intensity of the laser beam received by the LiDAR sensor. This intensity value can indicate the material composition or properties of objects in the target environment. For example, based on the reflectivity of a material, the properties of the material (e.g., the type of material, such as metal, wood, concrete, etc.) can be inferred, and the overlay information can indicate these properties by assigning different color values ​​to different properties. Alternatively or concurrently, in some embodiments, the pixels of the generated representation data 1100 may be overlaid on a map of the target environment scanned from the same viewpoint. For example, pixels of representation data 1100 associated with points in point cloud data projected from a top-down viewpoint may be overlaid on a two-dimensional map provided by a mapping service from the same top-down viewpoint.

[0116] In some embodiments, the generated representation data (based on perspective projection, orthographic projection, ray tracing, or user-selected image generation) can also be processed to adjust the brightness of the image preview, for example, automatically adjusting the brightness of the entire image preview 1100 to the range most suitable for visualization. For example, the distribution of brightness values ​​of pixels in the image preview can be obtained, and then the mean and standard deviation can be calculated based on the assumption of a Gaussian distribution. Finally, according to an appropriate standard deviation range (typically ±5 or a user-defined range), the brightness values ​​within this range can be linearly mapped to the entire desired brightness range (e.g., between 0 and 255).

[0117] As described above, the generated representation data (e.g., a 2D image preview) can be a standard image file format such as JPEG, GIF, PNG, TIFF, BMP, etc. The pixel format of each pixel in the 2D image preview can include information such as pixel position (x and y coordinates), intensity value, RGB color value, etc. In some embodiments, when the intensity value of a pixel is less than 0, it indicates that the pixel uses RGB coloring. In one embodiment, the pixel format of the generated representation data is as follows.

[0118] struct preview_pixel

[0119] {

[0120] uint32 x;

[0121] uint32 y;

[0122] int8 intensity;

[0123] uchar red;

[0124] uchar green;

[0125] uchar blue;

[0126] };

[0127] Figure 12 Examples of inserting representation data and metadata into map data based on different file formats according to various embodiments are shown. Figure 12 Examples of point cloud data in LAS file format 1200 and PLY file format 1210 are shown. It should be understood that similar techniques disclosed herein can be performed using other standard 3D map data file formats. Figure 12 As shown, a standard LAS file 1200 can have a data structure that includes at least a header block 1202, a main point cloud data block 1204, and a tail block 1206. The header block 1202 typically includes general data such as file format, file creation time, total number of points, and coordinate boundaries. The main point cloud data block 1204 includes map data for all collected points in the point cloud data. The tail block 1206 includes extended information, such as projection information, which is appended to the end of the LAS file without rewriting the entire file.

[0128] As described above, representation data for a LAS file can be generated by projecting 3D point cloud data stored in a LAS file onto a 2D image plane using perspective projection, orthographic projection, or ray tracing techniques. In other embodiments, a user can select and assign representation data from existing images or images drawn by the user. After selecting representation data, it can be associated with point cloud data that includes pre-projected point cloud data. This association can be implemented based on the file format of the point cloud data (such as a LAS file or a PLY file). Taking LAS file 1200 as an example, one way to associate representation data with a LAS file is to store the representation data as a separate image file in a standard image file format such as JPEG, GIF, PNG, TIFF, BMP, etc. In this embodiment, metadata for this separate image file is generated and stored in LAS file 1200, for example, in header block 1202. The metadata for this separate image file may include a pointer that directs a visualization application to the memory address of the separate image file to retrieve the image file and display it to the user as a preview image. In this way, the visualization application only needs to read the header block 1202 of the LAS file 1200 to retrieve image previews with smaller file sizes (typically in kilobytes or megabytes), instead of reading the entire LAS file 1200, which has a huge file size (typically in gigabytes). This significantly reduces processing time and data transfer bandwidth, and provides users with an intuitive preview image to identify a specific LAS file among multiple LAS files. The metadata of the image preview file may also include other information, such as an end pointer indicating the end position of the inserted data, or length data indicating the size of the inserted data.

[0129] Another method for associating representation data with a LAS file is to append the representation data to the tail block 1206 of the original LAS file via data insertion 1208A to create a combined LAS file that includes point cloud data and its associated representation data. In other embodiments, representation data can be inserted in the middle of the main point cloud data block 1204. Furthermore, metadata associated with the generated representation data can be generated. This metadata can be information related to the inserted representation data, such as information related to the start position, end position, or length / size of the inserted representation data. The generated metadata can be inserted into the LAS file in the header block 1202. For example, metadata indicating the start position of the inserted representation data can be written from the GUID data 1 data block to the GUID data 3 data block in the header block 1202 via data insertion 1208B. Similarly, metadata indicating the length of the inserted representation data can be written from the GUID data 4 data block in the header block 1202 via data insertion 1208B.

[0130] After inserting representation data into the original LAS file 1208A and metadata of the representation data into the original LAS file 1208B, a combined LAS file is formed. This combined LAS file and the original LAS file share the same standard point cloud data format. By maintaining consistency in file format between the combined and original point cloud data, any commercial point cloud visualization application / service / browser provided online or offline by any third-party vendor can recognize the combined point cloud data without further modifications. The combined LAS file includes representation data and its metadata, where the representation data serves as a preview image, allowing users to easily identify specific point cloud files across multiple files. For example, when a user wants to examine recorded point cloud data files using a visualization application, a preview image of each recorded point cloud data file can be displayed, allowing the user to visually identify the file of interest. Users do not need to open all point cloud data files to examine their contents; they only need to open the file of interest to perform further applications, such as post-processing the file of interest to perform playback of that point cloud record, or further editing the file of interest. This saves users time selecting from multiple point cloud data files. It can also provide an image preview before downloading point cloud data to visualization applications, thus saving data communication bandwidth used to download point cloud data files from movable objects or computing devices.

[0131] like Figure 12 As shown, a standard PLY file 1210 can have a data structure comprising at least a header block 121, a main point cloud data block 1214, and a tail block 1216. The header block 1212 typically includes general data, such as file format, variant versions of the file format, etc. The main point cloud data block 1214 includes map data for all points collected in the point cloud data. The tail block 1216 includes extended information, such as projection information, which is appended to the end of the PLY file without rewriting the entire file.

[0132] As described above, representation data for a PLY file can be generated by projecting 3D point cloud data stored in the PLY file onto a 2D image plane using perspective projection, orthographic projection, or ray tracing techniques. In other embodiments, a user can select and assign representation data from existing images or user-drawn images. After selecting representation data, it can be associated with point cloud data that includes pre-projected point cloud data. This association can be implemented based on the file format of the point cloud data (such as a LAS file or a PLY file). Taking PLY file 1210 as an example, one way to associate representation data with a PLY file is to store the representation data as a separate image file in a standard image file format such as JPEG, GIF, PNG, TIFF, BMP, etc. In this embodiment, metadata for this separate image file is generated and stored in PLY file 1210, for example, in header block 1212. The metadata for this separate image file may include a pointer that directs a visualization application to the memory address of the separate image file to retrieve the image file and display it to the user as a preview image. In this way, the visualization application only needs to read the header block 1212 of the PLY file 1210 to retrieve an image preview with a smaller file size (typically in kilobytes or megabytes), instead of reading the entire PLY file 1210, which has a huge file size (typically in gigabytes). This significantly reduces processing time and data transfer bandwidth, and provides the user with an intuitive preview image to identify a specific PLY file among multiple PLY files. The metadata of the image preview file may also include other information, such as an end pointer indicating the end position of the inserted data, or length data indicating the size of the inserted data.

[0133] Another method for associating representation data with a PLY file is to append the representation data to the end block 1216 of the original PLY file via data insertion 1218A to create a combined PLY file that includes point cloud data and its associated representation data. In other embodiments, representation data can be inserted in the middle of the main point cloud data block 1214. Furthermore, metadata associated with the generated representation data can be generated. This metadata can be information related to the inserted representation data, such as information about the start position, end position, or length / size of the inserted representation data. The generated metadata can be inserted into the PLY file in the header block 1212. For example, metadata indicating the start position, end position, or length / size of the inserted representation data can be written to the comment line of the header block 1212 via data insertion 1218B.

[0134] Alternatively, metadata or the inserted representation data can also be saved as a user-defined element of a PLY file and written to header block 1212. An example of a four-pixel user-defined element for metadata insertion can be defined by the user as follows.

[0135] element dji_pc_thumbnail_pixel 4

[0136] property uint32 x

[0137] property uint32 y

[0138] property int8 intensity

[0139] property uchar red

[0140] property uchar green

[0141] property uchar blue

[0142] 0,0,-1,100,100,100

[0143] 0,1,-1,100,100,100

[0144] 1,0,-1,100,100,100

[0145] 1,1,-1,100,100,100

[0146] After inserting representation data into the original PLY file 1218A and metadata of the representation data into the original PLY file 1218B, a combined PLY file is formed. This combined PLY file and the original PLY file have the same standard point cloud data format. By maintaining consistency in file format between the combined and original point cloud data, any commercial point cloud visualization application / service / browser provided online or offline by any third-party vendor can recognize the combined point cloud data without further modifications. The combined PLY file includes representation data and its metadata, where the representation data serves as a preview image, allowing users to easily identify specific point cloud files across multiple files. For example, when a user wants to examine recorded point cloud data files using a visualization application, a preview image of each recorded point cloud data file can be displayed to the user, allowing them to visually identify the file of interest. Users do not need to open all point cloud data files to examine their contents; they only need to open the file of interest to perform further applications, such as post-processing the file of interest to perform playback of the point cloud record of interest, or further editing the file of interest. This saves users time selecting from multiple point cloud data files. It can also provide an image preview before downloading point cloud data to visualization applications, thus saving data communication bandwidth used to download point cloud data files from movable objects or computing devices.

[0147] Figure 13 Examples of combined map data and representation data rendered on a client device's display according to various embodiments are shown. The client device's display 1300 can be used to display a point cloud data folder or file by executing a visualization application on the client device. For example... Figure 13 As shown, the left pane 1302 of the display 1300 can display surveying tasks recorded by the user. For example, surveying tasks may include task folders (such as...). Figure 13A folder named "Task 1" is provided, which stores combined point cloud data files (such as files named "Point Cloud 1.las" and "Point Cloud 2.ply" under the "Task 1" folder), including the recorded point cloud, its associated representation data, and metadata. As mentioned above, the combined map data can be any standard point cloud file format, such as LAS or PLY files, which can be recognized by any commercial point cloud visualization application / service / browser provided online or offline by any third-party vendor without further modification. In other embodiments, the representation data generated according to this disclosure can optionally be saved as separate image files, whose filenames are associated with the combined point cloud data files (such as files named "Point Cloud 1.jpg" and "Point Cloud 2.jpg" under the "Task 1" folder). In embodiments, the representation data or separate image files can be other standard image file formats, such as JPEG, GIF, PNG, TIFF, BMP files, etc. In some embodiments, the filename of a separate image preview file can be the same as its associated combined point cloud data file. In other embodiments, other naming rules can be applied to save separate image preview files.

[0148] Once a user opens the "Task 1" folder, for example by clicking the folder icon, the right pane 1304 of the monitor 1300 can display all files stored in the "Task 1" folder. Figure 13 As shown, the representation data is inserted into the original LAS file so that the user can view a preview image 1306 of the combined point cloud LAS file. The representation data can optionally be saved as a separate image file 1308 that the user can also view. Figure 13 As shown, representation data is inserted into the original PLY file so that the user can view a preview image 1310 of the combined point cloud PLY file. The representation data can optionally be saved as a separate image file 1312 that the user can also view. This allows the user to intuitively select point cloud data of interest for further processing, such as playing back the point cloud data recording using a playback application, or editing or post-processing the point cloud data using a third-party application.

[0149] Figure 14 A flowchart illustrating a method for generating representation data using map data acquired by a sensor in a moving object environment, according to various embodiments, is shown. At operation / step 1402, the method may include receiving three-dimensional data acquired by a sensor. In some embodiments, the sensor is a scanning sensor, including a light detection and ranging (LiDAR) sensor connected to the moving object. In some embodiments, the three-dimensional map data includes point cloud data (e.g., LAS or PLY files) acquired by the LiDAR sensor, including multiple points and corresponding color data. In some embodiments, the moving object is a drone (UAV). In some embodiments, the sensor is connected to the UAV.

[0150] In operation / step 1404, the method may include generating representation data by projecting three-dimensional map data onto a two-dimensional plane based on a selected viewpoint. In some embodiments, the representation data is image data (e.g., JPEG, GIF, PNG, TIFF, BMP files). The representation data provides a preview of the three-dimensional map data, wherein the file size of the representation data is smaller than the three-dimensional map data for fast rendering. In some embodiments, the selected viewpoint is as follows: Figure 8 The aforementioned top-down view, side-down view, or view with the most feature points. In some embodiments, this means that data is projected onto a two-dimensional plane perpendicular to the central axis of the selected viewpoint. In some embodiments, such as Figure 9A and Figure 9B As shown, three-dimensional map data is projected onto a two-dimensional plane using perspective projection or orthographic projection. In other embodiments, representation data can be obtained in other ways, such as by ray tracing of three-dimensional map data, selection by a user from existing images, or assignment by a user through user drawing, etc. In some embodiments, projecting three-dimensional map data onto a two-dimensional plane can be performed by: (1) transforming point cloud data to a coordinate system associated with a selected viewpoint, and (2) scanning through the points of the point cloud data and projecting them onto a two-dimensional plane in that coordinate system to generate representation data.

[0151] In operation / step 1406, the method may include associating representation data with 3D map data. In some embodiments, the 3D map data is associated with a file format such as a PLY file or a LAS file, and the representation data is associated with 2D map data based on that file format. In some embodiments, this association includes generating combined map data based on the generated representation data, the original 3D map data, and their file formats, such as... Figure 15 As further described in the text.

[0152] Figure 15 A flowchart illustrating a method for associating representation data with map data to generate combined map data according to various embodiments is shown. Figure 15As shown, composite map data is generated as output based on the input 3D map data, the representation data generated from the 3D map data, and the file format of the 3D map data. In some embodiments, the association is implemented by inserting the generated representation data into the 3D map data to generate the composite map data, wherein the insertion position of the representation data is determined based on the file format. In some embodiments, metadata of the representation data may also be generated and inserted into the 3D map data so that the original 3D map data, the representation data, and the metadata are all included in the output composite map data. The metadata of the redirected data includes information related to the inserted representation data, such as the start position, end position, and / or length / size of the inserted data.

[0153] In operation / step 1502, the method may include generating composite map data based on representation data, 3D map data, and a file format associated with the 3D map data. In some embodiments, the output composite map data has the same file format as the input original 3D map data, so that any commercial point cloud visualization application / service / browser provided online or offline by any third-party vendor can recognize the data without further modification. As shown in operations / steps 1504 and 1506, the operation / step of associating representation data with map data to generate composite map data may include two sub-operations / sub-steps of adding representation data and its metadata to the 3D map data using data interpolation techniques.

[0154] In operation / step 1504, the method may include inserting representation data into the 3D map data at a starting position (e.g., at the midpoint of the 3D map data or appended to the tail of the 3D map data). In some embodiments, representation data is inserted at the midpoint of the 3D map data or appended to the tail of the 3D map data, for example, by inserting representation data into the tail block of a PLY or LAS file.

[0155] In operation / step 1506, the method may include inserting metadata representing data (including a start pointer, an end pointer, and / or length data) into the 3D map data (e.g., in the header of the 3D map data). In some embodiments, the metadata representing the data includes a start pointer indicating the start position of the inserted representation data, an end pointer indicating the end position of the inserted representation data, or length data indicating the length of the inserted representation data. In some embodiments, the metadata is inserted into the header of the 3D map data, for example, into one or more GUID blocks in the header block of an LAS file, or into one or more comment lines in the header block of a PLY file. In some embodiments, the metadata may include custom information (e.g., indicating the generation time of the data) and be inserted into the 3D map data, for example, by using user-defined elements of a PLY file to insert metadata representing the data.

[0156] In some embodiments, before inserting representation data and metadata, the method may further include downsampling the 3D map data by removing outlier data and generating representation data based on the downsampled 3D map data.

[0157] In some embodiments, the generated representation data can be colored based on overlay information, such as color values, intensity values, or height values ​​corresponding to 3D mapping point cloud data. In some embodiments, coloring is implemented based on predefined coloring drawing logic.

[0158] In some embodiments, the brightness of the generated representation data can be adjusted based on the intensity value corresponding to the 3D map data.

[0159] In some embodiments, a user can define user-defined data associated with the representation data, and the method can further include updating the representation data and metadata based on the user-defined data. For example, a user can set a specific size for the representation data to display a suitable image preview (e.g., as a small / medium / large image). The representation data can be updated based on the user-defined size, and the metadata can also be updated to reflect this change.

[0160] Figure 16 Examples of supporting movable object interfaces in a software development environment are shown, according to various embodiments. Figure 16As shown, the movable object interface 1603 can be used to provide access to the movable object 1601 in a software development environment 1600, such as a software development kit (SDK) environment. As used herein, the SDK can be an onboard SDK implemented in an onboard environment connected to the movable object 1601. The SDK can also be a mobile SDK implemented in an off-board environment connected to a mobile device. Furthermore, the movable object 1601 can include various functional modules AC 1611-1613, and the movable object interface 1603 can include different interface components AC 1631-1633. Each interface component AC 1631-1633 in the movable object interface 1603 corresponds to a module AC 1611-1613 in the movable object 1601. In some embodiments, the interface components can be rendered on a user interface of a display of a mobile device or other computing device communicating with the movable object. In such an example, the rendered interface components can include optional command buttons for receiving user input / instructions to control the corresponding functional modules of the movable object.

[0161] According to various embodiments, the movable object interface 1603 may provide one or more callback functions to support a distributed computing model between the application and the movable object 1601.

[0162] The application can use this callback function to confirm whether the movable object 1601 has received the command. Furthermore, the application can use this callback function to receive the execution result. Therefore, even if the application and the movable object 1001 are spatially and logically separate, they can still interact.

[0163] As shown in Figure 10, interface components AC 1631-1633 can be associated with listeners AC 1641-1643. Listeners AC 1641-1643 can instruct interface components AC 1631-1633 to use appropriate callback functions to receive information from (one or more) related modules.

[0164] Furthermore, the data manager 1602, which prepares data 1620 for the movable object interface 1603, can separate and package the relevant functions of the movable object 1601. The data manager 1602 can be onboard, connected to or located on the movable object 1601, and prepares the data 1620, which is then transmitted to the movable object interface 1603 via communication between the movable object 1601 and the mobile device. Alternatively, the data manager 1602 can be external, connected to or located on the mobile device, and prepares the data 1620 for the movable object interface 1603 via communication within the mobile device. Additionally, the data manager 1602 can be used to manage data exchange between the application and the movable object 1601. Therefore, application developers do not need to participate in the complex data exchange process.

[0165] For example, an onboard or mobile SDK can provide a series of callback functions for delivering instant messages and receiving execution results from a movable object. The onboard or mobile SDK can configure the lifecycle of these callback functions to ensure stable and complete information exchange. For instance, the onboard or mobile SDK can establish a connection between a movable object and an application on a smartphone (e.g., using an Android or iOS system). After the smartphone system's lifecycle has ended, callback functions for receiving information from the movable object can leverage patterns within the smartphone system and update statements accordingly based on different stages of the smartphone system's lifecycle.

[0166] Figure 17 Examples of movable object interfaces according to various embodiments are shown. Figure 17 As shown, the movable object interface 1703 can be rendered on a display of a mobile device or other computing device representing the state of different components of the movable object 1701. Therefore, applications in the movable object environment 1700, such as applications 1704-1706, can access and control the movable object 1701 through the movable object interface 1703. As described above, these applications may include inspection application 1704, viewing application 1705, and calibration application 1706.

[0167] For example, the movable object 1701 may include various modules such as camera 1711, battery 1712, gimbal 1713 and flight controller 1714.

[0168] Accordingly, the movable object interface 1703 may include a camera component 1721, a battery component 1722, a gimbal component 1723, and a flight controller component 1724, which will be rendered on a computing device or other computing device to receive input / commands from the user using 1704-1706.

[0169] In addition, the movable object interface 1703 may include a ground station component 1726 associated with the flight controller component 1724. The ground station component is used to perform one or more flight control operations that require high-level privileges.

[0170] Figure 18 Examples of components of a movable object in a software development kit (SDK) according to various embodiments are shown. Figure 18 As shown, the drone class 1801 in SDK 1800 is an aggregation of other components 1802-1807 of a movable object (e.g., a drone). The drone class 1801, which has access to other components 1802-1807, can exchange information with and control other components 1802-1807.

[0171] According to various embodiments, an application can only be accessed by one instance of the drone class 1801. Alternatively, multiple instances of the drone class 1801 can exist in an application.

[0172] In the SDK, applications can connect to instances of the Drone class 1801 to upload control commands to the movable object. For example, the SDK can include functionality for establishing a connection to the movable object. Furthermore, the SDK can also disconnect from the movable object using end-connection functionality. Once connected to the movable object, developers can access other classes (such as Camera class 1802, Battery class 1803, Gimbal class 1804, and Flight Controller class 1805). The Drone class 1801 can then be used to invoke specific functions, such as providing access data that the flight controller can use to control the behavior of the movable object and / or restrict its movement.

[0173] According to various embodiments, the application can use battery class 1803 to control the power supply of a mobile object. Furthermore, the application can use battery class 1803 to plan and test various flight mission schedules. Since the battery is one of the most constrained components of a mobile object, the application carefully considers the battery's state, not only for the safety of the mobile object but also to ensure that the mobile object can complete its designated tasks. For example, battery class 1803 can be configured such that if the battery power is low, the mobile object can terminate the mission and return immediately. For example, if it is determined that the mobile object's battery power is below a threshold level, the battery class can cause the mobile object to enter a power-saving mode. In power-saving mode, the battery class can shut down or reduce the available power of components that are not essential for the mobile object's return. For example, cameras and other accessories not used for navigation can be powered off, thereby increasing the available power of flight controllers, motors, navigation systems, and any other systems required for the mobile object's return and safe landing.

[0174] By using the SDK, applications can obtain the current state and information of the battery by calling functions to request information from the drone battery class. In some embodiments, the SDK may include functionality for controlling the frequency of such feedback.

[0175] According to various embodiments, applications can use the camera class 1802 to define various operations of a movable object, such as a camera in a drone. For example, in the SDK, the camera class includes functions for receiving media data from an SD card, acquiring and setting photo parameters, taking photos, and recording videos.

[0176] Applications can use the Camera class 1802 to change settings for photos and recordings. For example, the SDK can include functionality that allows developers to resize captured photos. Additionally, applications can use the Media class to maintain photos and recordings.

[0177] According to various embodiments, the application can use the gimbal class 1804 to control the viewing angle of a moving object. For example, the gimbal class can be used to configure the actual viewing angle, such as setting the first-person view of the moving object. Furthermore, the gimbal class can also be used to automatically stabilize the gimbal, focusing it in one direction. Additionally, the application can use the gimbal class to change the viewing angle for detecting different objects.

[0178] According to various embodiments, applications can use flight controller class 1805 to provide various flight control information and statuses regarding a movable object. As described above, the flight controller class may include functions for receiving and / or requesting access to data used to control the movement of the movable object across different areas within the movable object environment.

[0179] By using the flight controller class, applications can, for example, monitor flight status using instant messaging. For instance, a callback function in the flight controller class can send back an instant message every 1000 milliseconds (1000ms).

[0180] Furthermore, flight controller classes allow users of the application to analyze real-time messages received from moving objects. For example, pilots can analyze data from each flight to further improve their flying skills.

[0181] According to various embodiments, an application can use ground station class 1807 to perform a series of operations for controlling movable objects.

[0182] For example, the SDK may require the application to have an SDK-LEVEL-2 key for using the ground station class. The ground station class provides one-click flight, one-click return, manual control of the drone via the application (i.e., joystick mode), setting cruise and / or waypoints, and various other mission scheduling functions.

[0183] According to various embodiments, an application can use a communication component to establish a network connection between the application and a movable object.

[0184] Many features can be implemented in, or using, hardware, software, firmware, or a combination thereof. Therefore, the features can be implemented using a processing system (e.g., including one or more processors). Exemplary processors may include, but are not limited to, one or more general-purpose microprocessors (e.g., single-core or multi-core processors), application-specific integrated circuits (ASICs), application-specific instruction set processors (ASICs), graphics processing units, physical processing units, digital signal processing units, coprocessors, network processing units, audio processing units, encryption processing units, etc.

[0185] Features may be implemented in, or by means of, a computer program product, which is a storage medium or computer-readable medium on which / therein stores instructions that can be used to program a processing system to perform any of the features described herein. Storage media may include, but are not limited to, any type of disk (including floppy disks, optical disks, DVDs, CD-ROMs, microdrives, and magneto-optical disks), ROM, RAM, EPROM, EEPROM, DRAM, VRAM, flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0186] Features stored on any machine-readable medium can be incorporated into software and / or firmware to control the hardware of the processing system and to enable the processing system to interact with other entities using the results. Such software or firmware may include, but is not limited to, application code, device drivers, operating systems, and execution environments / containers.

[0187] The features of this invention can also be implemented in hardware, for example, using hardware components such as application-specific integrated circuits (ASICs) and field-programmable gate arrays (FPGAs). Implementing a hardware state machine to perform the functions described herein will be readily apparent to those skilled in the art.

[0188] Furthermore, the present invention can be conveniently implemented using one or more conventional general-purpose or special-purpose digital computers, computing devices, machines, or microprocessors (including one or more processors, memory, and / or computer-readable storage media programmed according to the teachings of this disclosure). It will be readily apparent to those skilled in the art that appropriate software code can be readily prepared based on the teachings of this disclosure.

[0189] Although various embodiments have been described above, it should be understood that they are merely illustrative and not limiting. It will be apparent to those skilled in the art that various changes in form and detail can be made without departing from the spirit and scope of the invention.

[0190] The invention has been described above with the aid of functional building blocks, which illustrate the performance of specific functions and the relationships between them. For ease of description, the boundaries of these functional building blocks are generally arbitrarily defined herein. Alternative boundaries can be defined as long as the specific functions and their relationships are properly performed. Therefore, any such alternative boundaries are within the scope and spirit of the invention.

[0191] The foregoing description has been provided for purposes of illustration and description and is not intended to be exhaustive or to limit the invention to the precise forms disclosed. The breadth and scope of the invention should not be limited to any of the exemplary embodiments described above. Many changes and variations will be apparent to those skilled in the art. These changes and variations include any related combinations of the disclosed features. The embodiments were chosen and described in order to best explain the principles of this disclosure and its practical application, thereby enabling others skilled in the art to understand the various embodiments of the invention and the various modifications suitable for the particular intended use. The scope of the invention is defined by the appended claims and their equivalents.

[0192] In the above embodiments, unless otherwise specifically stated, disjunctive languages ​​such as “at least one of A, B, or C” should be understood to mean A, B, or C, or any combination thereof (e.g., A, B, and / or C). Therefore, disjunctive languages ​​are not intended to, and should not be construed as, implying that a given embodiment requires at least one A, at least one B, or at least one C to be present.

Claims

1. A method for generating representation data of three-dimensional map data, comprising: Receive 3D map data acquired by sensors; The three-dimensional map data is projected onto a two-dimensional plane based on a selected viewpoint, and then representation data is generated by a processor. as well as Associating the representation data with the 3D map data via the processor includes: inserting the representation data into the 3D map data to form combined map data or storing the representation data in a separate file and associating it with the 3D map data, wherein the representation data is a 2D image showing a previewable 3D map data.

2. The method according to claim 1, wherein, The 3D map data is point cloud data, the sensor is a light detection and ranging LiDAR sensor, and the representation data is image data.

3. The method according to claim 1, wherein, The representation data is configured to provide a preview of the 3D map data, and the size of the representation data is smaller than that of the 3D map data so that the representation data is rendered faster compared to the 3D map data.

4. The method according to claim 1, wherein, The selected viewpoint is a top-down viewpoint, a side-view viewpoint, or the viewpoint with the most feature points in the three-dimensional map data.

5. The method according to claim 1, wherein, The 3D map data is point cloud data, and projecting the 3D map data onto a 2D plane based on a selected viewpoint includes: Transform the point cloud data to a coordinate system associated with the selected viewpoint; and The representation data is generated by scanning each point of the point cloud data and projecting it onto a two-dimensional plane in the coordinate system.

6. The method according to claim 1, wherein, The 3D map data is projected onto the 2D plane using orthographic or perspective projection, or the representation data is obtained by performing ray tracing on the 3D map data.

7. The method according to claim 1, wherein, The 3D map data is associated with a file format, and associating the representation data with the 3D map data further includes: Combined map data is generated based on the representation data, the 3D map data, and the file format, wherein the combined map data and the 3D map data have the same file format.

8. The method according to claim 7, wherein, Generating the combined map data includes: The representation data is inserted into the 3D map data via the processor; and The metadata representing the data is inserted into the 3D map data via the processor.

9. The method according to claim 8, wherein, The representation data is inserted at the starting position of the three-dimensional map data, and the metadata of the representation data includes at least one of a start pointer, an end pointer, and length data, wherein the start pointer indicates the starting position of the inserted representation data, the end pointer indicates the ending position of the inserted representation data, and the length data indicates the length of the inserted representation data.

10. The method according to claim 8, wherein, The representation data is inserted at the midpoint of the 3D map data or appended to the end of the 3D map data, and the metadata is inserted at the beginning of the 3D map data.

11. The method according to claim 8, wherein, The file format of the 3D map data is LAS file, and the metadata is inserted into the GUID block in the header of the LAS file; or the file format of the 3D map data is PLY file, and the metadata is inserted into the comments in the header of the PLY file.

12. The method according to claim 1, further comprising: The processor colors the representation data based on color values, intensity values, or height values ​​corresponding to the 3D point cloud data, wherein the coloring is based on predefined coloring drawing logic.

13. The method according to claim 1, further comprising: The brightness of the representation data is adjusted by the processor based on the intensity value corresponding to the three-dimensional map data.

14. The method of claim 8, further comprising: The processor receives user-defined data associated with the representation data; as well as Based on the user-defined data, the representation data and the metadata are updated via the processor.

15. A system for generating representation data of three-dimensional map data, comprising: A sensor, which is connected to a movable object and is configured to acquire three-dimensional map data of the environment; as well as A processor that communicates with the sensor or the movable object, and the processor is configured to: Receive the three-dimensional map data acquired by the sensor; The 3D map data is projected onto a 2D plane based on a selected viewpoint to generate representation data; as well as Associating the representation data with the 3D map data includes: inserting the representation data into the 3D map data to form combined map data, or storing the representation data in a separate file and associating it with the 3D map data, wherein the representation data is a 2D image showing a previewable 3D map data.

16. The system according to claim 15, wherein, The 3D map data is associated with a file format, and when the representation data is associated with the 3D map data, the processor is further configured to: Based on the representation data, the 3D map data, and the file format, combined map data is generated.

17. The system according to claim 16, wherein, When generating the combined map data, the processor is also configured to: Insert the representation data into the 3D map data; and The metadata representing the data is inserted into the 3D map data.

18. An apparatus for generating representation data of three-dimensional map data, comprising: processor; as well as A storage medium storing instructions that, when executed by the processor, cause the processor to perform the following operations: Receive 3D map data acquired by sensors; The 3D map data is projected onto a 2D plane based on a selected viewpoint to generate representation data; as well as Associating the representation data with the 3D map data includes: inserting the representation data into the 3D map data to form combined map data, or storing the representation data in a separate file and associating it with the 3D map data, wherein the representation data is a 2D image showing a previewable 3D map data.

19. The device according to claim 18, wherein, The 3D map data is associated with a file format, and when the representation data is associated with the 3D map data, the instructions, when executed by the processor, also cause the processor to perform the following operations: Based on the representation data, the 3D map data, and the file format, combined map data is generated.

20. The device according to claim 19, wherein, When generating the combined map data, the instructions, when executed by the processor, also cause the processor to perform the following operations: Insert the representation data into the 3D map data; and The metadata representing the data is inserted into the 3D map data.

Images