Method and system for serving an object

Abstract

A system for servicing an object and methods of making and using the same. A mobile platform can perform one or more tasks on an object while maintaining a distance from the object. Thus, the mobile platform can avoid collisions with the object during servicing. If the position or orientation of the object changes during servicing, the service point of the object can be updated as needed. Servicing can include performing one or more tasks on the object, and the mobile platform can advantageously include one or more payloads to perform one or more selected tasks. Thus, servicing the object can be advantageously automated and require less human intervention.

Description

[0001] Copyright Notice

[0002] Portions of this patent document contain material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights. Technical Field

[0003] Embodiments of the present invention relate to mobile platform technology, and more particularly but not exclusively, to methods and systems for object services. Background Art

[0004] Object services, such as surface inspection, are very important for the routine maintenance of various objects, such as passenger aircraft, ships, and buildings. For example, due to environmental factors such as salt spray, dust, lightning strikes, and foreign object bombardment, the surface paint layer of a passenger aircraft is easily damaged. Such damage can result in defects such as paint layer peeling or cracking. The defect can be located anywhere on the fuselage of the passenger aircraft.

[0005] Traditionally, defect inspection mainly relies on humans to visually inspect at close range with the assistance of ladders. Human visual inspection usually requires covering the entire fuselage of the passenger aircraft. Therefore, such inspection often consumes a large amount of manpower and time.

[0006] In view of the above, there is a need for methods and systems for detecting objects that overcome the disadvantages of existing methods and systems. Summary of the Invention

[0007] The present disclosure relates to a system for servicing an object and methods of making and using the same.

[0008] According to a first aspect disclosed herein, moving towards a selected service point of the object, wherein the acquisition of the position of the selected service point is based on: the spatial relationship between the selected service point and the object, and the position and orientation of the object; maintaining the distance between the mobile platform and the object; and performing a task on the object during the maintaining.

[0009] According to a second aspect disclosed herein, a method for servicing an object by a mobile platform, comprising: maintaining the distance between the mobile platform and the object, wherein the distance is determined and / or dynamically adjusted based on one or more operating conditions of the mobile platform; and performing a task on the object during the maintaining.

[0010] According to a third aspect disclosed herein, a method for serving an object by a mobile platform includes: moving towards a selected service point of the object, wherein a path of the movement is determined according to a position of the selected service point; maintaining a distance between the mobile platform and the object; and performing a task on the object during the maintaining.

[0011] According to a fourth aspect disclosed herein, there is provided a system for serving an object, including: a travel controller, configured to operate on a mobile platform and guide the mobile platform to serve at least one object as described above.

[0012] According to a fifth aspect disclosed herein, there is provided an unmanned aerial vehicle (UAV) for serving an object, including: a travel controller, guiding the UAV to serve at least one object as described above; and one or more propellers, configured to move the UAV based on instructions from the travel controller.

[0013] According to a sixth aspect disclosed herein, there is provided a non-transitory computer-readable storage medium storing one or more programs, the one or more programs including instructions which, when executed by a travel controller, instruct the travel controller to serve at least one object as described above. Brief Description of the Drawings

[0014] Figure 1 is an example diagram showing an embodiment of a mobile platform shown in relation to an object.

[0015] Figure 2 is shown installed in Figure 1 an example diagram of an embodiment of a mobile platform control system on a mobile platform.

[0016] Figure 3 is shown by Figure 1 an exemplary top-level process schematic diagram of an embodiment of a method for serving an object by a mobile platform in.

[0017] Figure 4 is shown Figure 1 an example diagram of an alternative embodiment of a mobile platform in which the mobile platform monitors the distance from an object in one or more directions.

[0018] Figure 5 is shown Figure 4 an example diagram of an alternative embodiment of a mobile platform in which the mobile platform includes at least one of an ultrasonic sensor and a vision sensor.

[0019] Figure 6 is an example diagram showing distance measurement using stereo observation for Figure 4 in.

[0020] Figure 7 is an example diagram showing distance measurement using triangulation in Figure 4 the distance measurement in

[0021] Figure 8 is an example diagram showing distance measurement using ultrasonic sensing in Figure 4 the distance measurement in

[0022] Figure 9 is an example diagram showing Figure 5 an exemplary detailed diagram of an embodiment of an ultrasonic sensor and a vision sensor in

[0023] Figure 10 is an example diagram showing Figure 4 an exemplary detailed diagram of another alternative embodiment of a mobile platform in , where the mobile platform includes an unmanned aerial vehicle (UAV).

[0024] Figure 11 is an exemplary flowchart showing determination of distance based on vision sensing and ultrasonic sensing in Figure 4 the distance in

[0025] Figure 12 is an example diagram showing Figure 1 an alternative embodiment of a mobile platform in , where the mobile platform moves towards a selected service point of an object.

[0026] Figure 13 is an example diagram showing changing the position of a selected service point by changing the Figure 12 position or orientation of an object in

[0027] Figure 14 is an example diagram showing determination of the position of an object through control points in Figure 12 the position of the object in

[0028] Figure 15 is an example diagram showing determination of the orientation of an object through one or more control points in Figure 12 the orientation of the object in

[0029] Figure 16 is an exemplary detailed diagram showing determination of the Figure 12 position and orientation of an object in , where two-dimensional determination of the position and orientation of the object is performed.

[0030] Figure 17 is an example diagram showing determination of the Figure 12 position of a selected service point in

[0031] Figure 18 is an example diagram showing Figure 12 an object in , where the mobile platform is provided with a travel path.

[0032] Figure 19 is an example diagram showing Figure 1 an alternative embodiment of the mobile platform in [[ID=]], where the mobile platform includes a Differential Global Positioning System (DGPS) module.

[0033] Figure 20 is an example diagram showing Figure 19 exemplary details of an alternative embodiment of the mobile platform in [[ID=]], where the mobile platform includes an unmanned aerial vehicle.

[0034] Figure 21 is an example diagram showing Figure 1 an alternative embodiment of the mobile platform in [[ID=]], where the mobile platform includes a payload.

[0035] Figure 22 is an example diagram showing Figure 21 exemplary details of an alternative embodiment of the mobile platform in [[ID=]], where the mobile platform includes an unmanned aerial vehicle.

[0036] Figure 23 is an example diagram showing Figure 1 another alternative embodiment of the mobile platform in [[ID=]], where the mobile platform transmits data to a computer.

[0037] Figure 24 is an example flowchart showing Figure 3 an alternative embodiment of the method in [[ID=]], where the mobile platform captures an image of an object.

[0038] Figure 25 is an example diagram showing Figure 1 another alternative embodiment of the mobile platform in [[ID=]], where the mobile platform operates in cooperation with one or more other mobile platforms.

[0039] Figure 26 is an example diagram showing a system for servicing Figure 1 an object in [[ID=]], where the system includes a terminal device associated with the mobile platform.

[0040] Figure 27 is an example diagram showing Figure 26 an alternative embodiment of the terminal device in [[ID=]].

[0041] Figure 28 is an example diagram showing Figure 26 another alternative embodiment of the terminal device in [[ID=]], where the terminal device is coupled to a remote control.

[0042] Figure 29 is an example showing the use of Figure 26Exemplary flowchart of an embodiment of a method for a terminal device to operate a mobile platform.

[0043] Figure 30 shows Figure 29 Exemplary flowchart of an alternative embodiment of the method in, where the terminal device acquires a model of an object.

[0044] It should be noted that the drawings are not drawn to scale, and for illustrative purposes, elements with similar structures or functions in all figures are generally denoted by the same reference numerals. It should also be noted that the drawings are only used to assist in describing the preferred embodiments. The drawings are not used to illustrate all aspects of the described embodiments and do not limit the scope of the present invention.

[0045] Detailed description of the present invention

[0046] Since existing methods and systems cannot effectively automatically detect objects, it can be demonstrated that methods and systems for improving efficiency in detecting objects and the automated detection process are needed and can provide a basis for a wide range of applications. Such applications can include detecting airplanes, ships, and buildings. Without being limited to detection, the methods and systems can be used for any industrial operation involving implementing functions at selected locations of objects. According to the embodiments disclosed herein, as Figure 1 shown, the system 100 can achieve this result.

[0047] The system 100 can be used to service the object 300. The object 300 can include any shaped structure. Exemplary structures can include airplanes, ships, space shuttles, buildings, bridges, mountains, caves, tunnels, transportation pipelines, water bodies, and / or the like. The shape can be fixed and / or vary over time. The object 300 can change its shape by itself and / or can change its shape due to external forces, including with and / or without human participation. The object 300 can be stationary and / or moving. For example, the object 300 can perform rotational and / or translational movements.

[0048] The system 100 can service the object 300, for example, by performing tasks (and / or taking actions) regarding (and / or involving) the object 300, including any tasks beneficial to the object 300. Exemplary actions can include detecting, repairing, providing maintenance for the object 300, and / or acquiring data of the object 300.

[0049] Figure 1System 100 is shown including mobile platform 200. Examples of mobile platform 200 can include, but are not limited to, bicycles, cars, trucks, ships, boats, trains, helicopters, airplanes, and various hybrids thereof. In some embodiments, mobile platform 200 can include an unmanned aerial vehicle (UAV). Popularly known as a "drone", an unmanned aerial vehicle is an aircraft without a human pilot (or operator) on board, and its flight can be automatically controlled or controlled by a remote pilot (or sometimes both). Currently, unmanned aerial vehicles are increasingly being found in civilian applications including various aerial operations such as data collection or delivery. The systems and methods of the present invention are applicable to a variety of types of unmanned aerial vehicles, including but not limited to quadcopters (also known as quadrotor helicopters or quadcopters), single-rotor, dual-rotor, tri-rotor, hexacopter, and octocopter rotary-wing unmanned aerial vehicles, fixed-wing unmanned aerial vehicles, and rotary-wing-fixed-wing hybrid unmanned aerial vehicles.

[0050] As Figure 1 shown, mobile platform 200 can operate to maintain a distance D from object 300. Distance D can refer to the spacing between mobile platform 200 and object 300. Distance D, for example, can include a predetermined distance and / or a predetermined distance range. Advantages of maintaining distance D can include, for example, preventing mobile platform 200 from colliding with object 300.

[0051] Although Figure 1 one distance D is shown, mobile platform 200 can be located at one or more positions at the same and / or different distances D from object 300.

[0052] Referring Figure 2 , mobile platform 200 is shown including mobile platform control system 210. Mobile platform control system 210 can include one or more sensors 230. Mobile platform control system 210 can include any number of sensors 230 as needed - for example, 1, 2, 3, 4, 5, 6, or even more sensors 230. Sensors 230 can be arranged on mobile platform control system 210 in any required manner, and the specific arrangement of sensors 230 depends on the application. Sensors 230 can advantageously be lightweight and capable of high-frequency data collection for real-time object distance detection and collision avoidance. Exemplary sensors 230 applicable to mobile platform control system 210 include, but are not limited to, color or monochrome vision sensors, photoelectric sensors, thermal / infrared sensors, time-of-flight sensors, ultrasonic sensors, multispectral imaging sensors, spectrophotometers, spectrometers, thermometers, illuminometers, microphones / sonic converters, etc.

[0053] The sensor 230 can interact with one or more processors 240. The processor 240 can serve as a travel controller for guiding part or all of the operations of the mobile platform 200. When the mobile platform 200 includes an unmanned aerial vehicle, the processor 240 can serve as a flight controller. Without limitation, each processor 240 can include one or more general-purpose microprocessors (e.g., single-core or multi-core processors), application-specific integrated circuits, application-specific instruction set processors, graphics processors, physics processing units, digital signal processing units, coprocessors, network processing units, audio processing units, encryption processing units, etc. The processor 240 can be configured to execute any method described herein, including but not limited to various operations regarding object distance detection and collision avoidance. In some embodiments, the processor 240 can include dedicated hardware for handling specific operations regarding object distance detection and collision avoidance - for example, processing ultrasonic data, processing visual sensor data, determining the distance D ( Figure 1 shown in) and controlling the mobile platform 200 based on the determined distance D.

[0054] In some embodiments, the processor 240 can be located physically close to the sensor 230. In this case, the processor 240 and the sensor 230 can be configured for local communication, for example, using a hardware connector and a bus. The advantage of local communication is that it can reduce transmission latency for real-time object distance detection and collision avoidance.

[0055] In other embodiments, the processor 240 can be at least partially remote from the sensor 230. The measurement data obtained by the sensor 230 can be transmitted to the remote processor 240 for processing. The remote processor 240 can generate a control signal based on the measurement data and send the control signal back to the mobile platform 200. Various wireless communication methods can be used for remote communication between the sensor 230 and the processor 240. Suitable communication methods include, for example, radio, Wi-Fi (Wireless Fidelity), cellular, satellite, and broadcast.

[0056] As Figure 2As shown, the mobile platform control system 100 may optionally include one or more additional hardware components (not shown). Exemplary additional hardware components include, but are not limited to, a memory 250 (e.g., random access memory (RAM), static RAM, dynamic RAM, read-only memory (ROM), programmable ROM, erasable programmable ROM, electrically erasable programmable ROM, flash memory, secure digital (SD) card, etc.), and / or one or more input / output interfaces (e.g., universal serial bus (USB), digital video interface (DVI), display port, serial advanced technology attachment (SATA), IEEE 1394 interface (also known as FireWire), serial, video graphics array (VGA), super video graphics array (SVGA), small computer system interface (SCSI), high definition multimedia interface (HDMI), audio port, and / or proprietary input / output interfaces). The mobile platform control system 210 may also optionally include one or more input / output devices 260 (e.g., buttons, keyboards, keypads, trackballs, displays, and monitors).

[0057] Figure 2 Shown is that the mobile platform 200 includes a mobile mechanism 220. The mobile mechanism 220 may include one or more rotors, propellers, blades, engines, electric motors, wheels, axles, magnets, nozzles, animals, or humans. For example, the mobile platform 200 may have one or more propulsion mechanisms. In certain embodiments, the mobile platform 200 may include an unmanned aerial vehicle, and the unmanned aerial vehicle may include one or more propellers to generate thrust to propel the unmanned aerial vehicle.

[0058] In some embodiments, when controlled by the mobile platform control system 210 and / or the processor 240, the mobile mechanism 220 may allow the mobile platform 200 to perform selected movements. Exemplary movements include taking off from a surface or ground, landing on the ground, traveling at a selected speed, traveling to a selected destination, hovering in the air at a selected position and / or direction, or a combination thereof.

[0059] For example, the mobile mechanism 220 may include one or more electric motors, each driving one or more propellers. Thus, the propellers may provide lift and / or thrust to the mobile platform 200. The rotational speed of each propeller may be independently and / or jointly changed to control the lift and / or thrust generated by each propeller, thereby adjusting the spatial position, speed, and / or acceleration of the mobile platform 200. For example, the spatial position, speed, and / or acceleration of the mobile platform 200 may be adjusted with respect to up to three translational degrees and up to three rotational degrees.

[0060] Figure 3It is an exemplary top - level flowchart showing an embodiment of a method 1000 for servicing an object by a mobile platform 200. At step 1100, a distance D from the object 300 is maintained. The mobile platform 200 can maintain the distance D from the object 300. The mobile platform 200 can maintain the distance D from any selected part of the object 300. For example, the mobile platform 200 can maintain the distance D from the surface of the object 300. The surface can refer to a selected surface area and / or the entire surface of the object 300.

[0061] To maintain the distance D, the mobile platform control system 210 can monitor the distance D through one or more sensors 230. When the mobile platform control system 210 ( Figure 2 shown therein) detects that the distance D has deviated from a predetermined distance, the mobile platform control system 210 can control the movement mechanism 220 ( Figure 2 shown therein) so that the mobile platform 200 adjusts its speed and / or position to keep the distance D within the predetermined distance and / or within a predetermined distance range.

[0062] As Figure 3 shown, at step 1200, a task is performed on the object 300 during the maintenance. The mobile platform 200 can perform one or more tasks on the object 300 during the maintenance of the distance D. The task can include any function that the mobile platform 200 can perform on the object at the distance D. In some non - limiting examples, the task can include capturing still and / or video images, making measurements, and / or applying substances such as paints.

[0063] To maintain the distance D, at step 1100, the mobile platform control system 210 can determine the distance D based on the measurements of the sensor 230 and compare the distance D with a predetermined distance. The predetermined distance can be determined based on one or more factors selected from a variety of factors. For example, at step 1200, the predetermined distance can be determined and / or dynamically adjusted based on the task performed by the mobile platform 200 and / or based on one or more operating conditions of the mobile platform 200.

[0064] Exemplary operating conditions can include environmental conditions around the mobile platform 200. Exemplary environmental conditions can include wind conditions. The wind around the mobile platform 200 can cause the mobile platform 200 to move its position. When the distance D is too short, since the mobile platform 200 may not have enough time and / or distance to adjust its movement to compensate for the unexpected position movement, the mobile platform 200 may easily collide with the object 300 under the wind force.

[0065] Additionally and / or alternatively, exemplary operating conditions may include controlling the accuracy of the system and / or the movement of the mobile platform 200. For example, the sensor 230 may have a certain measurement error and / or tolerance. The mobile platform control system 210 and / or the mobile mechanism 220 may also have a certain error. Additionally and / or alternatively, any position tracking device for measuring the position of the mobile platform 200 may have a certain measurement error. Therefore, the actual value of the distance D may deviate from the predetermined value that the distance D needs to maintain. When the distance D is too short, the mobile platform 200 may collide with the object 300. Therefore, for safe operation, the mobile platform 200 may maintain the distance D to be greater than a predetermined distance lower limit. For example, when the mobile platform 200 is an unmanned aerial vehicle, the exemplary distance lower limit may be at least 60 centimeters.

[0066] Additionally and / or alternatively, the mobile platform 200 may maintain the distance D to optimally perform a certain task. For example, the mobile platform 200 may maintain the distance D to be greater than the lower limit and / or less than the upper limit to capture an image of a selected area of an object within a selected field of view size and / or resolution.

[0067] Therefore, to avoid collisions and / or successfully perform tasks, the mobile platform 200 may maintain the distance D within a predetermined distance range. In some embodiments, the predetermined distance range may be based on factors such as the operating conditions of the mobile platform 200. That is, the distance D maintained between the mobile platform 200 and the object 300 may be based on the operating conditions of the mobile platform 200.

[0068] The mobile platform 200 may monitor the distance D to adjust its movement to maintain the distance D. The distance D may include one or more distances D1, D2,..., Dn. The distance may be measured in the same and / or different directions relative to the mobile platform 200. The distance may be measured between the same and / or different positions on the mobile platform 200 and the object 300. The distances maintained by the mobile platform 200 at different positions on the object 300 may be the same and / or different. In one example, the mobile platform 200 may be at least partially surrounded by the object 300. Therefore, the mobile platform 200 may monitor the distance D in one or more directions relative to the mobile platform 200. Advantageously, the mobile platform 200 may avoid collisions in one or more directions.

[0069] See Figure 4 , the object 300 may be at a first distance D1 from the mobile platform 200 in a first direction relative to the mobile platform 200 and at a second distance D2 from the mobile platform 200 in a second direction relative to the mobile platform 200. The first direction may be the same and / or different from the second direction. The mobile platform 200 may monitor the first and second distances D1, D2 in any suitable manner, including sequentially and / or simultaneously.

[0070] Although Figure 4 two directions are shown, the mobile platform 200 can monitor any number of distances D1, D2 in any number of directions without limitation. Although Figure 4 the distances D1, D2 between the mobile platform 200 and the object 300 are shown, the distances D1, D2 can be between the mobile platform 200 and two or more objects. For example, an obstacle (not shown) can be adjacent to the object 300. When monitoring the distances D1, D2 to the object 300, the mobile platform 200 can monitor the distance between the mobile platform 200 and the obstacle to avoid collision with the obstacle.

[0071] The mobile platform 200 can monitor the distance D in any manner. See Figure 5 , for example, the sensor 230 can include an ultrasonic sensor 111 and / or a vision sensor 112 for measuring the distance D. The mobile platform 200 can monitor the distance D through the ultrasonic sensor 111 and / or the vision sensor 112.

[0072] Although Figure 5 the sensor 230 is shown to include an ultrasonic sensor 231 and / or a vision sensor 232, the mobile platform 200 can include any additional and / or alternative sensor 230 for measuring the distance D ( Figure 1 shown in). For example, the mobile platform 200 can include a time-of-flight sensor. Although Figure 5 the sensor 230 is shown to include one ultrasonic sensor 231, the sensor 230 can include multiple identical and / or different ultrasonic sensors 231. Although Figure 5 the sensor 230 is shown to include one vision sensor 232, the sensor 230 can include multiple identical and / or different vision sensors 232.

[0073] Although only separate devices are shown for illustrative purposes, two or more sensors 230 of the mobile platform control system 210 can be partially or fully integrated into a single device, and / or share one or more overlapping physical components, such as a housing, a microchip, an optical sensor, a detector, a communication port, etc. For example, in some embodiments, the ultrasonic sensor 231 can share a processor 240 with the vision sensor 232. In other embodiments, the sensors 230 can be physically separate devices for ease of replacement and modularity.

[0074] The vision sensor 232 can determine the distance D through vision sensing. Vision sensing can refer to sensing by analyzing one or more images of an object of interest. Exemplary vision sensing methods can include stereo observation (and / or stereovision). The vision sensor 232 can include two or more imaging devices, each of which captures an image of a point of interest on the object 300. The difference in the positions of the points of interest in the images can be used to provide the distance D.

[0075] See Figure 6 , which shows that the vision sensor 232 includes a first imaging device 232a and a second imaging device 232b. Figure 6 Shows a method of determining the distance D using stereo observation with reference to two imaging devices 232a and 232b. Each of the imaging devices 232a and 232b can sense the same feature point of interest 301 on the object 300, but at different spatial coordinates, as shown by the coordinate axes (Xa, Ya, Za) and (Xb, Yb, Zb). The imaging devices 232a and 232b sense the feature point of interest 301 along their respective optical axes 2320a and 2320b, respectively, resulting in two different two-dimensional images 2321a and 2321b of the feature point of interest 301. The two-dimensional images 2321a and 2321b are generally different and obtained from different positions, unless the imaging devices 232a and 232b are positioned such that their optical axes 2320a and 2320b coincide. Accordingly, in most cases, a binocular disparity d (as shown in Equation (4)) can be found between the images 2321a and 2321b, as described below with reference to Figure 7 as described.

[0076] See Figure 7 , the two-dimensional images 2321a and 2321b can be compared to determine the distance D between the pair of imaging devices 232a and 232b (or equivalently, the vision sensor 232) and the feature point of interest 301. Triangulation can be used to determine the distance D using the binocular disparity d (shown in Equation (4)). Specifically, the position of the feature point of interest 301 has an indication i, represented by its coordinates (x i , y i , z i ), and can be given as follows:

[0077]

[0078]

[0079]

[0080] where c x and c y represent the respective center coordinates of the imaging devices 232a and 232b, xi and y i represent the coordinates of the object of interest 150 in one or both of the images 2321a and 2321b, b is the baseline (in other words, the distance between the center coordinates of the imaging devices 232a and 232b), f is the focal length of each of the imaging devices 232a and 232b (it is assumed here that the imaging devices have the same focal length), i is an indication of the multiple feature points of interest 301 of the object 300, d is the binocular disparity between the images 2321a and 2321b, which is represented as follows:

[0081]

[0082] Based on the images 2321a and 2321b and using equations (3-(4), the vision sensor 232 can determine the coordinate z i . The distance D can be equal to and / or based on the magnitude of the coordinate z i .

[0083] The measurements of the vision sensor 232 can include identifying and / or selecting the feature points 301 of the object 300 in the images 2321a and 2321b. The feature points 301 can be identified based on machine vision and / or artificial intelligence methods, etc. Suitable methods include feature detection, extraction, and / or matching techniques such as RANSAC (Random Sample Consensus), Shi and Tomasi (Shi&Tomasi) corner detection, SURF blob (Speeded Up Robust Features) detection, MSER blob (Maximally Stable Extremal Regions) detection, SURF (Speeded Up Robust Features) descriptor, SIFT (Scale-Invariant Feature Transform) descriptor, FREAK (Fast Retina Keypoint) descriptor, BRISK (Binary Robust Invariant Scalable Keypoints) descriptor, HOG (Histogram of Oriented Gradients) descriptor, and so on.

[0084] When the distance D is less than and / or equal to 5 meters, the range of the distance measurement error of the exemplary vision sensor 232 can be in the range of 1 centimeter to 50 centimeters. When the object 300 has obvious texture, the vision sensor 232 can identify the feature points 301. If the object 300 does not have sufficient obvious texture, the vision sensor 232 cannot easily identify the feature points 301 and / or cannot identify a large number of feature points 301. The error of measuring the distance D may increase. Exemplary objects 300 without sufficient obvious texture can include a monochromatic wall, smooth glass, and / or the like.

[0085] See Figure 8, shows the ultrasonic sensor 231. The ultrasonic sensor 231 can emit ultrasonic waves 231c at a high frequency and evaluate the ultrasonic echo 231d, which is received after being reflected from the object 300. Based on the time difference between the transmitted ultrasonic wave 231c and the received ultrasonic echo 231d, the distance D to the object 300 can be determined. The ultrasonic sensor 231 can include at least one ultrasonic transmitter 231a and at least one ultrasonic receiver 231b. Each ultrasonic transmitter 231a and / or ultrasonic receiver 231b can be an ultrasonic transducer that converts an electrical signal into ultrasonic waves (or vice versa). Exemplary ultrasonic transducers include piezoelectric transducers and capacitive transducers. In some embodiments, the ultrasonic sensor 231 can include an array of ultrasonic transducers in a one-dimensional or two-dimensional configuration, enabling the construction of an ultrasonic depth map. The ultrasonic depth map can display the respective distances between the mobile platform 200 and one or more points of the object 300.

[0086] In certain embodiments, when the distance D is less than or equal to 5 meters, the distance measurement error range of the exemplary ultrasonic sensor 231 can be in the range of 20 centimeters to 50 centimeters. For an object 300 without sufficient distinct texture, such as a monochromatic wall and / or smooth glass, the ultrasonic sensor 231 can measure the distance D with high accuracy. However, for an object 300 that absorbs and / or attenuates ultrasonic waves, the ultrasonic sensor 231 can measure the distance D with low accuracy. Exemplary objects 300 that absorb and / or attenuate ultrasonic waves include certain types of carpets.

[0087] See Figure 9 , the sensor 230 can include multiple sensor groups 230A. Each sensor group 230A can include one or more ultrasonic sensors 231 and / or one or more visual sensors 232. Each sensor group 230A can monitor their respective distances D ( Figure 1 shown in). The sensor groups 230A can be arranged to face the same and / or different directions respectively to monitor the respective distances D in the corresponding directions.

[0088] Figure 9 Shows four sensor groups 230A for measuring the distance D in four directions 202, each direction being parallel and / or perpendicular to each other. Optionally, the sensor 230 can include a fifth and / or sixth sensor group 230A (not shown), which faces a direction 203 perpendicular to the square formed by the four sensor groups 230A. More sensor groups 230A can be added to measure the distance D in more directions.

[0089] Although Figure 9It is shown that the sensor group 230A is arranged in a rectangular and / or square shape, but the mobile platform 200 may include any number of the same and / or different sensor groups 230A that are arranged in any suitable shape and have a size and / or dimension suitable for being disposed on the mobile platform. For example, the sensor group 230A may be arranged in a triangular, linear, parallelogram, polygonal, and / or polyhedral shape.

[0090] Although Figure 9 It is shown that the sensor group 230A includes an ultrasonic sensor 231 and a vision sensor 232, but the sensor group 230A may include only one of the ultrasonic sensor 231 and the vision sensor 232. Additionally and / or alternatively, the sensor group 230A may include one or more ultrasonic sensors 231 and / or one or more vision sensors 232. The advantage of using multiple processors and / or sensor types to detect the distance D to the object 300 is that the measurement is made more accurate through redundancy and is more robust to the limitations of a particular sensor and / or sensor type.

[0091] See Figure 10 , a detailed view of an exemplary mobile platform 200 is shown. Figure 10 It is shown that the mobile platform 200 is a drone. The sensor group 230A may be mounted on a fitting plate 230B on the mobile platform 200.

[0092] When mounted on the mobile platform 200, the sensor group 230A may include sensors 230 ( Figure 9 shown in) for measuring the respective distances D between the mobile platform 200 and the object 300 (and / or other objects or obstacles). The measured distances D may be in multiple directions, depending on the orientation of the sensors 230. Exemplary directions may include front, back, left, right, up, and / or down with respect to the mobile platform 200.

[0093] See Figure 11 , an exemplary method 1110 for determining the distance D between the mobile platform 200 and the object 300 ( Figure 1 shown in) is shown, which uses at least one of an ultrasonic measurement value (also referred to herein as a first measurement value) and a vision measurement value (also referred to herein as a second measurement value). In step 1111, the ultrasonic sensor 231 ( Figure 8 shown in) may be used to measure the distance D to obtain a first measurement value. In step 1112, the vision sensor 232 ( Figure 7 shown in) may be used to measure the distance D to obtain a second measurement value. In step 1113, the distance D may be determined based on at least one of the first measurement value and the second measurement value.

[0094] The distance D can be determined based on the result of combining the first measurement and the second measurement in any predetermined manner. For example, the distance D can be determined as a simple average and / or a weighted average of the first measurement and the second measurement.

[0095] For some objects, the first measurement or the second measurement may have particularly limited accuracy. Thus, in some examples, based on the characteristics of the object 300 and / or the measurement method, the distance D can be based on one of the first measurement and the second measurement.

[0096] For example, during obtaining the second measurement by the vision sensor 232, the vision sensor 232 may not detect the texture of interest on the object 300. When the object 300 has the texture of interest, the vision sensor 232 can identify one or more feature points 301 in the captured image of the object 300. The number of feature points 301 can be equal to and / or greater than a predetermined number limit. Reaching the predetermined number limit can ensure that the second measurement has the desired accuracy. Thus, when the second measurement obtained by the vision sensor 232 does not detect the texture of interest on the object 300, the distance D can be determined based on the first measurement obtained by the ultrasonic sensor 231.

[0097] In another example, during obtaining the first measurement by the ultrasonic sensor 231, the ultrasonic receiver 231b ( Figure 8 shown in) may receive ultrasonic echoes below a predetermined intensity limit. The exemplary predetermined intensity limit can be based on a selected portion of the ultrasonic intensity emitted by the ultrasonic transmitter 231a. When the ultrasonic echoes are below the predetermined intensity limit, the emitted ultrasonic waves may be absorbed and / or attenuated by the object 300. In this case, the accuracy of the ultrasonic sensor 231 may be low. Thus, when the first measurement obtained by the ultrasonic sensor 231 receives ultrasonic echoes below the predetermined intensity limit, the distance D can be determined based on the second measurement obtained by the vision sensor 232.

[0098] In some examples, tasks can be performed on selected positions of the object 300. Refer to Figure 12 , the mobile platform 200 can move to (or close to) the selected service point 310 of the object 300. The service point 310 of the object 300 can include the target destination where the mobile platform 200 can perform tasks. In other words, the mobile platform 200 can aim to reach the service point 310 to complete the task. The service point 310 can have any spatial relationship with the object 300. For example, the service point 310 can be above the surface of the object 300, adjacent to the surface, and / or below the surface. Additionally and / or alternatively, the service point 310 can be at a position far from the object 300 (or distal).

[0099] Additionally and / or alternatively, the object 300 may move towards the service point 310 to perform a task. However, the object 300 does not necessarily have to reach the service point 310. For example, when the service point 310 is on the surface of the object 300, the mobile platform 200 may approach (or move to be close to) the service point 310 while maintaining a distance D from the service point 310 ( Figure 1 as shown in). In another example, the service point 310 may be located at a position a distance D from the surface. In this case, the mobile platform 200 may move to (and / or reach) the service point 310.

[0100] The mobile platform 200 may locate the service point 310 based on the position of the service point 310. The position of the service point 310 may include the coordinates of the service point 310 in an absolute coordinate system (or global coordinate system). The absolute coordinate system may be a system based on which the movement of the mobile platform 200 can be guided. That is, the control system 210 ( Figure 2 as shown in) may instruct the mobile platform 200 to move to and / or reach a certain position, which may be based on the absolute coordinate system. For illustrative purposes, Figure 12 it is shown that the absolute coordinate system is a Cartesian coordinate system including three coordinate axes X1, Y1, Z1, and the coordinates of the selected service point 310 are (x1, y1, z1) in the absolute coordinate system.

[0101] The object 300 may define a relative coordinate system (or local coordinate system). The relative coordinate system can be used to define the spatial relationship between the selected service point 310 and the object 300. For illustrative purposes, Figure 12 it is shown that the relative coordinate system is a Cartesian coordinate system including three coordinate axes X2, Y2, Z2, and the coordinates of the selected service point 310 are (x2, y2, z2) in the relative coordinate system. The coordinates (x2, y2, z2) of the selected service point 310 in the relative coordinate system may represent the spatial relationship between the selected service point 310 and the object 300.

[0102] A given service point 310 may be defined and / or selected based on the coordinates of the service point 310 in the relative coordinate system, regardless of what absolute coordinate system is used. For example, when the object 300 includes an aircraft, the selected service point 310 may be the tip of the right wing of the object 300. This selection may be made based on the object 300 in the relative coordinate system.

[0103] For a given service point 310, the coordinates (x2, y2, z2) in the relative coordinate system can be fixed regardless of how the object 300 is placed in the absolute coordinate system. The coordinates (x1, y1, z1) in the absolute coordinate system can vary depending on how the object 300 is placed in the absolute coordinate system. The coordinates in the relative and absolute coordinate systems can be referred to as relative coordinates and absolute coordinates, respectively.

[0104] See Figure 12 and Figure 13 , Figure 13 where the position and / or orientation of the object 300 can be different from Figure 12 the position and / or orientation of the object 300 in

[0105] Accordingly, Figure 13 the relative coordinate system (or the three coordinate axes X2, Y2, Z2) in Figure 12 can be different from the relative coordinate system in Figure 12 and Figure 13 In

[0106] Therefore, Figure 13 in

[0107] the coordinates of the selected service point 310 in the relative coordinate system can still be (x2, y2, z2). However, the coordinates of the selected service point 310 in the absolute coordinate system have changed to (x1’, y1’, z1’).

[0108] For the selected service point 310 with coordinates (x2, y2, z2) in the relative coordinate system, the coordinates (x1, y1, z1) in the absolute coordinate system can be obtained based on the conversion relationship between the absolute and relative coordinate systems. That is, for a given service point 310, the conversion relationship is the relationship between the absolute coordinates and the relative coordinates. When the conversion relationship is known, the absolute coordinates (x1, y1, z1) can be obtained based on the relative coordinates (x2, y2, z2).

[0109]

[0110] where a1, b1, c1, a2, b2, c2, a3, b3, and c3 are the conversion coefficients of the conversion relationship between the absolute and relative coordinate systems. When the object 300 is stationary, the conversion relationship can be fixed, and / or when the object 300 is moving, the conversion relationship can be changed.

[0111] In some embodiments, when the position and orientation of the object 300 in the absolute coordinate system are determined, the conversion relationship can be determined. Thus, the coordinates (x1, y1, z1) can be based on the position and orientation of the object 300 in the absolute coordinate system.

[0112] The conversion relationship can be determined in any way. For example, a method capable of providing the values of the conversion coefficients can be used to determine the conversion relationship. In some examples, the conversion relationship can be determined based on one or more control points. That is, the position and orientation of the object 300 can be based on one or more control points. A control point can refer to a point having at least partially known coordinates for calculating the conversion relationship.

[0113] See Figure 14 , the position of the object 300 can be determined by at least one control point 330a. The control point 330a can have a predetermined spatial relationship with the object 300, represented by the relative coordinates (x2a, y2a, x2a). The control point 330a can have absolute coordinates (x1a, y1a, x1a).

[0114] The position of the object 300 can be represented by the absolute coordinates (x1a, y1a, x1a). When the position of the object 300 is determined, the mobile platform 200 can locate the object 300. However, in order for the mobile platform 200 to locate the selected service point 310, the orientation of the object 300 may need to be determined.

[0115] The orientation of the object 300 can be determined by one or more control points. See Figure 15 , the control points can include the control point 330a and the control points 330b, 330c. The control points 330b, 330c can have respective predetermined spatial relationships with the object 300, represented by the relative coordinates (x2b, y2b, x2b), (x2c, y2c, x2c). The control point 330b can have absolute coordinates (x1b, y1b, x1b), (x1c, y1c, x1c).

[0116] The orientation 302 of the object 300 can include the orientations 302B, 302C. The orientations 302B, 302C can be vectors. The orientation 302B can be represented by the relative coordinates (x2b - x2a, y2b - y2a, z2b - z2a) and / or the absolute coordinates (x1b - x1a, y1b - y1a, z1b - z1a). The orientation 302C can be represented by the relative coordinates (x2c - x2a, y2c - y2a, z2c - z2a) and / or the absolute coordinates (x1c - x1a, y1c - y1a, z1c - z1a). Thus, through the control points 330a - 330c, it can be determined how the object 300 is oriented in the absolute coordinate system.

[0117] Based on the position and / or orientation of an object in an absolute coordinate system, the mobile platform 200 can locate a selected service point 310.

[0118] That is, a conversion relationship between the absolute and relative coordinate systems can be obtained. For example, through Equation (5), each of the control points 330a at (x1a, y1a, x1a) and (x2a, y2a, x2a), the control points 330b at (x1b, y1b, x1b) and (x2b, y2b, x2b), and the control points 330c at (x1c, y1c, x1c) and (x2c, y2c, x2c) can provide three equations. Nine equations can be used to solve for the transformation coefficients a1, b1, c1, a2, b2, c2, a3, b3, and c3. Thereby, the conversion relationship can be obtained.

[0119] See Figure 16 , a detailed view of exemplary control points 330a, 330b is shown in relation to the object 300. Figure 16 The object 300 is shown as an aircraft 300A. Figure 16 The control point 330a is shown as a point at a predetermined distance L1 in front of the foremost part of the aircraft 300A along the centerline 303 of the aircraft. Figure 16 The control point 330b is shown as a point at a predetermined distance L2 in front of the control point 330a along the centerline 303. L1 and L2 can be the same and / or different. Figure 16 The direction 302B is shown as the forward direction of the aircraft 300A.

[0120] As Figure 16 shown, the three-dimensional orientation of the object 300 can be at least partially known. For example, the aircraft 300A can be flush with the ground. Therefore, a two-dimensional conversion relationship can fully describe the position and / or orientation of the aircraft 300A. That is, both the absolute and relative coordinate systems can respectively include two axes (X1, Y1), (X2, Y2).

[0121] For example, the conversion relationship can have the following expression:

[0122]

[0123] Therefore, through the control points 330a, 330b, the position and orientation of the aircraft 300A in the absolute coordinate system can be determined.

[0124] Based on the position and / or orientation of an object in an absolute coordinate system, the mobile platform 200 ( Figure 12 shown in

[0125] That is to say, the conversion relationship between the absolute and relative coordinate systems can be obtained. For example, through Equation (6), the control points 330a at (x1a, y1a) and (x2a, y2a), and the control points 330b at (x1b, y1b) and (x2b, y2b) can be used to solve for the conversion coefficients a1, b1, a2, and b2. Thus, the conversion relationship can be obtained.

[0126] See Figure 17 , which shows an exemplary method 1120 for determining the position of the selected service point 310 ( Figure 12 shown in). At step 1121, the spatial relationship between the selected service point 310 and the object 300 can be obtained. For example, the spatial relationship between the selected service point 310 and the object 300 can be obtained based on the model of the object 300 (not shown). For example, the spatial relationship can be expressed as the coordinates (x2, y2, z2) of the selected service point 310 in the relative coordinate system ( Figure 12 shown in).

[0127] The model can refer to a replica of the object 300 that partially and / or fully replicates the shape, size, and / or dimensions of the object 300. Exemplary models can include computerized two-dimensional and / or three-dimensional models.

[0128] At step 1122, the position and orientation of the object 300 can be obtained. For example, the position can be obtained through the control point 330a ( Figure 14 shown in). For example, the orientation of the object 300 can be obtained through the control points 330a, 330b, and / or 330c ( Figure 15 and Figure 16 shown in).

[0129] The control points 330a - 330c can have respective predetermined spatial relationships with the object 300. The predetermined spatial relationships can be represented by relative coordinates, which can be obtained based on the model of the object 300.

[0130] Additionally and / or alternatively, the control points 330a - 330c can have their respective absolute coordinates, which can be obtained by any navigation method. Exemplary navigation methods can include the Global Positioning System (GPS). For example, a position tracking device (such as a GPS device) can be positioned at the selected control point 310 and measure the absolute coordinates. The position tracking device can be placed at the selected control point by any method and / or tool, such as a human operator, a robotic arm, and / or a lift.

[0131] To improve the accuracy of the position tracking device in measuring the absolute coordinates, the spacing between the control points and / or service points can be greater than a predetermined spacing limit. For example, L1 and / or L2 ( Figure 16As shown, it may be greater than a predetermined spacing limit. Thus, the position tracking device can avoid misidentifying two points (such as control points 330a, 330b) as a single point. An exemplary predetermined spacing limit may be greater than and / or equal to 1 meter.

[0132] In step 1123, based on the spatial relationship between the selected service point 310 and the object 300, and / or the conversion relationship between the relative and absolute coordinate systems, the position of the selected service point 310 can be determined.

[0133] Although the Cartesian coordinate system is used to illustrate the absolute and relative coordinate systems, the absolute and / or relative coordinate systems can include any same and / or different types of coordinate systems. Exemplary coordinate systems can include cylindrical coordinate systems, spherical coordinate systems, and / or geographic coordinate systems.

[0134] Although Figure 14 and Figure 15 it is shown that the control point 330a is used to determine the position and orientation of the object 300, the position and orientation of the object 300 can be respectively based on any same and / or different control points. Figure 14 and Figure 16 it is shown that the position of the object 300 can be determined by one control point. Figure 15 and Figure 16 it is shown that the orientation of the object 300 can be determined by 3 and 2 control points respectively. However, the position and / or orientation of the object 300 can be determined by any number of control points. The number of control points used to determine the position and orientation of the object 300 is the same and / or different.

[0135] See Figure 18 As shown, an exemplary region of interest 320 of the object 300 is shown. There may be one or more service points 310 on the region 320. The mobile platform 200 can move along the travel path 340 to the selected service point 310. When there are one or more service points 310 on the region 320, the travel path 340 can connect multiple selected service points 310.

[0136] The mobile platform 200 ( Figure 12 as shown) can move along the travel path 340 to pass through the selected service point 310 and perform a task on at least one selected service point 310. For example, when the mobile platform 200 includes an unmanned aerial vehicle, the mobile platform 200 can fly along the travel path 340. When the mobile platform 200 travels and / or performs a task, the travel path can be predetermined, fixed, and / or dynamically adjusted.

[0137] See Figure 19, the mobile platform 200 may include a position tracking device 280 for determining the position of the mobile platform 200. Exemplary position tracking device 280 may include a Global Positioning System (GPS) module and / or a Differential GPS (or DGPS) module (not shown).

[0138] When the position tracking device 280 includes a DGPS module, the position tracking device 280 may operate to communicate with a base station 600. The base station 600 may include a GPS receiver established at a precise known geographical location. For example, in the United States and Canada, the base station 600 may include a DGPS base station system operated by the United States Coast Guard (USCG) and the Canadian Coast Guard (CCG). The DGPS base station system may communicate with the position tracking device 280 at a long wave radio frequency between 285 kHz and 325 kHz near main waterways and ports.

[0139] The base station 600 may calculate its own position based on satellite signals and compare the calculated position with the known position. The difference between the calculated position and the known position may provide a differential correction. Thus, the base station 600 may automatically and / or in response to the position tracking device 280 broadcast the differential correction to the position tracking device 280.

[0140] The base station 600 may communicate with the position tracking device 280 by any wired and / or wireless communication method. Exemplary methods may include radio frequency (RF) communication.

[0141] When the position tracking device 280 measures the position of the position tracking device 280 and / or the mobile platform 200, the position tracking device 280 may apply the differential correction to correct the GPS data recorded by the position tracking device 280. Thus, the corrected GPS data is a DPGS signal. After the correction, the GPS data of the position of the mobile platform 200 has high accuracy. For example, if no correction is made, GPS may measure the position of the mobile platform 200 with an error of about 2 meters. When the correction is made, the position tracking device 280 may measure the position of the mobile platform 200 with an error as small as 10 centimeters. Thus, the mobile platform 200 may move to a selected position with high accuracy.

[0142] Although Figure 19 For illustrative purposes, the position tracking device 280 is shown communicating with one base station 600, but the position tracking device 280 may communicate with any predetermined number of identical and / or different base stations 600. Although Figure 19Illustrated is that the base station 600 communicates with the position tracking device 280, but the position tracking device 280 does not necessarily receive differential corrections from the base station 600. Additionally and / or alternatively, the position tracking device 280 can receive differential corrections from one or more satellites. For example, wide area DGPS (WADGPS) and / or satellite-based augmentation systems can include one or more ground base stations located at precisely surveyed points. The base station 600 can acquire data such as differential corrections. The base station 600 can measure one or more satellites, satellite signals, and / or other environmental factors that may affect the signals received by the position tracking device 280. The base station 600 can send the measurement results and / or the results of the differential corrections to one or more satellites. The satellites can broadcast correction information to the position tracking device 280.

[0143] See Figure 20 , a detailed view of the mobile platform 200 is shown. Figure 20 The mobile platform 200 is shown as an unmanned aerial vehicle. The position tracking device 280 is shown to include a DGPS module 280A mounted on the mobile platform 200.

[0144] In some embodiments, the base station 600 ( Figure 19 shown therein) can be portable and can include a GPS receiver. The base station 600 can be set at a selected location. The location where the base station 600 is established can be accurately surveyed and recorded as a known location. In some embodiments, the base station 600 can communicate with the DGPS module 280A via a 433 MHz radio frequency signal. By using the portable base station 600, an operator can advantageously operate the unmanned aerial vehicle at any location without being restricted to the location of a fixed base station 600.

[0145] See Figure 21 , the mobile platform 200 is shown capable of connecting to a payload 290. The payload 290 can perform tasks on an object 300. For example, the payload 290 can perform tasks on a selected service point 310 ( Figure 12 shown therein).

[0146] Optionally, as Figure 21 shown, the payload 290 can be connected to the mobile platform 200 via a carrier 292. The carrier 292 can allow the payload 290 to rotate relative to the mobile platform 200 about one or more axes, such as three axes, the X (or pitch) axis, the Z (or roll) axis, and the Y (or yaw) axis. Thus, even if the orientation of the mobile platform 200 remains unchanged, the payload 290 can be oriented in any of a plurality of directions.

[0147] Optionally, the carrier 292 may stabilize the payload 290 against the movement of the mobile platform 200. For example, the carrier 292 may provide a shock-absorbing function to reduce the vibration of the payload 290 caused by the movement of the mobile platform 200. Additionally and / or alternatively, the carrier 292 may include one or more sensors to measure the unwanted movement of the payload 290 due to the position drift of the mobile platform 200. Accordingly, the payload 292 may compensate for the movement by adjusting the orientation and / or position of the payload 290. Thus, even in the case where the mobile platform 200 vibrates, sways, and / or drifts, the payload 290 may remain stable and perform tasks on the selected service point 310 with higher positioning accuracy.

[0148] Although Figure 21 it is shown that the mobile platform 200 includes one payload 290 and one carrier 292, the mobile platform 200 may include any number of identical and / or different payloads 290. The payload 290 may be connected to the mobile platform 200 by any number of identical and / or different carriers 292. Some payloads 290 may be connected to the mobile platform 200 without the need for a carrier 292.

[0149] The mobile platform 200 may perform one or more identical and / or different tasks by using the payload 290. In some embodiments, the mobile platform 200 may perform each identical and / or different task by using respective payloads 290. Additionally and / or alternatively, the payload 290 may perform one or more identical and / or different tasks. Additionally and / or alternatively, multiple payloads 290 may perform tasks collaboratively.

[0150] See Figure 22 , a detailed view of the mobile platform 200 is shown. Figure 22 The mobile platform 200 is shown as an unmanned aerial vehicle. The payload 290 may include a camera 290A. The camera 290A may capture still and / or video images of the object 300.

[0151] The carrier 292 may include a gimbal 292A. In some embodiments, the gimbal 292A may include a three-axis gimbal. The mobile platform 200 may control the gimbal 292A to adjust the orientation of the camera 290A. Thus, when the mobile platform 200 hovers in the air in a fixed orientation, the camera 290A may face any direction among multiple directions to capture images of the object 300 from multiple perspectives.

[0152] See Figure 23, the mobile platform 200 can communicate with the task manager 400. The task manager 400 can include a computing device. Exemplary task managers 400 can include mobile phones, smartphones, tablets, personal computers, server computers, and / or the like. The communication between the mobile platform 200 and the task manager 400 can be wired and / or wireless. For example, both the task manager 400 and / or the mobile platform 200 can include radio frequency circuitry (not shown). The radio frequency circuitry can include, but is not limited to, an antenna, at least one amplifier, a tuner, one or more oscillators, a subscriber identity module (SIM) card, a transceiver, a coupler, an LNA (such as a low noise amplifier), a duplexer, etc. The radio frequency circuitry can communicate with other devices via a wireless communication network. The wireless communication can use any communication standard or protocol, including but not limited to, GSM (Global System for Mobile Communications), GPRS (General Packet Radio Service), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), LTE (Long Term Evolution), email, SMS (Short Message Service).

[0153] When performing a task using the payload 290, the mobile platform 200 can obtain data 204. For example, when the payload 290 includes a camera 290A ( Figure 22 as shown in), the data 204 can include an image captured by the camera 290A.

[0154] Additionally and / or alternatively, when the payload 290 includes an instrument for measuring the properties of an object 300, the data 204 can accordingly include the measurement results obtained by the instrument. Exemplary measurements can include directing light onto the surface of the object 300 to measure surface properties such as the light reflection coefficient.

[0155] The mobile platform 200 can provide the data 204 to the task manager 400. The task manager 400 can process and / or analyze the data 204. Based on the processing and / or analysis, the task manager 400 can determine the subsequent tasks to be performed by the mobile platform 200 and generate a command 206 accordingly. Exemplary commands 206 can instruct the mobile platform 200 to update the task and perform the updated task. Additionally and / or alternatively, exemplary commands 206 can instruct the mobile platform 200 to adjust the orientation of the payload 290 ( Figure 21 as shown in). Additionally and / or alternatively, exemplary commands 206 can instruct the mobile platform 200 to move to a different location, pause the task, and / or stop the task.

[0156] See Figure 24 , an exemplary method 1200 of servicing an object 300 using the mobile platform 200 is shown. In step 1210, by using the camera 290A of the mobile platform 200 ( Figure 22As shown, it can capture an image of the object 300. For example, the camera 290A can capture an image of the surface of the object 300.

[0157] In step 1220, the image can be transmitted to the task manager 400 ( Figure 23 as shown) for analysis of the object 300. For example, when the camera 290A captures an image of the surface of the object 300, the analysis of the object 300 can include an analysis of the surface of the object 300 based on the image.

[0158] For example, the task manager 400 can present and / or display the image to enable an operator to visually inspect the object 300. Additionally and / or alternatively, the task manager 400 can transmit the image to a third-party computer system for analysis, inspection, and / or storage.

[0159] Additionally and / or alternatively, the task manager 400 can perform one or more image feature recognition (and / or feature detection) methods. A feature can refer to a part of interest in an image. Exemplary features can include edges, corners, points, and / or ridges. A feature can have selected characteristics such as shape, size, dimension, color, texture, and / or similar characteristics. The selection of features depends on the type of task that the mobile platform 200 is configured to perform.

[0160] The feature recognition method can determine whether a selected type of feature exists at a point in the image. Exemplary feature recognition methods can include edge detection and / or image binarization.

[0161] The task manager 400 can generate a command 206 ( Figure 23 ) based on the analysis of the image. For example, when the task manager 400, a third party, and / or a human detects a feature of interest in the image, the task manager 400 can select the feature and send a command 206 for capturing a magnified image of the selected feature. Additionally and / or alternatively, the task manager 400 can send a command 206 to adjust the orientation of the camera 290A to capture the selected feature from a different perspective to better observe the feature.

[0162] By using the disclosed method, the automation and efficiency of servicing the object 300 can be advantageously improved. For example, to inspect the surface of the object 300, the mobile platform 200 can repeatedly return to a selected location of the object 300 for repeated inspections, even if the position and / or orientation of the object 300 are different at each inspection.

[0163] Additionally and / or alternatively, the mobile platform 200 can advantageously approach the object 300 and avoid colliding with the object 300. Thus, the mobile platform 200 can capture an image of the object 300 at a close range. Therefore, the image can present the features and / or topography of the surface of the object 300 with sufficient clarity and / or resolution. Therefore, the surface can be detected by analyzing the image.

[0164] Additionally and / or alternatively, the mobile platform 200 can transmit the image to the task manager 400 for analysis. The task manager 400 can be away from the object 300. When the mobile platform 200 captures an image, the image can be analyzed in real time. Additionally and / or alternatively, image-based feedback can be generated in real time, and the task manager 400 can generate a command 206 based on the feedback. Thus, the mobile platform 200 can operate based on the command 206 to timely obtain a new image with adjusted content and / or quality.

[0165] Additionally and / or alternatively, the implementation of real-time analysis and / or feedback requires and / or does not require manual intervention or participation. Even when a person analyzes the image, the person does not need to be present at the location of the object 300. Therefore, labor costs and time can be reduced.

[0166] See Figure 25 , which shows multiple mobile platforms 200A - 200C serving the object 300. The mobile platforms 200A - 200C can move to service points 310A - 310C respectively. The mobile platforms 200A - 200C can perform the same and / or different tasks respectively.

[0167] The mobile platforms 200A - 200C can cooperate with each other to complete the task. That is, each of the mobile platforms 200A - 200C can perform at least a part of a task to jointly perform the task. Additionally and / or alternatively, each of the mobile platforms 200A - 200C can perform one or more of multiple tasks to jointly perform multiple tasks.

[0168] For example, when the object 300 is on the ground, the mobile platform 200A can capture an image of the upper surface of the object 300. The mobile platform 200C can travel between the object 300 and the ground to capture an image of the lower surface of the object 300. Therefore, by working together cooperatively, the mobile platforms 200A and 200C can respectively capture images presenting different parts of the object 300 and complement each other.

[0169] Multiple mobile platforms 200A - 200C can perform tasks on the object 300 simultaneously and / or sequentially. The multiple mobile platforms 200A - 200C can be the same and / or different. Exemplary mobile platforms can include unmanned aerial vehicles and / or ground vehicles. Although Figure 25Three mobile platforms are shown, but any number of identical and / or different mobile platforms 200 can cooperate to perform tasks. Multiple mobile platforms 200 can serve one service point. Additionally and / or alternatively, one mobile platform 200 can serve multiple service points.

[0170] See Figure 26 , the system 100 may further include a terminal device 500 for communicating with the mobile platform 200. The terminal device 500 may include any computer system. Exemplary terminal devices 500 may include mobile phones, smart phones, tablets, personal computers, server computers, and / or the like.

[0171] During the operation of the mobile platform 200, the terminal device 500 may be away from the mobile platform 200 to serve the object 300. The terminal device 500 may provide instructions to the mobile platform 200 to implement the functions described herein.

[0172] See Figure 27 , the terminal device 500 may include a processor (and / or central processing unit, or CPU) 510. Although Figure 21 one processor 510 is shown, the terminal device 500 may include any number of identical and / or different processors 510.

[0173] The terminal device 500 may include a memory 520. The memory 520 may include high-speed random access memory (RAM). Additionally and / or alternatively, the memory 520 may include non-volatile memory, such as one or more disk storage devices, flash memory devices, and / or other non-volatile solid-state storage devices. Although Figure 27 one memory 520 is shown, the terminal device 500 may include any number of identical and / or different memories 520.

[0174] The processor 510 may run (or execute) various software programs and / or instruction sets stored in the memory 520 to perform various functions of the terminal device 500.

[0175] The terminal device 500 may include a communication module 530. The communication module 530 may operate to communicate with another computer system (such as the mobile platform control system 210 ( Figure 2 shown), and / or a third-party computer system). Exemplary communication modules 530 may include radio frequency circuits, which may wirelessly receive and / or transmit information. The terminal device 500 may communicate with the mobile platform 200 directly and / or through a routing device.

[0176] Additionally and / or alternatively, the terminal device 500 may include a display system 540. In some embodiments, the display system 540 may include a touch-sensitive display, also referred to as a "touch screen". The display system 540 may display information stored in the terminal device 500 for presentation.

[0177] See Figure 28 , the terminal device 500 may be connected to a remote controller 502. For example, the terminal device 500 may be connected to the remote controller 502 through an external communication port that supports standards such as Universal Serial Bus (USB), FireWire, and / or the like. The remote controller 502 may include one or more processors (not shown) for at least partially controlling the operation of the mobile platform 200. The remote controller 502 may communicate with the mobile platform 200 by wired and / or wireless means. For example, various wireless communication methods may be used for remote communication between the mobile platform 200 and the remote controller 502. Suitable communication methods include, for example, radio, Wi-Fi (Wireless Fidelity), cellular, satellite, and broadcast.

[0178] The terminal device 500 may communicate with the mobile platform 200 through the remote controller 502. That is, the mobile platform 200 may transmit data to the remote controller 502. The remote controller 502 may forward the data to the terminal device 500. On the other hand, the terminal device 500 may forward instructions to the remote controller 502. The remote controller 502 may transmit the instructions to the mobile platform 200. That is to say, the remote controller 502 may operate as a data exchange channel to route communication between the terminal device 500 and the mobile platform 200.

[0179] Figure 29 is an exemplary top-level flowchart showing an embodiment of a method 2000 for servicing an object 300 by the mobile platform 200. The terminal device 500 ( Figure 27 shown therein) may implement the method 2000.

[0180] In step 2100, based on the model of the object 300, one or more service points 310 may be determined. The terminal device 500 may determine the service points 310 based on the model of the object 300 ( Figure 12 shown therein).

[0181] For example, the object 300 may include an aircraft, such as a passenger plane. Therefore, the model may include three-dimensional information of the aircraft. Exemplary three-dimensional information may include the length, width, and / or height of the fuselage, the inclination angle of the front surface of the aircraft, the inclination angle of the wing root and / or wing tip, the inclination angle between the wing root and wing tip, the width of the horizontal tail and / or horizontal tail, or a combination thereof.

[0182] At step 2200, the guiding mobile platform 200 performs one or more tasks at the identified service point 310. The terminal device 500 may guide the mobile platform 200 to perform tasks. For example, the terminal device 500 may guide the mobile platform 200 to capture an image of the object 300.

[0183] Additionally and / or alternatively, the terminal device 500 may guide the mobile platform 200 to maintain a distance D from the object 300 ( Figure 1 as shown in). The terminal device 500 may determine the distance range of the maintained distance D. For example, the terminal device 500 may determine the lower limit and / or upper limit of the holding distance D.

[0184] See Figure 30 , which shows a method 2100 for identifying one or more service points 310. The terminal device 500 ( Figure 27 as shown in) may implement the method 2100.

[0185] At step 2110, a model of the object 300 may be obtained. The terminal device 500 may obtain the model from any source. For example, the manufacturer and / or builder of the object 300 may provide the model. The object 300 may be measured by a specific person / machine to generate the model. When the shape, size, and / or dimensions of the object 300 change, the model may change accordingly.

[0186] The terminal device 500 may obtain the model from another computer system and / or model library. Additionally and / or alternatively, the terminal device 500 may store the model in the memory 520 in advance and retrieve the model from the memory 520.

[0187] At step 2120, an area of interest 320 is selected based on the model. The terminal device 500 may select an area 320 on the object 300 ( Figure 18 as shown in). For example, the area 320 may be a surface area of interest of the object 300, such as a surface area with a high probability of damage during the use of the object 300.

[0188] The terminal device 500 may autonomously select the area 320 based on a predetermined selection criterion and / or select the area 320 based on a selection instruction input to the terminal device 500.

[0189] At step 2130, the distribution of the service points 310 on the area 320 is determined. The distribution of the service points 310 may refer to the density of the service points 310 and / or the spatial relationship between the service points 310 and the object 300. The terminal device 500 may determine the distribution of the service points 310 based on any criterion.

[0190] For example, when the mobile platform 200 is positioned at a distance D from the service point 310, the payload 290 can perform tasks on a sub-region of the region 320. The size and / or shape of the sub-region can be based on the distance D and / or the nature of the payload 290 ( Figure 21 as shown in). For example, the camera 290A ( Figure 22 as shown in) can capture an image of a sub-region of a certain size at the distance D.

[0191] To perform tasks on the region 320 completely, the terminal device 500 can determine the density based on the size and / or shape of the sub-region and the region 320. The terminal device 500 can determine and / or select the service point 310 according to the determined density. Additionally and / or alternatively, the terminal device 500 can determine the travel path of the mobile platform 200 to pass through the service point 310.

[0192] Although the method 2100 may include steps 2110 - 2130, one or more of 2110 - 2130 can be omitted.

[0193] Additionally and / or alternatively, the terminal device 500 can determine the position of the selected service point 310 ( Figure 12 as shown in). Based on the model, the terminal device 500 can determine the spatial relationship between the selected service point 310 and the object 300.

[0194] Additionally and / or alternatively, based on the model, the terminal device 500 can determine the spatial relationship between the selected control points (such as the control points 330a - 330c shown in Figure 15 and Figure 16 ) and the object 300.

[0195] To determine the absolute coordinates of the control point, a position tracking device can be positioned at the control point to measure the absolute coordinates of the control point 330. The position tracking device can send the measurement result to the terminal device 500.

[0196] In some embodiments, the position tracking device 280 ( Figure 19 as shown in) on the mobile platform 200 can be used to determine the absolute coordinates of the control point 330. In one example, the DGPS module 280A ( Figure 19 as shown in) on the mobile platform 200 can be used to determine the absolute coordinates of the control point 330. In other words, the mobile platform 200 can be positioned at the selected control point by a person and / or a machine. The DGPS module 280A can measure the GPS position of the mobile platform 200, thereby obtaining the position of the control point in the absolute coordinate system. When the position and / or orientation of the object 300 change, the measurement of the absolute coordinates of the control point can be performed, so that the position of the service point 310 can be updated.

[0197] Additionally and / or alternatively, when the mobile platform 200 includes a camera 290A ( Figure 22 shown in), the terminal device 500 may direct the mobile platform 200 to capture an image of the object 300 from one or more perspectives. The mobile platform 200 may direct the carrier 292 to adjust the orientation of the camera 290A, thereby adjusting the angle of the captured image.

[0198] Additionally and / or alternatively, the mobile platform 200 may send the image to the terminal device 500 for analysis. The terminal device 500 may analyze the image in real time and / or send the image to a third party for analysis and / or storage. Optionally, the display system 540 ( Figure 27 shown in) may display the image for inspection.

[0199] Various embodiments also disclose a computer program product including instructions for operating the mobile platform 100. The program / software may be stored in a (non-transitory) computer-readable storage medium, including, for example: read-only memory (ROM), random access memory (RAM), memory, registers, computer hard drives, removable disks, CD-ROMs, optical disks, floppy disks, magnetic disks, etc. The program / software may include encoded instructions to direct one or more processors on a computer device to perform the methods according to the various disclosed embodiments. For example, the program / software may be executed on a computer and / or mobile device in the form of an application, an app, and / or application software.

[0200] Embodiments of the present invention may have various modifications and alternative forms, and specific examples thereof have been shown by way of example in the drawings and described in detail herein. However, it should be understood that the embodiments of the present invention are not limited to the specific forms or methods disclosed, and on the contrary, the embodiments of the present invention are intended to cover all modifications, equivalent forms, and alternative manners.

Claims

1. A method for serving an object through a mobile platform, the method comprising: Selecting a region of interest of the object on a model containing the object, the region of interest including at least one sub-region, wherein the selection is autonomously selected in response to a predetermined selection criterion and / or selected in response to a received selection instruction; Determining one or more service points associated with the object based on the at least one sub-region of the region of interest of the object; Maintaining a distance between the mobile platform and the object, wherein the distance is determined and / or dynamically adjusted based on a task performed by the mobile platform and / or one or more operating conditions of the mobile platform; and During the maintaining, controlling the mobile platform to perform a task on the at least one sub-region at the one or more service points; Wherein the size and / or shape of the sub-region is determined based on at least one of the distance and the nature of a payload performing a task on the mobile platform.

2. The method according to claim 1, wherein, There are multiple mobile platforms, and different ones of the mobile platforms respectively move to different ones of the service points to jointly serve the object.

3. The method according to claim 2, wherein the task includes image capture, and different ones of the mobile platforms respectively move to different ones of the service points to capture images of different parts of the object, and the angle for capturing the images is adjustable.

4. The method according to claim 1 or 2, wherein The payload includes a camera.

5. The method according to claim 4, wherein, The camera captures images of the object from perspectives in multiple different directions.

6. The method according to claim 1 or 2, wherein The operating conditions include at least one of the following: Environmental conditions around the mobile platform, movement of the mobile platform, and accuracy of a control system of the mobile platform.

7. The method according to claim 1 or 2, further comprising: Determining a spatial relationship between the service point and the object based on the size and / or shape of the sub-region.

8. The method according to claim 1, further comprising: Controlling the mobile platform to move towards the service point of the object.

9. The method according to claim 8, wherein, Controlling the mobile platform to move towards the service point according to the position of the service point, and the determination of the position of the service point includes at least one of the following: Determining the position of the service point based on the position of the object, and determining the position of the object through one or more control points each having a predetermined spatial relationship with the object; Determining the position of the service point based on the orientation of the object, and determining the orientation of the object through one or more control points each having a predetermined spatial relationship with the object; Determining the position of the object based on one or more control points each having predetermined relative coordinates; Determining the orientation of the object based on one or more control points each having predetermined relative coordinates.

10. The method according to claim 9, wherein, It further includes at least one of the following: Determining the position of the object based on a first control point among the control points, and determining the orientation of the object based on the first control point and a second control point among the control points; Selecting control points with a spacing therebetween greater than a predetermined spacing limit; Obtaining the coordinates of a control point among the control points in an absolute coordinate system by using a position tracking device on the mobile platform, wherein the mobile platform is located at the control point.

11. The method according to claim 1 further comprises: Determining a travel path of the mobile platform to pass through one or more of the service points.

12. The method according to claim 11 further comprises: Pre-determining, fixing and / or dynamically adjusting the travel path while the mobile platform is moving and / or performing a task.

13. The method according to claim 1 or 2, wherein The maintaining includes at least one of the following: Maintaining a distance between the mobile platform and the object surface; Maintaining the distance within a predetermined distance range; Maintaining the distance based on the operating conditions of the mobile platform.

14. The method according to claim 1 or 2 further comprises monitoring the distance in one or more directions relative to the mobile platform, and / or Monitoring the distance by at least one of ultrasonic sensing and visual sensing.

15. The method according to claim 14, wherein The monitoring the distance includes: Measuring the distance by the ultrasonic sensing to obtain a first measurement value; Measuring the distance by the visual sensing to obtain a second measurement value; and Determining the distance based on at least one of the first measurement value and the second measurement value.

16. The method according to claim 15, wherein, The determining includes: When the second measurement value does not detect an interesting texture on the object, determining the distance based on the first measurement value; and / or When the intensity of the ultrasonic echo received by the first measurement value is lower than a predetermined intensity limit, determining the distance based on the second measurement value.

17. The method according to claim 1 or 2, wherein The performing includes at least one of the following: Performing one or more different tasks; Capturing an image of the object by using a camera of the mobile platform and transmitting the image to a computer associated with the mobile platform for analysis of the object; Cooperating with one or more other mobile platforms to perform the task.

18. The method according to claim 17, wherein The performing includes at least one of the following: Performing each different task by using a corresponding payload on the mobile platform; Capturing a magnified image of a feature of the object based on the analysis of the object; Capturing an image of the object, the image supplementing one or more images captured by the other mobile platform.

19. A system for servicing an object, comprising: A travel controller for operating on a mobile platform and guiding the mobile platform to: Perform the method according to any one of claims 1 to 18.

20. A mobile platform for servicing an object, comprising: A travel controller for guiding the mobile platform to: Perform the method according to any one of claims 1 to 18; And One or more propellers for moving the mobile platform based on instructions from the travel controller.

21. The mobile platform according to claim 20, wherein, The mobile platform is an unmanned aerial vehicle.

22. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs including instructions which, when executed by a travel controller, instruct the travel controller to service the object according to the method according to any one of claims 1 - 18.

Images