Method and system for servicing an object

Abstract

A system for serving objects and methods of making and using the same. The mobile platform may perform one or more tasks on the object while maintaining a distance from the object. Accordingly, the mobile platform may avoid collisions with the object during service. If the position or direction of the object changes during the service, the service point of the object can be updated as needed. The service may include performing one or more tasks on the object, and the mobile platform may advantageously include one or more payloads to perform one or more selected tasks. Thus, service objects may advantageously be automated and require less human intervention.

Description

[0001] This application is a divisional application of the invention patent application with the application date of December 15, 2015, application number 202110572742.1, and invention name “Method and system for serving objects”.

[0002] Copyright Notice

[0003] Portions of the disclosure of this patent document contain material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent files or records, but otherwise reserves all copyright rights whatsoever. Technical Field

[0004] Embodiments of the present invention relate to mobile platform technology, and more particularly, but not exclusively, to a method and system for object services. Background Art

[0005] Object services, such as surface inspection, are crucial for routine maintenance of various objects, including aircraft, ships, and buildings. For example, the surface paint of aircraft can be easily damaged by environmental factors such as salt spray, sand and dust, lightning strikes, and foreign object impact. This damage can result in defects such as paint peeling or cracking. These defects can be located anywhere on the aircraft's fuselage.

[0006] Traditionally, defect inspections rely primarily on close-up visual inspections by humans using ladders. This often requires a human visual inspection across the entire fuselage of an aircraft, consuming significant manpower and time.

[0007] In view of the foregoing, there is a need for methods and systems for detecting objects that overcome the shortcomings of existing methods and systems. Summary of the Invention

[0008] The present disclosure relates to a system for serving objects and methods of making and using the same.

[0009] According to the first aspect disclosed herein, movement is performed toward a selected service point of the object, wherein the position of the selected service point is obtained based on: a spatial relationship between the selected service point and the object, and a position and orientation of the object; maintaining a distance between the mobile platform and the object; and performing a task on the object during the maintenance.

[0010] According to a second aspect disclosed herein, a method for servicing an object via a mobile platform comprises: maintaining a distance between the mobile platform and the object, wherein the distance is determined and / or dynamically adjusted based on one or more operating conditions of the mobile platform; and performing a task on the object during the maintenance period.

[0011] According to a third aspect disclosed herein, a method for serving an object via a mobile platform comprises: moving to a selected service point of the object, wherein a path of movement is determined based on a position of the selected service point; maintaining a distance between the mobile platform and the object; and performing a task on the object during the maintenance period.

[0012] According to a fourth aspect disclosed herein, a system for serving an object is provided, comprising: a travel controller for operating on a mobile platform and guiding the mobile platform to serve at least one object as described above.

[0013] According to a fifth aspect disclosed herein, there is provided an unmanned aerial vehicle (UAV) for servicing objects, comprising: a travel controller for guiding the UAV to serve at least one object as described above; and one or more propellers for moving the UAV based on instructions from the travel controller.

[0014] According to the sixth aspect disclosed herein, a non-transitory computer-readable storage medium storing one or more programs is provided, wherein the one or more programs include instructions that, when executed by a travel controller, instruct the travel controller to serve at least one object as described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is an example diagram illustrating one embodiment of a mobile platform shown in relation to an object.

[0016] Figure 2 It shows the installation Figure 1 An example diagram of an embodiment of a mobile platform control system on a mobile platform in FIG.

[0017] Figure 3 It is shown by Figure 1 An exemplary top-level flowchart of an embodiment of a method for a mobile platform to serve an object in FIG.

[0018] Figure 4 It shows Figure 1 An example diagram of an alternative embodiment of a mobile platform in which the mobile platform monitors the distance to an object from one or more directions.

[0019] Figure 5 It shows Figure 4 An example diagram of an alternative embodiment of a mobile platform in which the mobile platform includes at least one of an ultrasonic sensor and a visual sensor.

[0020] Figure 6 is a diagram showing the use of stereoscopic observation Figure 4 An example diagram of distance measurement in .

[0021] Figure 7 The use of triangulation is shown Figure 4 An example diagram of distance measurement in .

[0022] Figure 8 is shown using ultrasonic sensing to Figure 4 An example diagram of distance measurement in .

[0023] Figure 9 It shows Figure 5 An exemplary detailed view of an embodiment of an ultrasonic sensor and a vision sensor in FIG.

[0024] Figure 10 It shows Figure 4 An exemplary detail diagram of another alternative embodiment of a mobile platform in which the mobile platform comprises an unmanned aerial vehicle (UAV).

[0025] Figure 11 The method is shown to determine the Figure 4 An exemplary flow chart of distances in .

[0026] Figure 12 It shows Figure 1 An example diagram of an alternative embodiment of a mobile platform in which the mobile platform moves toward a selected service point of an object.

[0027] Figure 13 It is shown that by changing Figure 12 An example diagram of changing the position or orientation of an object in the image to change the position of the selected service point.

[0028] Figure 14 It shows that the control points are determined by Figure 12 An example diagram of the position of objects in .

[0029] Figure 15 is shown by one or more control points Figure 12 An example diagram of the orientation of objects in .

[0030] Figure 16 Is shown to determine Figure 12 An exemplary detailed view of the position and orientation of an object in FIG, wherein the position and orientation of the object are determined two-dimensionally.

[0031] Figure 17 Is shown to determine Figure 12 An exemplary flow chart of the location of a selected service point in FIG.

[0032] Figure 18 It shows Figure 12 An example diagram of an object in which a mobile platform is provided with a travel path.

[0033] Figure 19 It shows Figure 1 An exemplary diagram of another alternative embodiment of a mobile platform in which the mobile platform includes a differential global positioning system (DGPS) module.

[0034] Figure 20 It shows Figure 19 An exemplary detailed view of an alternative embodiment of a mobile platform in which the mobile platform comprises an unmanned aerial vehicle.

[0035] Figure 21 It shows Figure 1 An example diagram of an alternative embodiment of a mobile platform in which the mobile platform includes a payload.

[0036] Figure 22 It shows Figure 21 An exemplary detailed view of an alternative embodiment of a mobile platform in which the mobile platform comprises an unmanned aerial vehicle.

[0037] Figure 23 It shows Figure 1 An exemplary diagram of another alternative embodiment of a mobile platform transmitting data to a computer.

[0038] Figure 24 It shows Figure 3 An exemplary flow chart of an alternative embodiment of the method in which a mobile platform captures an image of an object.

[0039] Figure 25 It shows Figure 1 An example diagram of another alternative embodiment of a mobile platform in which the mobile platform operates in cooperation with one or more other mobile platforms.

[0040] Figure 26 Is shown for service Figure 1 An example diagram of an embodiment of a system for an object in FIG. 1 , wherein the system includes a terminal device associated with a mobile platform.

[0041] Figure 27 It shows Figure 26 An example diagram of an alternative embodiment of a terminal device in .

[0042] Figure 28 It shows Figure 26 FIG. 5 is an exemplary diagram of another alternative embodiment of a terminal device in which the terminal device is coupled to a remote control.

[0043] Figure 29 It shows the use of Figure 26An exemplary flow chart of an embodiment of a method for a terminal device in operating a mobile platform.

[0044] Figure 30 It shows Figure 29 An exemplary flow chart of an alternative embodiment of the method in which the terminal device obtains a model of the object.

[0045] It should be noted that the drawings are not drawn to scale, and for illustrative purposes, elements with similar structure or function are generally represented by the same reference numerals throughout the figures. It should also be noted that the drawings are only used to assist in describing the preferred embodiments. The drawings are not intended to illustrate all aspects of the described embodiments and are not intended to limit the scope of the present invention. DETAILED DESCRIPTION

[0046] Because existing methods and systems are unable to effectively and automatically detect objects, it can be demonstrated that methods and systems that improve the efficiency of detecting objects and automating the detection process are needed and can provide a basis for a wide range of applications. Such applications can include inspecting aircraft, ships, and buildings. Not necessarily limited to inspection, the methods and systems can be used in any industrial operation that involves performing a function at a selected location on an object. According to the embodiments disclosed herein, Figure 1 The illustrated system 100 can achieve this result.

[0047] System 100 can be used to service object 300. Object 300 can include any shaped structure. Exemplary structures include aircraft, ships, space shuttles, buildings, bridges, mountains, caves, tunnels, transportation pipelines, bodies of water, and / or the like. The shape can be fixed and / or change over time. Object 300 can change shape spontaneously and / or due to external forces, including human and / or human intervention. Object 300 can be stationary and / or moving. For example, object 300 can undergo rotational and / or translational motion.

[0048] System 100 can service object 300, for example, by performing tasks (and / or taking actions related to object 300) on (and / or involving) object 300, including any tasks that benefit object 300. Exemplary actions can include inspecting, repairing, providing maintenance for object 300, and / or acquiring data about object 300.

[0049] Figure 1System 100 is shown including a mobile platform 200. Examples of mobile platform 200 may include, but are not limited to, bicycles, cars, trucks, ships, boats, trains, helicopters, airplanes, and various combinations thereof. In some embodiments, mobile platform 200 may include an unmanned aerial vehicle (UAV). Colloquially referred to as a "drone," a UAV is an aircraft that does not have a human pilot (or operator) on board, and its flight may be controlled automatically or by a remote pilot (or sometimes both). Currently, UAVs are increasingly being used in civilian applications including various aerial operations (e.g., data collection or delivery). The systems and methods of the present invention are applicable to various types of UAVs, including but not limited to quadcopters (also known as quadrotors or quadrotors), single-rotor, dual-rotor, tri-rotor, hexacopter, and octorotor rotary-wing UAVs, fixed-wing UAVs, and rotary-fixed-wing hybrid UAVs.

[0050] like Figure 1 As shown, mobile platform 200 can operate to maintain a distance D from object 300. Distance D can refer to the spacing between mobile platform 200 and object 300. Distance D can include, for example, a predetermined distance and / or a predetermined distance range. Advantages of maintaining distance D can include, for example, preventing mobile platform 200 from colliding with object 300.

[0051] although Figure 1 One distance D is shown, but mobile platform 200 may be located at one or more of the same and / or different distances D from object 300 .

[0052] See also Figure 2 , shows a mobile platform 200 including a mobile platform control system 210. The mobile platform control system 210 may include one or more sensors 230. The mobile platform control system 210 may include any number of sensors 230 as desired—e.g., 1, 2, 3, 4, 5, 6, or even more sensors 230. The sensors 230 may be arranged on the mobile platform control system 210 in any desired manner, with the specific arrangement of the sensors 230 depending on the application. The sensors 230 may advantageously be lightweight and capable of high-frequency data collection to facilitate real-time object distance detection and collision avoidance. Exemplary sensors 230 suitable for use in the mobile platform control system 210 include, but are not limited to, color or monochrome vision sensors, photoelectric sensors, thermal / infrared sensors, time-of-flight sensors, ultrasonic sensors, multispectral imaging sensors, spectrophotometers, spectrometers, thermometers, illuminometers, microphones / acoustic transducers, and the like.

[0053] The sensor 230 can interact with one or more processors 240. The processor 240 can act as a range controller for directing some or all operations of the mobile platform 200. When the mobile platform 200 includes an unmanned aerial vehicle, the processor 240 can act as a flight controller. Without limitation, each processor 240 can include one or more general-purpose microprocessors (e.g., single-core or multi-core processors), application-specific integrated circuits, application-specific instruction set processors, graphics processors, physical operation processors, digital signal processing units, coprocessors, network processing units, audio processing units, encryption processing units, etc. The processor 240 can be configured to perform any of the methods described herein, including but not limited to various operations related to object distance detection and collision avoidance. In some embodiments, the processor 240 may include specialized hardware for handling specific operations related to object distance detection and collision avoidance - for example, processing ultrasonic data, processing visual sensor data, determining the distance D ( Figure 1 ) and controlling the mobile platform 200 based on the determined distance D.

[0054] In some embodiments, processor 240 may be located physically close to sensor 230. In this case, processor 240 and sensor 230 may be configured for local communication, for example, using a hardware connector and bus. The advantage of local communication is that it can reduce transmission delays to facilitate real-time object distance detection and collision avoidance.

[0055] In other embodiments, the processor 240 can be at least partially remote from the sensor 230. Measurement data obtained by the sensor 230 can be transmitted to the remote processor 240 for processing. The remote processor 240 can generate control signals based on the measurement data and send the control signals back to the mobile platform 200. Various wireless communication methods can be used for remote communication between the sensor 230 and the processor 240. Suitable communication methods include, for example, radio, wireless fidelity (Wi-Fi), cellular, satellite, and broadcast.

[0056] like Figure 2As shown, the mobile platform control system 100 may include one or more additional hardware components (not shown) as needed. Exemplary additional hardware components include, but are not limited to, memory 250 (e.g., random access memory (RAM), static RAM, dynamic RAM, read-only memory (ROM), programmable ROM, erasable programmable ROM, electrically erasable programmable ROM, flash memory, secure digital (SD) card, etc.), and / or one or more input / output interfaces (e.g., universal serial bus (USB), digital video interface (DVI), display port, serial ATA (SATA), IEEE 1394 interface (also known as FireWire), serial, video graphics array (VGA), super video graphics array (SVGA), small computer system interface (SCSI), high-definition multimedia interface (HDMI), audio port, and / or proprietary input / output interface). The mobile platform control system 210 may also include one or more input / output devices 260 (e.g., buttons, keyboard, keypad, trackball, display, and monitor) as needed.

[0057] Figure 2 Mobile platform 200 is shown as including a motion mechanism 220. Mobility mechanism 220 may include one or more rotors, propellers, blades, engines, motors, wheels, shafts, magnets, nozzles, animals, or humans. For example, mobile platform 200 may have one or more propulsion mechanisms. In some embodiments, mobile platform 200 may include an unmanned aerial vehicle (UAV), and the UAV may include one or more propellers to generate thrust, thereby propelling the UAV.

[0058] In some embodiments, the movement mechanism 220, when controlled by the mobile platform control system 210 and / or the processor 240, can allow the mobile platform 200 to perform selected movements. Exemplary movements include taking off from a surface or the ground, landing on the ground, traveling at a selected speed, traveling to a selected destination, hovering in the air at a selected position and / or direction, or a combination thereof.

[0059] For example, the movement mechanism 220 may include one or more electric motors, each of which drives one or more propellers. Thus, the propellers can provide lift and / or thrust to the mobile platform 200. The rotational speed of each propeller can be varied independently and / or collectively to control the lift and / or thrust generated by each propeller, thereby adjusting the spatial position, velocity, and / or acceleration of the mobile platform 200. For example, the spatial position, velocity, and / or acceleration of the mobile platform 200 can be adjusted with respect to up to three degrees of translation and up to three degrees of rotation.

[0060] Figure 31 is an exemplary top-level flow chart illustrating one embodiment of a method 1000 for servicing an object via a mobile platform 200. At step 1100, a distance D is maintained from an object 300. The mobile platform 200 may maintain the distance D from the object 300. The mobile platform 200 may maintain the distance D from any selected portion of the object 300. For example, the mobile platform 200 may maintain the distance D from the surface of the object 300. The surface may refer to a selected surface area and / or the entire surface of the object 300.

[0061] To maintain the distance D, the mobile platform control system 210 may monitor the distance D via one or more sensors 230. Figure 2 When the mobile platform control system 210 detects that the distance D has deviated from the predetermined distance, the mobile mechanism 220 ( Figure 2 ), so that the mobile platform 200 adjusts the speed and / or position to maintain the distance D within a predetermined distance and / or a predetermined distance range.

[0062] like Figure 3 As shown, at step 1200, a task is performed on object 300 during the maintenance period. Mobile platform 200 may perform one or more tasks on object 300 during the maintenance period at distance D. A task may include any function that mobile platform 200 may perform on an object at distance D. In certain non-limiting examples, a task may include capturing still and / or video images, performing measurements, and / or applying a substance such as paint.

[0063] To maintain distance D, at step 1100, mobile platform control system 210 may determine distance D based on measurements from sensor 230 and compare distance D to a predetermined distance. The predetermined distance may be determined based on one or more factors selected from a variety of factors. For example, at step 1200, the predetermined distance may be determined and / or dynamically adjusted based on the task being performed by mobile platform 200 and / or based on one or more operating conditions of mobile platform 200.

[0064] Exemplary operating conditions may include environmental conditions surrounding mobile platform 200. Exemplary environmental conditions may include wind conditions. Wind surrounding mobile platform 200 may cause mobile platform 200 to shift position. When distance D is too short, mobile platform 200 may easily collide with object 300 due to wind forces because mobile platform 200 may not have sufficient time and / or distance to adjust its motion to compensate for the unexpected shift in position.

[0065] Additionally and / or alternatively, exemplary operating conditions may include the accuracy of the control system and / or the movement of the mobile platform 200. For example, the sensor 230 may have a certain measurement error and / or tolerance. The mobile platform control system 210 and / or the mobile mechanism 220 may also have a certain error. Additionally and / or alternatively, any position tracking device used to measure the position of the mobile platform 200 may have a certain measurement error. Therefore, the actual value of the distance D may deviate from the predetermined value that the distance D needs to maintain. When the distance D is too short, the mobile platform 200 may collide with the object 300. Therefore, for safe operation, the mobile platform 200 may maintain the distance D to be greater than a predetermined distance lower limit. For example, when the mobile platform 200 is an unmanned aerial vehicle, the exemplary distance lower limit may be at least 60 centimeters.

[0066] Additionally and / or alternatively, the mobile platform 200 can maintain the distance D to optimally perform a task. For example, the mobile platform 200 can maintain the distance D above a lower limit and / or below an upper limit to capture an image of a selected area of ​​the object within a selected field of view size and / or resolution.

[0067] Therefore, to avoid collision and / or successfully perform a mission, mobile platform 200 may maintain distance D within a predetermined distance range. In some embodiments, the predetermined distance range may be based on factors such as the operating conditions of mobile platform 200. That is, the distance D maintained between mobile platform 200 and object 300 may be based on the operating conditions of mobile platform 200.

[0068] Mobile platform 200 can monitor distance D and adjust its movement to maintain distance D. Distance D can include one or more distances D1, D2, ..., Dn. Distances can be measured in the same and / or different directions relative to mobile platform 200. Distances can be measured between mobile platform 200 and the same and / or different locations on object 300. The distances maintained by mobile platform 200 at different locations on object 300 can be the same and / or different. In one example, mobile platform 200 can be at least partially surrounded by object 300. Therefore, mobile platform 200 can monitor distance D in one or more directions relative to mobile platform 200. Advantageously, mobile platform 200 can avoid collisions in one or more directions.

[0069] See also Figure 4 , object 300 may be at a first distance D1 from mobile platform 200 in a first direction relative to mobile platform 200, and at a second distance D2 from mobile platform 200 in a second direction relative to mobile platform 200. The first direction may be the same as and / or different from the second direction. Mobile platform 200 may monitor the first and second distances D1 and D2 in any suitable manner, including sequentially and / or simultaneously.

[0070] although Figure 4 Two directions are shown, but the mobile platform 200 can monitor any number of distances D1, D2 in any number of directions without limitation. Figure 4 Although distances D1 and D2 between mobile platform 200 and object 300 are shown, distances D1 and D2 may be between mobile platform 200 and two or more objects. For example, an obstacle (not shown) may be adjacent to object 300. When monitoring distances D1 and D2 from object 300, mobile platform 200 may monitor the distance between mobile platform 200 and the obstacle to avoid collision with the obstacle.

[0071] The mobile platform 200 can monitor the distance D in any manner. Figure 5 For example, the sensor 230 may include an ultrasonic sensor 111 and / or a visual sensor 112 for measuring the distance D. The mobile platform 200 may monitor the distance D through the ultrasonic sensor 111 and / or the visual sensor 112.

[0072] although Figure 5 The sensor 230 is shown to include an ultrasonic sensor 231 and / or a visual sensor 232, but the mobile platform 200 may include any additional and / or alternative sensors for measuring the distance D ( Figure 1 For example, the mobile platform 200 may include a time-of-flight sensor. Figure 5 The sensor 230 is shown to include one ultrasonic sensor 231, but the sensor 230 may include a plurality of the same and / or different ultrasonic sensors 231. Figure 5 Sensor 230 is shown as including one vision sensor 232 , but sensor 230 may include a plurality of the same and / or different vision sensors 232 .

[0073] Although shown as separate devices for illustrative purposes only, two or more sensors 230 of the mobile platform control system 210 may be partially or fully integrated into a single device and / or share one or more overlapping physical components, such as housings, microchips, light sensors, detectors, communication ports, etc. For example, in some embodiments, the ultrasonic sensor 231 may share the processor 240 with the visual sensor 232. In other embodiments, the sensors 230 may be physically separate devices to facilitate replacement and modularity.

[0074] Visual sensor 232 can determine distance D through visual sensing. Visual sensing can refer to sensing by analyzing one or more images of the object of interest. Exemplary visual sensing methods may include stereoscopy (and / or stereo vision). Visual sensor 232 may include two or more imaging devices, each capturing an image of a point of interest on object 300. Differences in the positions of the points of interest in the images can be used to provide distance D.

[0075] See also Figure 6 , it is shown that the visual sensor 232 includes a first imaging device 232a and a second imaging device 232b. Figure 6 A method for determining a distance D using stereoscopic observation with reference to two imaging devices 232a, 232b is shown. Each imaging device 232a and 232b can perceive the same feature point of interest 301 on the object 300, but with different spatial coordinates, as shown by the coordinate axes (Xa, Ya, Za) and (Xb, Yb, Zb). The imaging devices 232a and 232b perceive the feature point of interest 301 along their respective optical axes 2320a and 2320b, respectively, thereby obtaining two different two-dimensional images 2321a and 2321b of the feature point of interest 301. The two-dimensional images 2321a and 2321b are generally different and obtained from different positions unless the imaging devices 232a and 232b are positioned so that their optical axes 2320a and 2320b coincide. Accordingly, in most cases, the binocular disparity d (as in equation (4)) can be found between the images 2321a and 2321b, as shown below with reference to Figure 7 described.

[0076] See also Figure 7 , the two-dimensional images 2321a and 2321b can be compared to determine the distance D between the paired imaging devices 232a and 232b (or equivalently, the visual sensor 232) and the feature point of interest 301. Triangulation can be used to determine the distance D using the binocular disparity d (shown in Equation (4)). Specifically, the location of the feature point of interest 301 is indicated by its coordinates (x i ,y i , z i ) can be given as follows:

[0077] Equation (1)

[0078] Equation (2)

[0079] Equation (3)

[0080] Among them, c x and c yrepresents the center coordinates of the imaging devices 232a and 232b, respectively, x i and y i represents the coordinates of the object of interest 150 in one or both images 2321a and 2321b, b is the baseline (in other words, the distance between the center coordinates of the imaging devices 232a and 232b), f is the focal length of each of the imaging devices 232a and 232b (assuming herein that the imaging devices have the same focal length), i is an indication of a plurality of feature points of interest 301 of the object 300, and d is the binocular disparity between the images 2321a and 2321b, which are represented herein as follows:

[0081] Equation (4)

[0082] Based on images 2321a and 2321b and using equations (3-(4), the vision sensor 232 can determine the coordinate z i The distance D can be equal to and / or based on the coordinate z i size.

[0083] Measurements by vision sensor 232 may include identifying and / or selecting feature points 301 of object 300 in images 2321a and 2321b. Feature points 301 may be identified based on machine vision and / or artificial intelligence methods, among others. Suitable methods include feature detection, extraction, and / or matching techniques, such as RANSAC (Random Sample Consensus), Shi and Tomasi (Shi & Tomasi) corner detection, SURF (Superior Robust Features) detection, MSER (Maximally Stable Extremal Regions) detection, SURF (Superior Robust Features) descriptors, SIFT (Scale-Invariant Feature Transform) descriptors, FREAK (Fast Retinal Keypoints) descriptors, BRISK (Binary Robust Invariant Scalable Keypoints) descriptors, HOG (Histogram of Oriented Gradients) descriptors, and the like.

[0084] When distance D is less than and / or equal to 5 meters, the distance measurement error range of exemplary vision sensor 232 may be within the range of 1 centimeter to 50 centimeters. When object 300 has a noticeable texture, vision sensor 232 may be able to identify feature points 301. If object 300 lacks sufficient texture, vision sensor 232 may not easily identify feature points 301 and / or may not be able to identify a large number of feature points 301. The error in measuring distance D may increase. Exemplary objects 300 lacking sufficient texture may include solid-colored walls, smooth glass, and / or the like.

[0085] See also Figure 8, shows an ultrasonic sensor 231. The ultrasonic sensor 231 can transmit ultrasonic waves 231c at a high frequency and evaluate ultrasonic echoes 231d received after being reflected from an object 300. Based on the time difference between transmitting the ultrasonic wave 231c and receiving the ultrasonic echo 231d, the distance D to the object 300 can be determined. The ultrasonic sensor 231 may include at least one ultrasonic transmitter 231a and at least one ultrasonic receiver 231b. Each ultrasonic transmitter 231a and / or ultrasonic receiver 231b may be an ultrasonic transducer that converts electrical signals into ultrasonic waves (or vice versa). Exemplary ultrasonic transducers include piezoelectric transducers and capacitive transducers. In some embodiments, the ultrasonic sensor 231 may include an array of ultrasonic transducers in a one-dimensional or two-dimensional configuration, thereby enabling the construction of an ultrasonic depth map. The ultrasonic depth map may display the distances between one or more points on the mobile platform 200 and the object 300.

[0086] In some embodiments, when distance D is less than or equal to 5 meters, the distance measurement error range of the exemplary ultrasonic sensor 231 may be in the range of 20 centimeters to 50 centimeters. For objects 300 without sufficient visible texture, such as solid-color walls and / or smooth glass, the ultrasonic sensor 231 may measure distance D with high accuracy. However, for objects 300 that absorb and / or attenuate ultrasonic waves, the ultrasonic sensor 231 may measure distance D with low accuracy. Exemplary objects 300 that absorb and / or attenuate ultrasonic waves include certain types of carpet.

[0087] See also Figure 9 , the sensor 230 may include a plurality of sensor groups 230A. Each sensor group 230A may include one or more ultrasonic sensors 231 and / or one or more visual sensors 232. Each sensor group 230A may monitor its own distance D ( Figure 1 ). The sensor groups 230A may be arranged to face the same and / or different directions, respectively, to monitor respective distances D in corresponding directions.

[0088] Figure 9 Four sensor groups 230A are shown for measuring distance D in four directions 202, each of which is parallel and / or perpendicular to one another. Optionally, sensor 230 may include a fifth and / or sixth sensor group 230A (not shown) facing a direction 203 perpendicular to the square formed by the four sensor groups 230A. More sensor groups 230A may be added to measure distance D in more directions.

[0089] although Figure 9While the sensor groups 230A are shown as being arranged in a rectangular and / or square configuration, the mobile platform 200 may include any number of the same and / or different sensor groups 230A arranged in any suitable configuration and sized and / or dimensioned to be disposed on the mobile platform. For example, the sensor groups 230A may be arranged in the shape of a triangle, a line, a parallelogram, a polygon, and / or a polyhedron.

[0090] although Figure 9 Sensor group 230A is shown to include an ultrasonic sensor 231 and a visual sensor 232, but sensor group 230A may include only one of ultrasonic sensor 231 and visual sensor 232. Additionally and / or alternatively, sensor group 230A may include one or more ultrasonic sensors 231 and / or one or more visual sensors 232. The advantages of using multiple processors and / or sensor types to detect the distance D from object 300 are that the measurement is more accurate through redundancy and the measurement is more robust to the limitations of specific sensors and / or sensor types.

[0091] See also Figure 10 , a detailed view of an exemplary mobile platform 200 is shown. Figure 10 The mobile platform 200 is shown as a drone. The sensor group 230A can be mounted on the accessory plate 230B on the mobile platform 200.

[0092] When mounted on the mobile platform 200, the sensor group 230A may include sensors 230 ( Figure 9 ), is used to measure various distances D between the mobile platform 200 and the object 300 (and / or other objects or obstacles). The distances D may be measured in a variety of directions, depending on the orientation of the sensor 230. Exemplary directions may include front, back, left, right, up, and / or down relative to the mobile platform 200.

[0093] See also Figure 11 , shows a method for determining the mobile platform 200 and the object 300 ( Figure 1 ) using at least one of an ultrasonic measurement value (also referred to herein as a first measurement value) and a visual measurement value (also referred to herein as a second measurement value). In step 1111, the ultrasonic sensor 231 ( Figure 8 In step 1112, the visual sensor 232 ( Figure 7 ) to measure the distance D to obtain a second measurement value. In step 1113, the distance D can be determined based on at least one of the first measurement value and the second measurement value.

[0094] The results of the first measurement and the second measurement may be combined in any predetermined manner to determine the distance D. For example, the distance D may be determined as a simple average and / or a weighted average of the first measurement and the second measurement.

[0095] For some objects, the first measurement or the second measurement may have particularly limited accuracy. Therefore, in some examples, based on characteristics of object 300 and / or the measurement method, distance D may be based on one of the first measurement or the second measurement.

[0096] For example, while visual sensor 232 is obtaining the second measurement, visual sensor 232 may not detect a texture of interest on object 300. If object 300 has a texture of interest, visual sensor 232 may identify one or more feature points 301 in the captured image of object 300. The number of feature points 301 may be equal to and / or greater than a predetermined number limit. Meeting the predetermined number limit ensures that the second measurement has the desired accuracy. Therefore, if the second measurement obtained by visual sensor 232 does not detect a texture of interest on object 300, distance D may be determined based on the first measurement obtained by ultrasonic sensor 231.

[0097] In another example, during the first measurement value obtained by the ultrasonic sensor 231, the ultrasonic receiver 231b ( Figure 8 3 ). The ultrasonic sensor 231 may receive an ultrasonic echo below a predetermined intensity limit. The exemplary predetermined intensity limit may be based on a selected portion of the intensity of the ultrasonic waves emitted by the ultrasonic transmitter 231a. When the ultrasonic echo is below the predetermined intensity limit, the emitted ultrasonic waves may be absorbed and / or weakened by the object 300. In this case, the accuracy of the ultrasonic sensor 231 may be reduced. Therefore, when the first measurement value obtained by the ultrasonic sensor 231 receives an ultrasonic echo below the predetermined intensity limit, the distance D may be determined based on the second measurement value obtained by the visual sensor 232.

[0098] In some examples, tasks can be performed on selected locations of object 300. Figure 12 , mobile platform 200 can move to (or approach) a selected service point 310 of object 300. Service point 310 of object 300 may include a target destination where mobile platform 200 can perform a task. In other words, mobile platform 200 may aim to reach service point 310 to complete a task. Service point 310 may have any spatial relationship with object 300. For example, service point 310 may be above, adjacent to, and / or below a surface of object 300. Additionally and / or alternatively, service point 310 may be located remotely from (or distal to) object 300.

[0099] Additionally and / or alternatively, the object 300 may move toward the service point 310 to perform the task. However, the object 300 does not necessarily need to reach the service point 310. For example, when the service point 310 is on the surface of the object 300, the mobile platform 200 may approach (or move close to) the service point 310 while maintaining a distance D ( Figure 1 In another example, the service point 310 may be located at a distance D from the surface. In this case, the mobile platform 200 may move to (and / or arrive at) the service point 310.

[0100] The mobile platform 200 can locate the service point 310 according to the location of the service point 310. The location of the service point 310 may include the coordinates of the service point 310 in an absolute coordinate system (or a global coordinate system). The absolute coordinate system may be based on a system capable of guiding the movement of the mobile platform 200. That is, the control system 210 ( Figure 2 ) can indicate that the mobile platform 200 moves to and / or reaches a certain location, which can be based on an absolute coordinate system. For illustrative purposes, Figure 12 It is shown that the absolute coordinate system is a Cartesian coordinate system including three coordinate axes X1, Y1, and Z1, and the coordinates of the selected service point 310 are (x1, y1, z1) in the absolute coordinate system.

[0101] The object 300 may define a relative coordinate system (or local coordinate system). The relative coordinate system may be used to define the spatial relationship between the selected service point 310 and the object 300. For illustrative purposes, Figure 12 The relative coordinate system is shown as a Cartesian coordinate system having three coordinate axes X2, Y2, and Z2, and the coordinates of the selected service point 310 are (x2, y2, z2) in the relative coordinate system. The coordinates (x2, y2, z2) of the selected service point 310 in the relative coordinate system can represent the spatial relationship between the selected service point 310 and the object 300.

[0102] A given service point 310 may be defined and / or selected based on the coordinates of the service point 310 in a relative coordinate system, regardless of the absolute coordinate system used. For example, when object 300 comprises an aircraft, the selected service point 310 may be the wingtip of the right wing of object 300. This selection may be made based on object 300 being in a relative coordinate system.

[0103] For a given service point 310, the coordinates (x2, y2, z2) in the relative coordinate system may be fixed regardless of how the object 300 is positioned in the absolute coordinate system. The coordinates (x1, y1, z1) in the absolute coordinate system may vary depending on how the object 300 is positioned in the absolute coordinate system. The coordinates in the relative coordinate system and the absolute coordinate system may be referred to as relative coordinates and absolute coordinates, respectively.

[0104] See also Figure 12 and Figure 13 , Figure 13 The position and / or orientation of the object 300 may be different from Figure 12 The position and / or orientation of object 300 in the maintenance process may be different when object 300 is moved. For example, object 300 may include an aircraft, which may be positioned differently at the maintenance facility each time. In another example, object 300 may be continuously moved while being serviced by mobile platform 200.

[0105] Accordingly, Figure 13 The relative coordinate system (or the three coordinate axes X2, Y2, Z2) in Figure 12 The relative coordinate system in is different. Figure 12 and Figure 13 In the example, the absolute coordinate system with three coordinate axes X1, Y1, and Z1 can remain unchanged.

[0106] Therefore, in Figure 13 In the relative coordinate system, the coordinates of the selected service point 310 may still be (x2, y2, z2). However, the coordinates of the selected service point 310 in the absolute coordinate system have changed to (x1', y1', z1').

[0107] For a selected service point 310 with coordinates (x2, y2, z2) in the relative coordinate system, the coordinates (x1, y1, z1) in the absolute coordinate system can be obtained based on the conversion relationship between the absolute coordinate system and the relative coordinate system. In other words, for a given service point 310, the conversion relationship is the relationship between the absolute coordinates and the relative coordinates. When the conversion relationship is known, the absolute coordinates (x1, y1, z1) can be obtained based on the relative coordinates (x2, y2, z2).

[0108] Exemplary conversion relationships may include the following expressions:

[0109] Equation (5)

[0110] Wherein a1, b1, c1, a2, b2, c2, a3, b3, and c3 are conversion coefficients of the conversion relationship between the absolute and relative coordinate systems. When the object 300 is stationary, the conversion relationship can be fixed, and / or, when the object 300 moves, the conversion relationship can be changed.

[0111] In some embodiments, the conversion relationship can be determined when the position and orientation of the object 300 in the absolute coordinate system are determined. Therefore, the coordinates (x1, y1, z1) can be based on the position and orientation of the object 300 in the absolute coordinate system.

[0112] The transformation relationship can be determined in any manner. For example, the transformation relationship can be determined using a method that provides a value for a transformation coefficient. In some examples, the transformation relationship can be determined based on one or more control points. In other words, the position and orientation of object 300 can be based on one or more control points. A control point can be a point with one or more coordinates that are at least partially known so that the transformation relationship can be calculated.

[0113] See also Figure 14 , the position of the object 300 can be determined by at least one control point 330a. The control point 330a can have a predetermined spatial relationship with the object 300 and be represented by relative coordinates (x2a, y2a, x2a). The control point 330a can have absolute coordinates (x1a, y1a, x1a).

[0114] The position of the object 300 can be represented by absolute coordinates (x1a, y1a, x1a). When the position of the object 300 is determined, the mobile platform 200 can locate the object 300. However, in order for the mobile platform 200 to locate the selected service point 310, it may be necessary to determine the direction of the object 300.

[0115] The orientation of the object 300 can be determined by one or more control points. Figure 15 The control points may include control point 330a and control points 330b and 330c. Control points 330b and 330c may each have a predetermined spatial relationship with object 300, represented by relative coordinates (x2b, y2b, x2b) and (x2c, y2c, x2c). Control point 330b may have absolute coordinates (x1b, y1b, x1b) and (x1c, y1c, x1c).

[0116] Direction 302 of object 300 may include directions 302B and 302C. Directions 302B and 302C may be vectors. Direction 302B may be represented by relative coordinates (x2b-x2a, y2b-y2a, z2b-z2a) and / or absolute coordinates (x1b-x1a, y1b-y1a, z1b-z1a). Direction 302C may be represented by relative coordinates (x2c-x2a, y2c-y2a, z2c-z2a) and / or absolute coordinates (x1c-x1a, y1c-y1a, z1c-z1a). Therefore, control points 330a-330c may be used to determine how object 300 is oriented in an absolute coordinate system.

[0117] Based on the position and / or orientation of the object in the absolute coordinate system, the mobile platform 200 may locate the selected service point 310 .

[0118] That is, the conversion relationship between the absolute and relative coordinate systems can be obtained. For example, through equation (5), the control point 330a at (x1a, y1a, x1a) and (x2a, y2a, x2a), the control point 330b at (x1b, y1b, x1b) and (x2b, y2b, x2b), and the control point 330c at (x1c, y1c, x1c) and (x2c, y2c, x2c) can each provide three equations. Nine equations can be used to solve the transformation coefficients a1, b1, c1, a2, b2, c2, a3, b3, and c3. Thus, the conversion relationship can be obtained.

[0119] See also Figure 16 , a detailed view of exemplary control points 330a, 330b is shown in relation to object 300. Figure 16 Object 300 is shown as aircraft 300A. Figure 16 Control point 330a is shown as a point along the centerline 303 of the aircraft a predetermined distance L1 forward of the forward-most portion of the aircraft 300A. Figure 16 Control point 330b is shown as a point a predetermined distance L2 forward of control point 330a along centerline 303. L1 and L2 may be the same and / or different. Figure 16 Direction 302B is shown as the forward direction of aircraft 300A.

[0120] like Figure 16 As shown, the three-dimensional orientation of object 300 may be at least partially known. For example, aircraft 300A may be level with the ground. Therefore, a two-dimensional transformation relationship can fully describe the position and / or orientation of aircraft 300A. That is, both absolute and relative coordinate systems may include two axes (X1, Y1) and (X2, Y2), respectively.

[0121] For example, the transformation relationship can be expressed as follows:

[0122] Equation (6)

[0123] Therefore, the position and orientation of the aircraft 300A in the absolute coordinate system can be determined by the control points 330a and 330b.

[0124] Based on the position and / or orientation of the object in the absolute coordinate system, the mobile platform 200 ( Figure 12 ) can locate the selected service point 310.

[0125] In other words, the conversion relationship between the absolute and relative coordinate systems can be obtained. For example, through equation (6), the control point 330a at (x1a, y1a) and (x2a, y2a) and the control point 330b at (x1b, y1b) and (x2b, y2b) can be used to solve for the conversion coefficients a1, b1, a2, and b2. This can be used to obtain the conversion relationship.

[0126] See also Figure 17 , showing a service point 310 ( Figure 12 In step 1121, the spatial relationship between the selected service point 310 and the object 300 may be obtained. For example, the spatial relationship between the selected service point 310 and the object 300 may be obtained based on a model (not shown) of the object 300. For example, the spatial relationship may be expressed as the coordinates (x2, y2, z2) of the selected service point 310 in the relative coordinate system ( Figure 12 shown in ).

[0127] A model may refer to a replica of object 300 that partially and / or completely replicates the shape, size, and / or dimensions of object 300. Exemplary models may include computerized two-dimensional and / or three-dimensional models.

[0128] In step 1122, the position and orientation of the object 300 may be obtained. For example, by controlling the point 330a ( Figure 14 For example, by controlling points 330a, 330b and / or 330c ( Figure 15 and Figure 16 ) can obtain the direction of the object 300.

[0129] The control points 330a-330c may have respective predetermined spatial relationships with the object 300. The predetermined spatial relationships may be expressed as relative coordinates, which may be obtained based on a model of the object 300.

[0130] Additionally and / or alternatively, control points 330a-330c can have respective absolute coordinates, which can be obtained by any navigation method. An exemplary navigation method can include a global positioning system (GPS). For example, a position tracking device (e.g., a GPS device) can be positioned at a selected control point 310 and measure the absolute coordinates. The position tracking device can be placed at the selected control point by any method and / or tool, such as a human operator, a robotic arm, and / or a lift.

[0131] To improve the accuracy of the absolute coordinates measured by the location tracking device, the distance between the control points and / or service points can be greater than the predetermined distance limit. For example, L1 and / or L2 ( Figure 16 ) can be greater than a predetermined spacing limit. Thus, the position tracking device can avoid misidentifying two points (e.g., control points 330a, 330b) as a single point. An exemplary predetermined spacing limit can be greater than and / or equal to 1 meter.

[0132] In step 1123 , the position of the selected service point 310 may be determined based on the spatial relationship between the selected service point 310 and the object 300 , and / or the transformation relationship between the relative and absolute coordinate systems.

[0133] Although absolute and relative coordinate systems are described using a Cartesian coordinate system, the absolute and / or relative coordinate systems may include any coordinate systems of the same and / or different types. Exemplary coordinate systems may include cylindrical coordinate systems, spherical coordinate systems, and / or geographic coordinate systems.

[0134] although Figure 14 and Figure 15 Control point 330a is shown for determining the position and orientation of object 300, but the position and orientation of object 300 may be based on any of the same and / or different control points, respectively. Figure 14 and Figure 16 It is shown that the position of the object 300 can be determined by one control point. Figure 15 and Figure 16 It is shown that the orientation of the object 300 can be determined by 3 and 2 control points, respectively. However, the position and / or orientation of the object 300 can be determined by any number of control points. The number of control points used to determine the position and orientation of the object 300 can be the same and / or different.

[0135] See also Figure 18 , an exemplary region of interest 320 of object 300 is shown. Region 320 may include one or more service points 310. Mobile platform 200 may move to a selected service point 310 along a travel path 340. When there are one or more service points 310 in region 320, travel path 340 may connect the multiple selected service points 310.

[0136] Mobile platform 200( Figure 12 3 ) can move along a travel path 340 to pass through the selected service points 310 and perform a task for at least one of the selected service points 310. For example, when the mobile platform 200 includes an unmanned aerial vehicle, the mobile platform 200 can fly along the travel path 340. When the mobile platform 200 travels and / or performs a task, the travel path can be predetermined, fixed, and / or dynamically adjusted.

[0137] See also Figure 19 , the mobile platform 200 may include a location tracking device 280 for determining the location of the mobile platform 200. An exemplary location tracking device 280 may include a global positioning system (GPS) module and / or a differential GPS (or DGPS) module (not shown).

[0138] When the location tracking device 280 includes a DGPS module, the location tracking device 280 can be operated to communicate with the base station 600. The base station 600 may include a GPS receiver established at a precise known geographic location. For example, in the United States and Canada, the base station 600 may include a DGPS base station system operated by the United States Coast Guard (USCG) and the Canadian Coast Guard (CCG). The DGPS base station system can communicate with the location tracking device 280 at long-wave radio frequencies between 285 kHz and 325 kHz near major waterways and ports.

[0139] Base station 600 can calculate its own position based on satellite signals and compare the calculated position to a known position. The difference between the calculated position and the known position can provide a differential correction. Thus, base station 600 can broadcast the differential correction to location tracking device 280 automatically and / or in response to location tracking device 280.

[0140] Base station 600 may communicate with location tracking device 280 via any wired and / or wireless communication method. An exemplary method may include radio frequency (RF) communication.

[0141] When the location tracking device 280 measures the location of the location tracking device 280 and / or the mobile platform 200, the location tracking device 280 may apply a differential correction to correct the GPS data recorded by the location tracking device 280. Thus, the corrected GPS data is a DPGS signal. After the correction is performed, the GPS data of the location of the mobile platform 200 has a high degree of accuracy. For example, without the correction, the GPS may measure the location of the mobile platform 200 with an error of approximately 2 meters. When the correction is performed, the location tracking device 280 may measure the location of the mobile platform 200 with an error as small as 10 centimeters. Thus, the mobile platform 200 may be moved to a selected location with a high degree of accuracy.

[0142] although Figure 19 For illustrative purposes, the location tracking device 280 is shown communicating with one base station 600, but the location tracking device 280 may communicate with any predetermined number of the same and / or different base stations 600. Figure 19 Base station 600 is shown in communication with location tracking device 280, but location tracking device 280 does not necessarily receive differential corrections from base station 600. Additionally and / or alternatively, location tracking device 280 may receive differential corrections from one or more satellites. For example, wide area DGPS (WADGPS) and / or satellite-based augmentation systems may include one or more ground base stations located at precisely surveyed points. Base station 600 may acquire data, such as differential corrections. Base station 600 may measure one or more satellites, satellite signals, and / or other environmental factors that may affect the signals received by location tracking device 280. Base station 600 may transmit the measurement results and / or the results of the differential corrections to one or more satellites. The satellites may broadcast correction information to location tracking device 280.

[0143] See also Figure 20 , showing a detailed view of the mobile platform 200. Figure 20 The mobile platform 200 is shown as a UAV. The position tracking device 280 is shown to include a DGPS module 280A mounted on the mobile platform 200.

[0144] In some embodiments, the base station 600 ( Figure 19 Base station 600 (shown in FIG) can be portable and include a GPS receiver. Base station 600 can be placed at a selected location. The location where base station 600 is located can be accurately surveyed and recorded as a known location. In some embodiments, base station 600 can communicate with DGPS module 280A via 433 MHz radio frequency signals. By using a portable base station 600, an operator can advantageously operate an unmanned aerial vehicle from any location, without being restricted to the fixed location of base station 600.

[0145] See also Figure 21 , showing that the mobile platform 200 can be connected to a payload 290. The payload 290 can perform tasks on the object 300. For example, the payload 290 can perform tasks on the selected service point 310 ( Figure 12 ) to perform the task.

[0146] Alternatively, as Figure 21As shown, payload 290 can be connected to mobile platform 200 via carrier 292. Carrier 292 can allow payload 290 to rotate relative to mobile platform 200 about one or more axes, such as three axes: X (or pitch), Z (or roll), and Y (or yaw). Therefore, even if the orientation of mobile platform 200 remains unchanged, payload 290 can be oriented in any of a variety of directions.

[0147] Optionally, carrier 292 can stabilize payload 290 against the motion of mobile platform 200. For example, carrier 292 can provide a shock-absorbing function to reduce vibrations of payload 290 caused by the motion of mobile platform 200. Additionally and / or alternatively, carrier 292 can include one or more sensors to measure unwanted movement of payload 290 due to positional drift of mobile platform 200. Thus, payload 292 can compensate for the movement by adjusting the orientation and / or position of payload 290. Thus, even in the event of vibration, shaking, and / or drift of mobile platform 200, payload 290 can remain stable and perform tasks to selected service points 310 with greater positioning accuracy.

[0148] although Figure 21 The mobile platform 200 is shown to include one payload 290 and one carrier 292, but the mobile platform 200 may include any number of the same and / or different payloads 290. The payloads 290 may be connected to the mobile platform 200 via any number of the same and / or different carriers 292. Some payloads 290 may be connected to the mobile platform 200 without a carrier 292.

[0149] Mobile platform 200 can perform one or more of the same and / or different tasks using payloads 290. In some embodiments, mobile platform 200 can perform each of the same and / or different tasks using a separate payload 290. Additionally and / or alternatively, payloads 290 can perform one or more of the same and / or different tasks. Additionally and / or alternatively, multiple payloads 290 can collaboratively perform tasks.

[0150] See also Figure 22 , showing a detailed view of the mobile platform 200. Figure 22 Mobile platform 200 is shown as an unmanned aerial vehicle. Payload 290 may include camera 290A. Camera 290A may capture still and / or video images of object 300.

[0151] Carrier 292 may include a gimbal 292A. In some embodiments, gimbal 292A may include a three-axis gimbal. Mobile platform 200 may control gimbal 292A to adjust the orientation of camera 290A. Thus, while mobile platform 200 hovers in mid-air at a fixed orientation, camera 290A can be oriented in any of a variety of directions, capturing images of object 300 from multiple perspectives.

[0152] See also Figure 23 , mobile platform 200 can communicate with task manager 400. Task manager 400 can include a computer device. Exemplary task managers 400 can include mobile phones, smartphones, tablet computers, personal computers, server computers, and / or the like. Communication between mobile platform 200 and task manager 400 can be wired and / or wireless. For example, task manager 400 and / or mobile platform 200 can include radio frequency circuitry (not shown). Radio frequency circuitry can include, but is not limited to, an antenna, at least one amplifier, a tuner, one or more oscillators, a subscriber identity (SIM) card, a transceiver, a coupler, an LNA (e.g., a low noise amplifier), a duplexer, and the like. The radio frequency circuitry can communicate with other devices via a wireless communication network. Wireless communication can use any communication standard or protocol, including, but not limited to, GSM (Global System for Mobile Communications), GPRS (General Packet Radio Service), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), LTE (Long Term Evolution), email, and SMS (Short Message Service).

[0153] When performing a task using payload 290, mobile platform 200 may obtain data 204. For example, when payload 290 includes camera 290A ( Figure 22 ), data 204 may include images captured by camera 290A.

[0154] Additionally and / or alternatively, when payload 290 includes instrumentation for measuring properties of object 300, data 204 may therefore include measurements obtained by the instrumentation. Example measurements may include directing light to a surface of object 300 to measure a property of the surface, such as light reflectance.

[0155] The mobile platform 200 may provide the data 204 to the task manager 400. The task manager 400 may process and / or analyze the data 204. Based on the processing and / or analysis, the task manager 400 may determine a subsequent task for the mobile platform 200 to perform and generate a command 206 accordingly. The exemplary command 206 may instruct the mobile platform 200 to update the task and perform the updated task. Additionally and / or alternatively, the exemplary command 206 may instruct the mobile platform 200 to adjust the payload 290 ( Figure 21Additionally and / or alternatively, the example commands 206 may instruct the mobile platform 200 to move to a different location, pause a task, and / or stop a task.

[0156] See also Figure 24 , shows an exemplary method 1200 for servicing an object 300 using a mobile platform 200. In step 1210, by using the camera 290A ( Figure 22 ) can capture an image of object 300. For example, camera 290A can capture an image of the surface of object 300.

[0157] At step 1220, the image may be transferred to the task manager 400 ( Figure 23 ), for analysis of the object 300. For example, when the camera 290A captures an image of the surface of the object 300, the analysis of the object 300 may include analysis of the surface of the object 300 based on the image.

[0158] For example, task manager 400 may present and / or display the images to enable an operator to visually inspect object 300. Additionally and / or alternatively, task manager 400 may transmit the images to a third party's computer system for analysis, inspection, and / or storage.

[0159] Additionally and / or alternatively, the task manager 400 may perform one or more image feature recognition (and / or feature detection) methods. A feature may refer to a portion of interest in an image. Exemplary features may include edges, corners, points, and / or ridges. Features may have selected characteristics, such as shape, size, dimension, color, texture, and / or the like. The selection of features depends on the type of task that the mobile platform 200 is configured to perform.

[0160] Feature recognition methods can determine whether a feature of a selected type is present at a point in an image. Exemplary feature recognition methods can include edge detection and / or image binarization.

[0161] The task manager 400 may generate a command 206 based on the analysis of the image ( Figure 23 For example, when task manager 400, a third party, and / or a human detects a feature of interest in an image, task manager 400 may select the feature and send command 206 to capture a zoomed-in image of the selected feature. Additionally and / or alternatively, task manager 400 may send command 206 to adjust the orientation of camera 290A to capture the selected feature from a different perspective to better observe the feature.

[0162] By using the disclosed method, it is advantageously possible to increase the automation and efficiency of servicing an object 300. For example, to inspect the surface of an object 300, the mobile platform 200 can repeatedly return to a selected location on the object 300 for repeated inspections, even if the location and / or orientation of the object 300 is different for each inspection.

[0163] Additionally and / or alternatively, mobile platform 200 can advantageously approach object 300 and avoid collision with object 300. Thus, mobile platform 200 can capture an image of object 300 at close range. Therefore, the image can present features and / or topography of the surface of object 300 with sufficiently high clarity and / or resolution. Therefore, the surface can be detected by analyzing the image.

[0164] Additionally and / or alternatively, mobile platform 200 can transmit images to task manager 400 for analysis. Task manager 400 can be remote from object 300. As mobile platform 200 captures images, the images can be analyzed in real time. Additionally and / or alternatively, feedback based on the images can be generated in real time, and task manager 400 can generate commands 206 based on the feedback. Thus, mobile platform 200 can act based on commands 206 to timely acquire new images with adjusted content and / or quality.

[0165] Additionally and / or alternatively, the implementation of real-time analysis and / or feedback requires and / or does not require human intervention or participation. Even if the image is analyzed manually, the person does not need to be present at the location of the object 300. Therefore, labor costs and time can be reduced.

[0166] See also Figure 25 , a plurality of mobile platforms 200A-200C are shown servicing an object 300. The mobile platforms 200A-200C can move to service points 310A-310C, respectively. The mobile platforms 200A-200C can perform the same and / or different tasks, respectively.

[0167] Mobile platforms 200A-200C can collaborate with each other to complete tasks. That is, each mobile platform 200A-200C can perform at least a portion of a task to jointly perform the task. Additionally and / or alternatively, each mobile platform 200A-200C can perform one or more of multiple tasks to jointly perform multiple tasks.

[0168] For example, when object 300 is on the ground, mobile platform 200A can capture an image of the upper surface of object 300. Mobile platform 200C can travel between object 300 and the ground to capture an image of the lower surface of object 300. Thus, by working in cooperation with each other, mobile platforms 200A and 200C can each capture images representing different portions of object 300, and complement each other.

[0169] Multiple mobile platforms 200A-200C can perform tasks on object 300 simultaneously and / or sequentially. Multiple mobile platforms 200A-200C can be the same and / or different. Exemplary mobile platforms can include unmanned aerial vehicles and / or ground vehicles. Although Figure 25 Three mobile platforms are shown, but any number of the same and / or different mobile platforms 200 can cooperate to perform a task. Multiple mobile platforms 200 can serve a service point. Additionally and / or alternatively, a mobile platform 200 can serve multiple service points.

[0170] See also Figure 26 The system 100 may further include a terminal device 500 for communicating with the mobile platform 200. The terminal device 500 may include any computer system. Exemplary terminal devices 500 may include mobile phones, smart phones, tablet computers, personal computers, server computers, and / or the like.

[0171] During operation of the mobile platform 200, the terminal device 500 may be located away from the mobile platform 200 to service the object 300. The terminal device 500 may provide instructions to the mobile platform 200 to implement the functions disclosed herein.

[0172] See also Figure 27 , the terminal device 500 may include a processor (and / or central processing unit, or CPU) 510. Although Figure 21 One processor 510 is shown in FIG. 5 , but the terminal device 500 may include any number of the same and / or different processors 510 .

[0173] The terminal device 500 may include a memory 520. The memory 520 may include a high-speed random access memory (RAM). Additionally and / or alternatively, the memory 520 may include a non-volatile memory, such as one or more disk storage devices, flash memory devices, and / or other non-volatile solid-state storage devices. Figure 27 One memory 520 is shown in FIG. 5 , but the terminal device 500 may include any number of the same and / or different memories 520 .

[0174] The processor 510 may run (or execute) various software programs and / or instruction sets stored in the memory 520 to perform various functions of the terminal device 500 .

[0175] The terminal device 500 may include a communication module 530. The communication module 530 may be operable to communicate with another computer system (such as the mobile platform control system 210 ( Figure 2The exemplary communication module 530 may include a radio frequency circuit that can wirelessly receive and / or transmit information. The terminal device 500 may communicate with the mobile platform 200 directly and / or through a routing device.

[0176] Additionally and / or alternatively, the terminal device 500 may include a display system 540. In some embodiments, the display system 540 may include a touch-sensitive display, also referred to as a “touch screen.” The display system 540 may display information stored in the terminal device 500 for presentation.

[0177] See also Figure 28 , the terminal device 500 can be connected to the remote controller 502. For example, the terminal device 500 can be connected to the remote controller 502 via an external communication port that supports standards such as Universal Serial Bus (USB), FireWire, and / or the like. The remote controller 502 may include one or more processors (not shown) for at least partially controlling the operation of the mobile platform 200. The remote controller 502 can communicate with the mobile platform 200 via wired and / or wireless methods. For example, various wireless communication methods can be used for remote communication between the mobile platform 200 and the remote controller 502. Suitable communication methods include, for example, radio, Wireless Fidelity (Wi-Fi), cellular, satellite, and broadcast.

[0178] The terminal device 500 can communicate with the mobile platform 200 through the remote controller 502. That is, the mobile platform 200 can transmit data to the remote controller 502. The remote controller 502 can forward the data to the terminal device 500. On the other hand, the terminal device 500 can forward instructions to the remote controller 502. The remote controller 502 can transmit instructions to the mobile platform 200. In other words, the remote controller 502 can operate as a data exchange channel to route communications between the terminal device 500 and the mobile platform 200.

[0179] Figure 29 is an exemplary top-level flow chart illustrating an embodiment of a method 2000 for serving an object 300 via a mobile platform 200. The terminal device 500 ( Figure 27 ) can implement method 2000.

[0180] In step 2100, one or more service points 310 may be determined based on the model of the object 300. The terminal device 500 may determine the service points 310 ( Figure 12 shown in ).

[0181] For example, object 300 may include an aircraft, such as a passenger plane. Therefore, the model may include three-dimensional information about the aircraft. Exemplary three-dimensional information may include the length, width, and / or height of the fuselage, the inclination angle of the aircraft's front surface, the inclination angle of the wing root and / or wingtip, the inclination angle between the wing root and wingtip, the width of the horizontal stabilizer and / or horizontal stabilizer, or a combination thereof.

[0182] At step 2200 , the mobile platform 200 is directed to perform one or more tasks at the identified service point 310 . The terminal device 500 may direct the mobile platform 200 to perform the tasks. For example, the terminal device 500 may direct the mobile platform 200 to capture an image of the object 300 .

[0183] Additionally and / or alternatively, the terminal device 500 may guide the mobile platform 200 to maintain a distance D ( Figure 1 ). The terminal device 500 may determine a distance range of the maintained distance D. For example, the terminal device 500 may determine a lower limit and / or an upper limit of the maintained distance D.

[0184] See also Figure 30 , showing a method 2100 for identifying one or more service points 310. The terminal device 500 ( Figure 27 ) can implement method 2100.

[0185] At step 2110, a model of the object 300 may be obtained. The terminal device 500 may obtain the model from any source. For example, the manufacturer and / or builder of the object 300 may provide the model. Object 300 may be measured by a specific person or machine to generate the model. As the shape, size, and / or dimensions of the object 300 change, the model may change accordingly.

[0186] The terminal device 500 may obtain the model from another computer system and / or a model library. Additionally and / or alternatively, the terminal device 500 may store the model in the memory 520 in advance and retrieve the model from the memory 520.

[0187] In step 2120, the region of interest 320 is selected based on the model. The terminal device 500 may select the region 320 ( Figure 18 For example, region 320 may be a surface region of interest on object 300, such as a surface region that has a high probability of being damaged during use of object 300.

[0188] The terminal device 500 may autonomously select the area 320 based on a predetermined selection criterion, and / or select the area 320 based on a selection instruction input to the terminal device 500 .

[0189] In step 2130, the distribution of service points 310 in area 320 is determined. The distribution of service points 310 may refer to the density of service points 310 and / or the spatial relationship between service points 310 and object 300. Terminal device 500 may determine the distribution of service points 310 based on any criteria.

[0190] For example, when mobile platform 200 is positioned at a distance D from service point 310, payload 290 may perform a task on a sub-area of ​​area 320. The size and / or shape of the sub-area may be based on the distance D and / or the payload 290 ( Figure 21 For example, camera 290A ( Figure 22 ) can capture an image of a sub-area of ​​a certain size at a distance D.

[0191] To completely perform the task on area 320, terminal device 500 may determine the density based on the size and / or shape of the sub-areas and area 320. Terminal device 500 may determine and / or select service point 310 based on the determined density. Additionally and / or alternatively, terminal device 500 may determine a travel path for mobile platform 200 to pass through service point 310.

[0192] Although method 2100 may include steps 2110 - 2130 , one or more of 2110 - 2130 may be omitted.

[0193] Additionally and / or alternatively, the terminal device 500 may determine the location of the selected service point 310 ( Figure 12 Based on the model, the terminal device 500 may determine the spatial relationship between the selected service point 310 and the object 300.

[0194] Additionally and / or alternatively, based on the model, the terminal device 500 may determine the selected control point (e.g. Figure 15 and Figure 16 The spatial relationship between the control points 330a-330c) and the object 300 is shown.

[0195] To determine the absolute coordinates of the control point, the location tracking device may be positioned at the control point to measure the absolute coordinates of the control point 330. The location tracking device may transmit the measurement result to the terminal device 500.

[0196] In some embodiments, the position tracking device 280 ( Figure 19 ) can be used to determine the absolute coordinates of the control point 330. In one example, the DGPS module 280A ( Figure 19) can be used to determine the absolute coordinates of control point 330. In other words, mobile platform 200 can be positioned at a selected control point by a person or machine. DGPS module 280A can measure the GPS position of mobile platform 200 to obtain the position of the control point in an absolute coordinate system. When the position and / or orientation of object 300 changes, the absolute coordinates of the control point can be measured, thereby updating the position of service point 310.

[0197] Additionally and / or alternatively, when the mobile platform 200 includes a camera 290A ( Figure 22 ), the terminal device 500 may guide the mobile platform 200 to capture an image of the object 300 at one or more viewing angles. The mobile platform 200 may guide the carrier 292 to adjust the direction of the camera 290A, thereby adjusting the angle at which the image is captured.

[0198] Additionally and / or alternatively, the mobile platform 200 may send the image to the terminal device 500 for analysis. The terminal device 500 may analyze the image in real time and / or send the image to a third party for analysis and / or storage. Optionally, the display system 540 ( Figure 27 ) can display the image for inspection.

[0199] Various embodiments also disclose computer program products including instructions for operating the mobile platform 100. The program / software may be stored in a (non-transitory) computer-readable storage medium, including, for example, read-only memory (ROM), random access memory (RAM), memory, registers, a computer hard disk, a removable disk, a CD-ROM, an optical disk, a floppy disk, a magnetic disk, etc. The program / software may include coded instructions to instruct one or more processors on a computer device to perform methods according to various disclosed embodiments. For example, the program / software may be executed on a computer and / or mobile device in the form of an application, an application, and / or application software.

[0200] The embodiments of the present invention are susceptible to various modifications and alternative forms, and specific examples thereof have been shown by way of example in the drawings and described in detail herein. However, it should be understood that the embodiments of the present invention are not limited to the specific forms or methods disclosed, but rather, the embodiments of the present invention are intended to cover all modifications, equivalents, and alternatives.

Claims

1. A method for serving an object via a mobile platform, comprising: determining a position of a service point relative to an object, wherein the position of the service point is determined by a plurality of control points, the control points having a predetermined spatial relationship with the object; According to the position of the service point, controlling the mobile platform to move toward the service point, wherein the mobile platform includes a plurality of mobile platforms, and different mobile platforms respectively move to different service points to jointly serve the object; maintaining a distance between the mobile platform and the object; and During the maintaining period, the mobile platform is controlled to perform a task on the object.

2. The method according to claim 1, wherein When the mobile platform is hovering in a fixed direction, the camera of the mobile platform faces any direction of multiple directions to capture images of the object from multiple perspectives.

3. The method according to claim 1, wherein The maintenance includes at least one of the following: maintaining a distance between the mobile platform and the surface of the object; maintaining said distance within a predetermined distance range; The distance is maintained based on operating conditions of the mobile platform.

4. The method according to claim 3, wherein: The mobile platform maintains the distance above a lower limit and / or below an upper limit to capture an image of a selected area of ​​the object within a selected field of view size and / or resolution.

5. The method according to claim 1, wherein controlling the mobile platform to perform a task on the object comprises controlling a camera of the mobile platform to capture an image of the object, and the method further comprises: sending the captured image of the object to a task manager in communication with the mobile platform; A control command is received from the task manager.

6. The method according to claim 5, wherein: The control command can indicate at least one of the following: based on detecting a feature of interest in the image, capturing a magnified image of the selected feature; Based on detecting a feature of interest in the image, adjusting the viewing angle of a camera of the mobile platform to better observe the selected feature; Instructing the mobile platform to update tasks and execute updated tasks; Instruct the mobile platform to pause the task and / or stop the task.

7. The method according to claim 1, wherein The controlling the mobile platform to perform a task on the object includes: Different mobile platforms have different payloads, and different mobile platforms use their respective payloads to perform tasks.

8. The method according to claim 1, wherein The position of the service point is determined by the control point and a predetermined spatial relationship between the control point and the object.

9. The method according to claim 1, wherein The distance is determined and / or dynamically adjusted based on one or more operating conditions of the mobile platform, wherein the operating conditions include at least one of the following: Environmental conditions surrounding the mobile platform, the movement of the mobile platform, and the accuracy of the control system of the mobile platform.

10. The method according to claim 1, wherein Different mobile platforms include unmanned aerial vehicles and ground vehicles.

Images