Control method, program, and mobile body system

Abstract

The present technique relates to a control method, a program, and a mobile body system, whereby a desired area on a structure can be easily designated as a target area to be imaged for inspection or the like. On the basis of time-series data of position information of an indication point, which is a point on a structure instructed by an indicator that can indicate a point on a structure, a target area to be imaged on the structure is specified, and movement control for moving a mobile body to the target area is carried out. The present technique can be applied to, for example, a mobile body system that controls a mobile body such as a drone.

Description

Control method, program, and mobile system

[0001] The present technology relates to a control method, a program, and a mobile system, and in particular to a control method, a program, and a mobile system that enable, for example, a desired area on a structure to be easily designated as a target area to be photographed for inspection or the like.

[0002] As a technology for using drones to inspect structures such as bridges and towers, a technology has been proposed in which a bright spot (image) formed by irradiating the structure with laser light is photographed, and the movement of the drone is controlled to follow the bright spot (see, for example, Patent Documents 1 and 2).

[0003] JP 2022-086062 A JP 2019-073182 A

[0004] When inspecting a structure, it may be necessary to specify an area (range, section) on the structure to be inspected and inspect that area, but the techniques described in Patent Documents 1 and 2 made it difficult to specify an area on the structure.

[0005] This technology was developed in consideration of such circumstances, and makes it possible to easily designate a desired area on a structure as the target area to be photographed for inspection, etc.

[0006] The control method of the present technology is a control method that includes identifying a target area on the structure to be photographed based on time series data of position information of an indicated point, which is a point on the structure indicated by an indicator that can indicate a point on the structure, and performing movement control to move a moving body to the target area.

[0007] The program of this technology is a program for causing a computer to execute processing including identifying a target area on a structure to be photographed based on time series data of position information of an indicated point, which is a point on the structure indicated by an indicator capable of indicating a point on the structure, and performing movement control to move a moving body to the target area.

[0008] In the control method and program of the present technology, a target area on the structure to be photographed is identified based on time series data of position information of an indicated point, which is a point on the structure indicated by an indicator capable of indicating a point on the structure, and movement control is performed to move the moving body to the target area.

[0009] A first mobile body system of the present technology is a mobile body system including: an indicator capable of indicating a point on a structure; a camera that captures an image of the structure; an image recognition unit that recognizes an indication point on the image, which is a point on the structure indicated by the indicator, and acquires first position information that represents the position of the indication point on the image; a 3D structure recognition unit that recognizes a 3D (dimensional) structure of the structure; a conversion unit that uses the 3D structure of the structure to convert time series data of the first position information into time series data of second position information that represents the position of the indication point on the structure; an identification unit that identifies a target area on the structure to be photographed based on the time series data of the second position information; and a movement control unit that performs movement control to move a mobile body to the target area.

[0010] In a first mobile body system of the present technology, a point on a structure is indicated by a pointer, and an image of the structure is captured. A pointer, which is the point on the structure indicated by the pointer, is recognized on the image, and first position information representing the position of the pointer on the image is acquired. A 3D (dimensional) structure of the structure is recognized, and using the 3D structure of the structure, time series data of the first position information is converted into time series data of second position information representing the position of the pointer on the structure. Based on the time series data of the second position information, a target area on the structure to be photographed is identified, and movement control is performed to move the mobile body to the target area.

[0011] The second mobile body system of the present technology is a mobile body system that includes an indicator capable of indicating a point on a structure, a distance measurement unit that measures the distance to an indication point, which is the point on the structure indicated by the indicator, a position calculation unit that uses the distance to calculate position information that represents the position of the indication point on the structure, an identification unit that identifies a target area on the structure to be photographed based on time series data of the position information, and a movement control unit that performs movement control to move the mobile body to the target area.

[0012] In a second mobile body system of the present technology, a point on a structure is indicated by an indicator, and a distance to an indicated point, which is the point on the structure indicated by the indicator, is measured. Using the distance, position information indicating the position of the indicated point on the structure is calculated. Based on time-series data of the position information, a target area on the structure to be photographed is identified, and movement control is performed to move the mobile body to the target area.

[0013] A third mobile body system of the present technology is a mobile body system that includes an indicator capable of indicating a point on a structure, an identification unit that identifies a target area on the structure to be photographed based on time series data of position information of an indication point, which is a point on the structure indicated by the indicator, and a movement control unit that performs movement control to move the mobile body to the target area.

[0014] In the third mobile body system of the present technology, a point on a structure is indicated by an indicator, and a target area on the structure to be photographed is identified based on time-series data of position information of the indicated point, which is the point on the structure indicated by the indicator, and movement control is performed to move the mobile body to the target area.

[0015] A mobile system can be made up of multiple independent devices and the like.

[0016] 1 is a diagram illustrating an example configuration of an embodiment of a mobile body system to which the present technology is applied. FIG. 2 is a diagram illustrating an example of designating a target area on a structure by an indication point indicated by an indicator 32. FIG. 3 is a block diagram illustrating an example hardware configuration of a mobile body 20. FIG. 4 is a block diagram illustrating an example functional configuration of the mobile body system 10. FIG. 5 is a flowchart illustrating processing of the mobile body system 10. FIG. 6 is a block diagram illustrating a first example functional configuration of a computer 25 included in the mobile body 20. FIG. 7 is a block diagram illustrating a second example functional configuration of the computer 25 included in the mobile body 20. FIG. 8 is a diagram illustrating an example of handling of position information for performing movement control of the mobile body 20. FIG. 9 is a diagram illustrating an example of designating a target area based on trajectory information by a target area designation unit 44. FIG. 10 is a block diagram illustrating another example functional configuration of the mobile body system 10. FIG. 11 is a block diagram illustrating an example hardware configuration of the computer 25.

[0017] <One embodiment of a mobile system to which the present technology is applied>

[0018] FIG. 1 is a diagram showing an example of the configuration of an embodiment of a mobile system to which the present technology is applied.

[0019] 1 , a mobile body system 10 includes a mobile body 20, a transmitter 31, and an indicator 32, and can be used for inspecting or testing structures such as bridges and towers. In the mobile body system 10, for example, the mobile body 20 identifies a target area on the structure to be photographed for inspection or the like, moves to the target area so that it can be photographed for inspection or the like, and photographs the target area. The image obtained by photographing the target area is visually inspected by a person or processed, and the target area is inspected or the like.

[0020] The mobile object 20 is, for example, a drone, and can move (fly) autonomously or can move under the control of the transmitter 31. A camera 21 is detachably attached to the mobile object 20, and images can be taken by the attached camera 21. Images taken by the camera 21 can be transmitted to the transmitter 31 by wireless communication and displayed.

[0021] The mobile body 20 is not limited to a mobile body that moves (flies) in the air (including outside the atmosphere), such as a drone. As the mobile body 20, in addition to a drone, for example, a mobile body that moves on water, underwater, or on land can be used.

[0022] The transmitter 31 can wirelessly exchange various information (data) with the mobile object 20. For example, a user can remotely control the mobile object 20 by operating the transmitter 31.

[0023] The indicator 32 is a device or other object capable of indicating a point on a structure, such as a bridge, to be inspected, and is, for example, a laser pointer. When the indicator 32 is a laser pointer, the user indicates a point on the structure with a laser beam emitted by the indicator 32, which is a laser pointer. That is, when the laser beam is irradiated onto the structure, a bright spot (image) is formed at the position of the structure where the laser beam is irradiated, and this bright spot is the point on the structure indicated by the laser beam. The point on the structure indicated by the indicator 32 (or the laser beam emitted by it) is also referred to as an indicated point.

[0024] The user can irradiate the structure with laser light using the indicator 32 to form a bright point as an indicator on the structure. The user can then move the indicator on the structure by performing an operation such as shaking the indicator 32, and designate a desired area on the structure as a target area using the moving indicator (time series data). Therefore, the user can easily designate a desired area on the structure as a target area. For example, the user can designate a desired area on the structure as a target area by moving the indicator as if tracing the desired area on the structure.

[0025] In the mobile body system 10 configured as described above, the user operates the indicator 32 to move the pointer to specify a target area. The mobile body system 10 sequentially acquires position information indicating the position of the pointer, and identifies the target area on the structure based on time-series data of the position information of the pointer. The mobile body system 10 then performs movement control to move the mobile body 20 to the target area so that the target area can be photographed for inspection, etc.

[0026] FIG. 2 is a diagram showing an example of specifying a target area on a structure by an indication point indicated by the indicator 32.

[0027] The user can move the pointer on the structure by operating the pointer 32, and specify a target area on the structure using the moving pointer. In Fig. 2, a rectangular area on the back side of the bridge girder of a bridge, which is a structure, is specified as the target area.

[0028] <Example of hardware configuration of moving body 20>

[0029] FIG. 3 is a block diagram showing an example of the hardware configuration of the mobile unit 20. As shown in FIG.

[0030] In FIG. 3, a moving body 20 includes a camera 21, a camera attitude control device 22, a space recognition sensor 23, a flight controller 24, a computer 25, and the like.

[0031] The camera 21 captures images (moving images and still images) and supplies them to the computer 25 .

[0032] The camera attitude control device 22 is configured by, for example, a gimbal, and controls the attitude of the camera 21 under the control of the computer 25. The camera attitude control device 22 also detects the attitude of the camera 21 and supplies attitude information representing the attitude to the computer 25.

[0033] The space recognition sensor 23 functions as a 3D structure recognition unit that detects (recognizes) (understands) the 3D (spatial) structure around the mobile body 20, such as the shapes of various objects present in the space around the mobile body 20. The space recognition sensor 23 supplies structural information representing the 3D structure around the mobile body 20 to the computer 25. The space recognition sensor 23 can be configured with sensors such as LiDAR (light detection and ranging), ToF (time of flight) sensors, multi-lens cameras, and event cameras (event-based cameras). The space recognition sensor 23 can also be configured with an object detection sensor implemented by a monocular camera and image processing such as SfM (structure from motion) using images captured by the monocular camera. The space recognition sensor 23 can be configured with one sensor from the LiDAR, ToF sensor, multi-lens camera, event camera, and object detection sensor, or a combination of multiple sensors.

[0034] The flight controller 24 controls the attitude of the moving body 20 under the control of the computer 25. The flight controller 24 also detects the position and attitude of the moving body 20, and supplies the computer 25 with position and attitude information representing the position and attitude.

[0035] The computer 25 is a computer called a companion computer or an internal computer, and performs various processes including control of each block using information from each block such as the camera 21 to the flight controller 24 that constitute the mobile object 20. The computer 25 also has the function of exchanging information with the transmitter 31 via wireless communication.

[0036] <Example of functional configuration of mobile system 10>

[0037] FIG. 4 is a block diagram showing an example of the functional configuration of the mobile system 10.

[0038] 4, the mobile body system 10 includes a transmitter 31 and an indicator 32, as well as a position information acquisition unit 41, a position information storage unit 42, a trajectory information acquisition unit 43, a target area identification unit 44, a movement control unit 45, a presentation control unit 46, and a status notification unit 47. Some or all of the blocks from the position information acquisition unit 41 to the status notification unit 47 can be provided in the mobile body 20. When some of the blocks from the position information acquisition unit 41 to the status notification unit 47 are provided in the mobile body 20, the remaining blocks can be provided in a single device separate from the mobile body 20, such as the transmitter 31, a server that manages the entire mobile body system 10, or a mobile terminal capable of communicating with the transmitter 31, such as a tablet that controls software updates for the transmitter 31, or the like, or can be provided in multiple, distributed locations.

[0039] The position information acquisition unit 41 sequentially (in time series) acquires position information of the designated points on the structure and supplies it to the position information storage unit 42 .

[0040] The position information storage unit 42 stores the position information of the indicated point supplied from the position information acquisition unit 41. The position information storage unit 42 has a storage capacity capable of storing multiple pieces of position information, and after storing position information until the storage capacity is reached, stores the newest position information by overwriting the oldest position information. The position information storage unit 42 can be configured, for example, with a volatile memory such as a dynamic random access memory (DRAM), or with a non-volatile memory such as a flash memory.

[0041] The trajectory information acquisition unit 43 acquires trajectory information representing the trajectory of the designated point (curve designation information representing the curve as a trajectory) from the position information of the designated point stored in the position information storage unit 42, and supplies the acquired trajectory information to the target area identification unit 44. For example, the trajectory information acquisition unit 43 can acquire, as trajectory information, time-series data of the position information of the designated point for a predetermined period of time, such as a predetermined number of seconds in the past. For example, the trajectory information acquisition unit 43 can acquire, as trajectory information, time-series data of the position information of the designated point for the most recent predetermined number of seconds. Furthermore, for example, the trajectory information acquisition unit 43 can acquire, as trajectory information, time-series data of the position information of the designated point from when the user performs an operation representing the start of designation of the target area to when the user performs an operation representing the end of designation of the target area.

[0042] The target area identification unit 44 identifies, as a target area, an area on the structure specified by the time-series data of the position information of the indication point, which is the trajectory information from the trajectory information acquisition unit 43. The target area identification unit 44 supplies area information representing the target area to the movement control unit 45 and the presentation control unit 46.

[0043] The movement control unit 45 has a path generation unit 45A and a control unit 45B, and performs movement control to move the moving body 20 to the target area represented by the area information from the target area identification unit 44. The path generation unit 45A generates a movement path (flight path) for moving the moving body 20 to the target area so that the target area can be photographed for inspection or the like, and supplies path information representing this movement path to the control unit 45B. The control unit 45B controls a drive mechanism (not shown) of the moving body 20 so that the moving body 20 moves according to the movement path represented by the path information from the path generation unit 45A (for example, generates a control signal for controlling the movement mechanism).

[0044] The movement control unit 45 can generate, as a movement route for moving the mobile body 20 to the target area, a route on an absolute coordinate system that defines absolute coordinates corresponding to latitude, longitude, and altitude, for example. Furthermore, as a movement route, it can generate, as an example, a route on a relative coordinate system that is based on (the current position of) the mobile body 20. When a path on the absolute coordinate system is generated as a movement route, it may not be possible to properly move the mobile body 20 to the target area in places where a global navigation satellite system (GNSS) cannot be used or where the accuracy of the GNSS is reduced. On the other hand, when a path on the relative coordinate system is generated as a movement route, it is possible to properly move the mobile body 20 to the target area even in places where the GNSS cannot be used or where the accuracy of the GNSS is reduced.

[0045] The presentation control unit 46 performs presentation control to present to the user the target area represented by the area information from the target area identification unit 44. For example, the presentation control unit 46 transmits the area information to the transmitter 31 via wireless communication to display the area information. The transmitter 31 has, for example, a touch panel 31A that serves as both a display unit and an operation unit, and displays the target area represented by the area information from the presentation control unit 46 on the touch panel 31A.

[0046] The status notification unit 47 collects the status of the mobile body 20 and notifies the user. For example, the status notification unit 47 collects the status of a drone body as the mobile body 20 and transmits status information representing that status to the transmitter 31 via wireless communication. The transmitter 31 displays the status of the mobile body 20 represented by the status information from the status notification unit 47 on the touch panel 31A.

[0047] The area information and status information can be transmitted to a device other than the transmitter 31, for example, a mobile terminal such as a tablet that can communicate with the transmitter 31, and displayed thereon.

[0048] The location information acquisition unit 41 to the presentation control unit 46 can be provided in the moving object 20, the transmitter 31, or a mobile terminal that can communicate with the transmitter 31. The location information acquisition unit 41 to the presentation control unit 46 can also be provided in two or more of the moving object 20, the transmitter 31, and a mobile terminal that can communicate with the transmitter 31.

[0049] FIG. 5 is a flowchart illustrating the processing of the mobile system 10 of FIG.

[0050] In step S11, the position information acquisition unit 41 sequentially acquires the position information of the designated points on the structure, supplies it to the position information storage unit 42 for storage, and the process proceeds to step S12.

[0051] In step S12, the trajectory information acquisition unit 43 acquires, for example, time-series data of the position information of the indicated point for the most recent predetermined number of seconds as trajectory information of the trajectory drawn by the indicated point from the position information of the indicated point stored in the position information storage unit 42. The trajectory information acquisition unit 43 supplies the trajectory information to the target area identification unit 44, and the process proceeds from step S12 to step S13.

[0052] In step S13, the target area identification unit 44 identifies, as a target area, an area on the structure specified by the trajectory information (time-series data of position information of the indicated point) from the trajectory information acquisition unit 43. The target area identification unit 44 supplies area information representing the target area to the movement control unit 45 and the presentation control unit 46, and the process proceeds from step S13 to step S14.

[0053] In step S14, the movement control unit 45 performs movement control to move the drone as the moving body 20 to the target area represented by the area information from the target area identification unit 44. For example, the movement control unit 45 generates a movement route for moving the moving body 20 to the target area so that the target area can be photographed for inspection or the like, and moves the moving body 20 according to the route information representing the movement route.

[0054] The presentation control unit 46 displays on the touch panel 31A the target area represented by the area information from the target area identification unit 44. By looking at the target area displayed on the touch panel 31A, the user can confirm whether the area they want to inspect (desired area) is the target area.

[0055] Furthermore, the status notification unit 47 collects the status of the moving object 20 and appropriately displays it on the touch panel 31 A. The user can check the status of the moving object 20 by looking at the status of the moving object 20 displayed on the touch panel 31 A.

[0056] <First Functional Configuration Example of Computer 25>

[0057] FIG. 6 is a block diagram showing a first example of the functional configuration of the computer 25 included in the mobile object 20. As shown in FIG.

[0058] In the figure, parts corresponding to those in FIG. 4 are given the same reference numerals, and the description thereof will be omitted below as appropriate.

[0059] 6, the computer 25 has the location information acquisition unit 41 to the status notification unit 47 in FIG. 4 (functions as the location information acquisition unit 41 to the status notification unit 47). Furthermore, in FIG. 6, the location information acquisition unit 41 has an image recognition unit 51, and the target area identification unit 44 has a trajectory conversion unit 61 and an area identification unit 62.

[0060] Images captured by the camera 21 at predetermined time intervals are supplied to the image recognition unit 51. For example, the user operates the transmitter 31 so that the camera 21 can capture an image of a structure, and as a result, the camera 21 supplies the captured image showing the structure to the image recognition unit 51.

[0061] The image recognition unit 51 recognizes the indicated point on the captured image by performing image recognition using the captured image from the camera 21. Furthermore, the image recognition unit 51 calculates and acquires first position information, which is position information indicating the position of the indicated point on the captured image, and supplies the first position information to the position information storage unit 42. The image recognition unit 51 acquires the first position information of the indicated point for each of the captured images in time series from the camera 21, and the first position information of the indicated point is thereby stored in time series in the position information storage unit 42.

[0062] 6, the first position information of the indicated point is stored in the position information storage unit 42. Therefore, the trajectory information of the trajectory of the indicated point acquired by the trajectory information acquisition unit 43 is time-series data of the first position information of the indicated point for the most recent predetermined number of seconds stored in the position information storage unit 42, i.e., time-series data of the position of the indicated point on the captured image.

[0063] The trajectory conversion unit 61 is supplied with trajectory information from the trajectory information acquisition unit 43, and is also supplied with attitude information representing the attitude of the camera 21 from the camera attitude control device 22. Furthermore, the trajectory conversion unit 61 is supplied with structure information representing the 3D structure around the moving body 20 from the space recognition sensor 23.

[0064] The trajectory conversion unit 61 converts the trajectory information from the trajectory information acquisition unit 43 using the attitude information of the camera 21 from the camera attitude control device 22 and the structure information from the spatial recognition sensor 23. The trajectory information from the trajectory information acquisition unit 43 is time-series data of first position information, which is position information representing the position of the pointer on the captured image. The trajectory conversion unit 61 converts the first position information representing the (two-dimensional) position of the pointer on the captured image into second position information representing the (three-dimensional) position of the pointer on the structure, using the attitude of the camera 21 represented by the attitude information of the camera 21 and the 3D structure of the structure represented by the structure information. The trajectory conversion unit 61 supplies the time-series data of the second position information to the area identification unit 62 as converted trajectory information.

[0065] The area specifying unit 62 is supplied with trajectory information from the trajectory conversion unit 61 , and also with position and orientation information representing the position and orientation of the moving body 20 from the flight controller 24 .

[0066] The area identification unit 62 identifies, as a target area, an area on the structure specified by the time-series data of the second position information of the indication point, which is the trajectory information from the trajectory conversion unit 61. The area identification unit 62 generates area information (specified area information on the 3D structure) representing the target area (area) in a coordinate system based on the moving body 20, using the position and attitude information of the moving body 20 from the flight controller 24, and supplies the area information to the movement control unit 45 and the presentation control unit 46.

[0067] As described above, in FIG. 6, the point indicated by the laser pointer serving as the indicator 32 is image-recognized, and the target area is identified based on the time series of the point indicated.

[0068] In FIG. 6 , the indicator 32 ( FIG. 1 ) may be a laser pointer or, for example, a thin, rod-shaped member (rod member) whose tip is processed into a specific shape or pattern. In this case, the user can designate the point of contact of the tip of the rod member on a structure as a designated point. The image recognition unit 51 acquires the position of the tip of the specific shape or pattern on the captured image as (first) position information of the designated point. Note that, while it is preferable to designate the designated point by bringing the tip of the rod member into contact with the structure, the tip of the rod member may be some distance away from the structure, such as several tens of centimeters. Even if the tip of the rod member is away from the structure, the target area identification unit 44 assumes, for example, that the tip of the rod member is in contact with the structure. The first position information representing the position of the designated point on the captured image is converted into second position information representing the position of the designated point on the structure, and the target area on the structure is identified based on the time-series data of the second position information. In this case, the target area will deviate from the area desired by the user. However, if the distance between the tip of the rod member and the structure is sufficiently small compared to the size of the target area, the deviation of the target area will be within an allowable range (error) for photography for inspection, etc.

[0069] <Second Functional Configuration Example of Computer 25>

[0070] FIG. 7 is a block diagram showing a second example of the functional configuration of the computer 25 included in the mobile object 20. As shown in FIG.

[0071] In the figure, parts corresponding to those in FIG. 4 are given the same reference numerals, and the description thereof will be omitted below as appropriate.

[0072] 7, the transmitter 31 has the position information acquisition unit 41 of the position information acquisition unit 41 to the status notification unit 47 in Fig. 4, and the computer 25 has the position information storage unit 42 to the status notification unit 47 of the position information acquisition unit 41 to the status notification unit 47 in Fig. 4. Furthermore, in Fig. 7, the transmitter 31 has a distance measurement unit 71. The position information acquisition unit 41 has a position and orientation recognition unit 81 and a position calculation unit 82, and the target area identification unit 44 has an area identification unit 91.

[0073] 7, the indicator 32 (FIG. 1) is a laser pointer with a distance measurement function attached to the transmitter 31, and the distance measurement function is controlled by a distance measurement unit 71. The distance measurement unit 71 measures the distance to an object, such as a structure, based on the laser light emitted by the laser pointer serving as the indicator 32 and the light reflected from the object.

[0074] The user designates a point on the structure as a pointer using the laser light emitted by the pointer 32. The user moves the pointer by operating the pointer 32, and designates a desired area on the structure as a target area using the moving pointer (time-series data).

[0075] The distance measurement unit 71 measures the distance to the designated point on the structure and supplies the result to the position calculation unit 82 .

[0076] The position and orientation recognition unit 81 recognizes the position and orientation of the transmitter 31 and supplies position and orientation information representing the position and orientation to the position calculation unit 82 .

[0077] The position calculation unit 82 calculates position information (second position information) representing the position of the indicated point on the structure using the distance from the distance measurement unit 71 to the indicated point on the structure and the position and orientation information of the transmitter 31 from the position and orientation recognition unit 81, and supplies it to the position information storage unit 42.

[0078] The area specifying unit 91 is supplied with trajectory information from the trajectory information acquiring unit 43 and also with position and orientation information of the moving body 20 from the flight controller 24 .

[0079] The area identification unit 91 identifies, as a target area, an area on the structure specified by the time-series data, based on time-series data of position information indicating the position on the structure of the indication point, which is the trajectory information from the trajectory information acquisition unit 43. The area identification unit 91 generates area information (specified area information on the 3D structure) indicating the target area (area) in a coordinate system based on the moving body 20, using the position and orientation information of the moving body 20 from the flight controller 24, and supplies this to the movement control unit 45 and the presentation control unit 46.

[0080] As described above, in Fig. 7, the position (information) of the indication point on the structure indicated by the laser pointer as the indicator 32 is calculated using the distance to the indication point obtained by distance measurement, and the target area is identified based on the time series of the indication point (the position of the indication point). In Fig. 7, the target area can be identified without image recognition of the indication point, so the user can specify the target area even from a distant position in the captured image where the indication point is less than one pixel or about several pixels in size.

[0081] FIG. 8 is a diagram illustrating an example of handling of position information for controlling the movement of the moving body 20. In FIG.

[0082] The relative coordinate system based on the transmitter 31 (or its center (center of gravity) etc.) at time t=t#i (i = 1, 2, ...) is also referred to as the transmitter-referenced coordinate system or xyz coordinate system.

[0083] The direction vector, which is the unit vector of the ranging direction (the direction from the transmitter 31 to the pointer p#i) of the distance to the pointer p#i in the transmitter-referenced coordinate system at time t=t#i, is expressed as vector (x#i, y#i, z#i). In this case, the coordinates (x, y, z coordinates) of the pointer p#i in the transmitter-referenced coordinate system at time t=t#i can be expressed as (r*x#i, r*y#i, r*z#i) using the distance r from the transmitter 31 to the pointer p#i.

[0084] An absolute coordinate system (a Cartesian coordinate system into which latitude, longitude, and height are converted) that defines absolute coordinates corresponding to latitude, longitude, and height is also called a global coordinate system or an XYZ coordinate system. The coordinates (X#i, Y#i, Z#i) of the indication point p#i in the global coordinate system at time t = t#i can be calculated by converting the xyz coordinates (r*x#i, r*y#i, r*z#i) of the indication point p#i in the transmitter-referenced coordinate system at time t = t#i based on the position and attitude of the transmitter-referenced coordinate system in the global coordinate system. For example, the XYZ coordinates (X#i, Y#i, Z#i) can be calculated by translating the xyz coordinates (r*x#i, r*y#i, r*z#i) according to the XYZ coordinates (Xs, Ys, Zs) of the transmitter 31 (center) (origin of the transmitter-referenced coordinate system) in the global coordinate system, and by rotating the coordinates according to the inclination of the xyz axes of the transmitter-referenced coordinate system relative to the xyz axes of the global coordinate system.

[0085] The position calculation unit 82 calculates a direction vector (x#i, y#i, z#i) based on the position and orientation information of the transmitter 31 from the position and orientation recognition unit 81, i.e., the position and orientation in the transmitter-referenced coordinate system. Furthermore, the position calculation unit 82 calculates x, y, and z coordinates (r*x#i, r*y#i, r*z#i) of the designated point p#i in the transmitter-referenced coordinate system at each time t#i using the direction vector (x#i, y#i, z#i) and the distance r from the distance measurement unit 71 to the designated point p#i at each time t#i. Then, based on the position and orientation in the transmitter-referenced coordinate system, the position calculation unit 82 converts the x, y, and z coordinates (r*x#i, r*y#i, r*z#i) into X, Y, and Z coordinates (X#i, Y#i, Z#i) of the designated point p#i in the global coordinate system at each time t#i. The XYZ coordinates (X#i, Y#i, Z#i) are supplied from the position calculation unit 82 to the position information storage unit 42 as position information representing the position of the indicated point p#i on the structure, and are stored therein.

[0086] The trajectory information acquisition unit 43 acquires, for example, time series data of the position information of the indication points p#1, p#2, ... for the most recent specified number of seconds from the position information of the indication points stored in the position information storage unit 42, as trajectory information of the trajectory drawn by the indication points, and supplies this to the area identification unit 91 of the target area identification unit 44.

[0087] Based on the trajectory information (time-series data of position information of indicated points) from the trajectory information acquisition unit 43, the area identification unit 91 identifies an area on the structure specified by the trajectory information as a target area.

[0088] The relative coordinate system based on the moving body 20 when the target area is identified is also referred to as a moving body reference coordinate system or an x'y'z' coordinate system. The area identification unit 91 generates area information representing the target area in the moving body reference coordinate system based on the position and attitude information of the moving body 20 from the flight controller 24, i.e., the position and attitude of the moving body reference coordinate system.

[0089] For example, the area enclosed by lines connecting the designated points p#1, p#2, ... for the most recent predetermined number of seconds can be identified as the target area. In this case, the coordinates (x'y'z' coordinates) (series) of the designated points p#1, p#2, ... for the most recent predetermined number of seconds in the mobile-object reference coordinate system can be generated as area information. The x'y'z' coordinates of the designated point p#i for the most recent predetermined number of seconds in the mobile-object reference coordinate system can be calculated by converting the XYZ coordinates of the designated point p#i in the global coordinate system based on the position and orientation of the mobile-object reference coordinate system in the global coordinate system. For example, the x'y'z' coordinates of the designated point p#i can be calculated by translating the XYZ coordinates of the designated point p#i according to the XYZ coordinates (Xd, Yd, Zd) of the origin of the mobile-object reference coordinate system in the global coordinate system and by rotating the XYZ coordinates of the designated point p#i according to the inclination of the x'y'z' axes of the mobile-object reference coordinate system relative to the XYZ axes of the global coordinate system.

[0090] The movement control unit 45 generates a movement path on the moving body reference coordinate system that moves the moving body 20 to the target area, which is identified by the x'y'z' coordinates of the indication points p#1, p#2, ... for the most recent specified number of seconds in the moving body reference coordinate system as area information, so that the target area can be photographed for inspection, etc.

[0091] <Identifying the target area>

[0092] FIG. 9 is a diagram showing an example of target area identification based on trajectory information by the target area identification unit 44. In FIG.

[0093] 9A is a diagram showing a first example of specifying a target area. For example, if the time-series data of the position information of the indicated point as the trajectory information acquired by the trajectory information acquisition unit 43 describes a single line enclosing a closed area, the target area identification unit 44 can identify the closed area as the target area.

[0094] 9B is a diagram showing a second example of specifying a target area. For example, if the time-series data of the position information of the indicated point as the trajectory information acquired by the trajectory information acquisition unit 43 describes a single line that encloses a part of a closed area, the target area identification unit 44 can identify, as the target area, the closed area enclosed by the single line and a line connecting the start point and end point of the single line.

[0095] 9C shows a third example of specifying a target area. For example, if the time-series data of the position information of the indicated points as trajectory information acquired by the trajectory information acquisition unit 43 depicts multiple lines enclosing a part of a closed area, the target area identification unit 44 can identify, as the target area, the closed area enclosed by the multiple lines and the lines connecting the end points of each of the multiple lines to the start point of the next line. For a line L among multiple lines, the next line is the line that is closest to line L in terms of distance or time among the lines that have not yet been designated as the next line.

[0096] FIG. 10 is a diagram showing another example of target area identification based on trajectory information by the target area identification unit 44. In FIG.

[0097] 10A is a diagram showing a fourth example of specifying a target area. For example, if the time-series data of the position information of the indicated point as trajectory information acquired by the trajectory information acquisition unit 43 depicts a plurality of lines that surround three or more small areas, the target area identification unit 44 can identify an area having each of the plurality of vertices as a target area.

[0098] 10B is a diagram showing a fifth example of specifying a target area. For example, if the time-series data of the position information of the indicated point as trajectory information acquired by the trajectory information acquisition unit 43 depicts one or more lines that fill in an area, the target area identification unit 44 can identify the area that is filled in by the one or more lines as the target area.

[0099] 10C is a diagram showing a sixth example of specifying a target area. For example, if the time-series data of the position information of the indicated point as the trajectory information acquired by the trajectory information acquisition unit 43 describes a line that goes around a closed area multiple times, the target area identification unit 44 can identify, as the target area, a closed area surrounded by a line that is an average of the multiple lines that go around a closed area.

[0100] <Other Functional Configuration Examples of the Mobile System 10>

[0101] FIG. 11 is a block diagram showing another example of the functional configuration of the mobile system 10.

[0102] In the figure, parts corresponding to those in FIG. 4 are given the same reference numerals, and the description thereof will be omitted below as appropriate.

[0103] In FIG. 11, the mobile system 10 includes a transmitter 31 and an indicator 32 as well as a position information acquisition unit 41 to a status notification unit 47 and an edit information reception unit 111 .

[0104] Therefore, the mobile system 10 in Fig. 11 is common to the case in Fig. 4 in that it includes a transmitter 31, an indicator 32, and components from a position information acquisition unit 41 to a status notification unit 47. However, the mobile system 10 in Fig. 11 differs from the case in Fig. 4 in that an editing information acceptance unit 111 is newly provided.

[0105] The editing information receiving unit 111 receives editing information from the transmitter 31 (or a mobile terminal such as a tablet that can communicate with the transmitter 31). The editing information receiving unit 111 controls the trajectory information acquisition unit 43 and / or the target area identification unit 44 in accordance with the editing information, thereby editing the trajectory represented by the trajectory information acquired by the trajectory information acquisition unit 43 and / or the target area identified by the target area identification unit 44. When the trajectory represented by the trajectory information or the target area is edited, the trajectory information acquisition unit 43 acquires the trajectory information again, and the target area is identified again by the target area identification unit 44.

[0106] Here, as described above, the presentation control unit 46 can present the target area represented by the area information to the user by displaying it on the touch panel 31A of the transmitter 31. Furthermore, the presentation control unit 46 can present to the user the trajectory of the indication point represented by the trajectory information used to identify the target area (the trajectory drawn by the time-series data of the position information of the indication point) by displaying it on the touch panel 31A of the transmitter 31 instead of or together with the target area.

[0107] When the user looks at the target area and the trajectory of the indication point displayed on the touch panel 31A, if the target area does not match the desired area, the user can edit the target area and / or the trajectory of the indication point displayed on the touch panel 31A by operating the touch panel 31A so that the target area matches the desired area.

[0108] The editing operations performed on the touch panel 31A to edit the target area or the locus of the pointer can include, for example, an UNDO operation to undo the most recent editing operation, and a REDO operation to undo the undoing of the most recent editing operation. Further, the editing operations can include deleting or changing a line of the locus of the pointer, moving the target area, etc.

[0109] For example, in an operation to delete some of the lines of the locus of the indicated points, one or more of the lines constituting the locus of the indicated points can be deleted. The locus information acquisition unit 43 reacquires locus information representing the lines remaining after the deletion, and supplies the information to the target area identification unit 44. The target area identification unit 44 re-identifies the target area based on the locus information re-acquired by the locus information acquisition unit 43.

[0110] For example, in an operation to change a part of the line of the locus of the designated points, the line constituting the locus of the designated points is displayed as a sequence of points, and some of the points in the sequence of points can be deleted or moved. The trajectory information acquisition unit 43 reacquires trajectory information for the sequence of points remaining after the deletion or the sequence of points after the movement as new designated points, and supplies the trajectory information to the target area identification unit 44. The target area identification unit 44 re-specifies the target area based on the trajectory information re-acquired by the trajectory information acquisition unit 43. Furthermore, for example, in an operation to change a part of the line of the locus of the designated points, the line constituting the locus of the designated points can be approximated by one or more predetermined curves, such as Bezier curves, and some of the curves can be deleted or moved. The trajectory information acquisition unit 43 reacquires trajectory information representing the curves remaining after the deletion or the curves after the movement, and supplies the trajectory information to the target area identification unit 44. The target area identification unit 44 re-specifies the target area based on the trajectory information re-acquired by the trajectory information acquisition unit 43.

[0111] For example, in the operation of moving the target area, the target area can be translated, and the target area specifying unit 44 re-specifies the target area after the translational movement.

[0112] As described above, in the mobile body system 10 of Fig. 11, the target area and / or the locus of the indicated point are edited in response to the user's operation (editing operation) of the touch panel 31A. Therefore, even if the user cannot operate the indicator 32 so that the target area coincides with the desired area, the user can edit the target area or the locus of the indicated point by operating the touch panel 31A to make the target area coincide with the desired area.

[0113] <Description of a computer to which this technology is applied>

[0114] Next, the above-described series of processes can be performed by hardware or software. When the series of processes is performed by software, the programs that make up the software are installed on a general-purpose computer or the like.

[0115] 12 is a block diagram showing an example of the configuration of an embodiment of a computer in which a program for executing the above-described series of processes is installed. For example, FIG. 12 shows an example of the hardware configuration of a computer 25.

[0116] The program can be recorded in advance on the hard disk 905 or ROM 903 as a recording medium built into the computer.

[0117] Alternatively, the program can be stored (recorded) on a removable recording medium 911 driven by the drive 909. Such a removable recording medium 911 can be provided as a so-called package software. Here, examples of the removable recording medium 911 include a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, and a semiconductor memory.

[0118] The program can be installed into the computer from the removable recording medium 911 as described above, or can be downloaded to the computer via a communication network or a broadcasting network and installed on the built-in hard disk 905. That is, the program can be transferred to the computer wirelessly from a download site via an artificial satellite for digital satellite broadcasting, or transferred to the computer via a wired network such as a LAN (Local Area Network) or the Internet.

[0119] The computer includes a CPU (Central Processing Unit) 902 , to which an input / output interface 910 is connected via a bus 901 .

[0120] When a user inputs a command via an input / output interface 910 by operating an input unit 907, the CPU 902 executes a program stored in a read-only memory (ROM) 903 in accordance with the command. Alternatively, the CPU 902 loads a program stored on a hard disk 905 into a random access memory (RAM) 904 and executes the program.

[0121] As a result, the CPU 902 performs processing according to the flowchart described above or processing performed by the configuration of the block diagram described above. Then, the CPU 902 outputs the processing results from the output unit 906 via the input / output interface 910, or transmits them from the communication unit 908, or further records them on the hard disk 905, as necessary.

[0122] The input unit 907 is made up of a keyboard, a mouse, a microphone, etc. The output unit 906 is made up of an LCD (Liquid Crystal Display), a speaker, etc.

[0123] In this specification, the processing performed by a computer according to a program does not necessarily have to be performed in chronological order according to the order described in the flowchart. In other words, the processing performed by a computer according to a program also includes processing that is executed in parallel or individually (for example, parallel processing or object-based processing).

[0124] The program may be processed by a single computer (processor), or may be distributed among multiple computers. Furthermore, the program may be transferred to and executed on a remote computer.

[0125] Furthermore, in this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0126] It should be noted that the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the present technology.

[0127] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0128] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0129] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0130] Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0131] The present technology can have the following configurations.

[0132] <1> A control method comprising: identifying a target area on a structure to be photographed based on time-series data of position information of a pointer, which is a point on the structure indicated by a pointer capable of indicating a point on the structure; and performing movement control to move a mobile object to the target area. <2> The control method described in <1>, further comprising: capturing an image of the structure; recognizing the pointer on the image and acquiring first position information representing the position of the pointer on the image; recognizing a 3D (dimensional) structure of the structure; and converting, using the 3D structure of the structure, the time-series data of the first position information into time-series data of second position information representing the position of the pointer on the structure, wherein the target area is identified based on the time-series data of the second position information. <3> The control method described in <1>, further comprising: measuring a distance to the pointer; and using the distance to calculate the position information representing the position of the pointer on the structure. <4> The control method according to any one of <1> to <3>, wherein, when the time-series data of the position information of the pointer draws a single line enclosing a closed region, the closed region is identified as the target region. <5> The control method according to any one of <1> to <3>, wherein, when the time-series data of the position information of the pointer draws a single line enclosing a part of the closed region, the closed region enclosed by the single line and a line connecting the start point and end point of the single line is identified as the target region. <6> The control method according to any one of <1> to <3>, wherein, when the time-series data of the position information of the pointer draws a plurality of lines enclosing a part of the closed region, the closed region enclosed by the plurality of lines and a line connecting the end points of each of the plurality of lines to the start point of the next line is identified as the target region. <7> The control method according to any one of <1> to <3>, wherein, when the time-series data of the position information of the pointer draws a plurality of lines enclosing a plurality of small regions, a region having each of the plurality of vertices as a vertex is identified as the target region.<8> The control method according to any one of <1> to <3>, wherein, when the time-series data of the position information of the pointer draws one or more lines that fill in an area, the area filled by the one or more lines is identified as the target area. <9> The control method according to any one of <1> to <3>, wherein, when the time-series data of the position information of the pointer draws lines that make multiple circuits around a closed area, the closed area enclosed by a line that averages the multiple circuits into one circuit is identified as the target area. <10> The control method according to any one of <1> to <9>, further comprising presenting a trajectory drawn by the time-series data of the position information of the pointer and / or the target area. <11> The control method according to <10>, further comprising editing the trajectory drawn by the time-series data of the position information of the pointer and / or the target area in response to a user operation. <12> The control method according to <11>, wherein editing of the trajectory drawn by the time-series data of the position information of the indicated point and / or the target area includes one or more of UNDO, REDO, deleting a part of the line of the trajectory, changing a part of the line of the trajectory, and moving the target area. <13> The control method according to any of <1> to <12>, wherein the indicator is a laser pointer, and the indicated point is a bright spot formed on the structure by irradiating the structure with a laser beam from the laser pointer. <14> The control method according to any of <1> to <13>, wherein the moving object is a drone. <15> The control method described in <1>, further comprising: the indicator is a laser pointer with a distance measurement function attached to a transmitter that remotely controls the moving body; the indication point is a bright point formed on the structure when laser light from the laser pointer is irradiated onto the structure; and calculating position information representing the position of the indication point on the structure using the distance to the indication point measured by the distance measurement function; and specifying the target area based on time series data of the position information.<16> A program causing a computer to execute a process including: specifying a target area on a structure to be photographed, based on time series data of position information of a pointer, which is a point on the structure indicated by a pointer capable of indicating a point on the structure, and performing movement control to move a mobile body to the target area. <17> A mobile body system including: a pointer capable of indicating a point on the structure, a camera that takes an image of the structure, an image recognition unit that recognizes the pointer, which is a point on the structure indicated by the pointer, on the image and acquires first position information that represents the position of the pointer on the image, a 3D structure recognition unit that recognizes a 3D (dimensional) structure of the structure, a conversion unit that converts time series data of the first position information into time series data of second position information that represents the position of the pointer on the structure, using the 3D structure of the structure, an identification unit that specifies a target area on the structure to be photographed, based on the time series data of the second position information, and a movement control unit that performs movement control to move a mobile body to the target area. <18> A mobile body system including: an indicator capable of indicating a point on a structure, a distance measuring unit that measures a distance to an indication point that is a point on the structure indicated by the indicator, a position calculation unit that calculates position information that represents the position of the indication point on the structure using the distance, an identification unit that identifies a target area on the structure to be photographed based on time series data of the position information, and a movement control unit that performs movement control to move a mobile body to the target area. <19> A mobile body system including: an indicator capable of indicating a point on a structure, an identification unit that identifies a target area on the structure to be photographed based on time series data of position information of the indication point that is the point on the structure indicated by the indicator, and a movement control unit that performs movement control to move a mobile body to the target area.

[0133] 10 Mobile body system, 20 Mobile body, 21 Camera, 22 Camera attitude control device, 23 Spatial recognition sensor, 24 Flight controller, 31 Transmitter, 31A Touch panel, 32 Indicator, 41 Position information acquisition unit, 42 Position information storage unit, 43 Trajectory information acquisition unit, 44 Target area identification unit, 45 Movement control unit, 45A Path generation unit, 45B Control unit, 46 Presentation control unit, 47 Status notification unit, 51 Image recognition unit, 61 Trajectory conversion unit, 62 Area identification unit, 81 Position and attitude recognition unit, 82 Position calculation unit, 91 Area identification unit, 111 Editing information acceptance unit, 901 Bus, 902 CPU, 903 ROM, 904 RAM, 905 Hard disk, 906 Output unit, 907 Input unit, 908 communication unit, 909 drive, 910 input / output interface, 911 removable recording medium

Claims

1. A control method comprising: identifying a target area on a structure to be photographed based on time-series data of position information of an indicated point, which is a point on the structure indicated by an indicator capable of indicating a point on the structure; and performing movement control to move a moving object to the target area.

2. The control method according to claim 1, further comprising: capturing an image showing the structure; recognizing the indication point on the image and acquiring first position information representing the position of the indication point on the image; recognizing a 3D (dimensional) structure of the structure; and converting, using the 3D structure of the structure, time series data of the first position information into time series data of second position information representing the position of the indication point on the structure; and identifying the target area based on the time series data of the second position information.

3. The control method according to claim 1, further comprising: measuring a distance to the indication point; and using the distance, calculating the position information representing the position of the indication point on the structure.

4. The control method according to claim 1, wherein when the time series data of the position information of the indication point draws a line enclosing a closed area, the closed area is identified as the target area.

5. The control method described in claim 1, wherein, when the time series data of the position information of the indicated point draws a line that encloses a part of a closed area, the closed area enclosed by the line and a line connecting the start point and end point of the line is identified as the target area.

6. The control method according to claim 1, wherein, when the time series data of the position information of the indicated point draws multiple lines that enclose a part of a closed area, the closed area enclosed by the multiple lines and the line connecting the end point of each of the multiple lines to the start point of the next line is identified as the target area.

7. The control method according to claim 1, wherein, when the time series data of the position information of the indicated point draws a plurality of lines each enclosing a plurality of small areas, an area having each of the plurality of areas as a vertex is identified as the target area.

8. The control method according to claim 1, wherein, when the time series data of the position information of the indication point draws one or more lines that fill an area, the area that is filled by the one or more lines is identified as the target area.

9. The control method according to claim 1, wherein, when the time series data of the position information of the indicated point draws a line that goes around a closed area multiple times, the closed area surrounded by a line that averages the multiple lines to one circumference is identified as the target area.

10. The control method according to claim 1, further comprising presenting a trajectory drawn by time-series data of position information of the indication point and / or the target area.

11. The control method according to claim 10, further comprising editing the trajectory drawn by the time-series data of the position information of the indicated point and / or the target area in response to a user operation.

12. The control method according to claim 11, wherein editing of the trajectory drawn by the time-series data of the position information of the indication point and / or the target area includes one or more of UNDO, REDO, deleting a part of the line of the trajectory, changing a part of the line of the trajectory, and moving the target area.

13. The control method according to claim 1, wherein the indicator is a laser pointer, and the indication point is a bright spot formed on the structure by irradiating the structure with laser light from the laser pointer.

14. The control method according to claim 1, wherein the moving object is a drone.

15. The control method described in claim 1 further includes: the indicator is a laser pointer with a distance measurement function attached to a transmitter that remotely controls the mobile body; the indication point is a bright spot formed on the structure when the laser light of the laser pointer is irradiated onto the structure; and calculating position information representing the position of the indication point on the structure using the distance to the indication point measured by the distance measurement function; and identifying the target area based on time series data of the position information.

16. A program for causing a computer to execute a process including: identifying a target area on a structure to be photographed based on time-series data of positional information of an indicated point, which is a point on the structure indicated by an indicator capable of indicating a point on the structure; and performing movement control to move a moving object to the target area.

17. A mobile body system including: an indicator capable of indicating a point on a structure; a camera that captures an image of the structure; an image recognition unit that recognizes an indication point on the image, which is a point on the structure indicated by the indicator, and acquires first position information that represents the position of the indication point on the image; a 3D structure recognition unit that recognizes the 3D (dimensional) structure of the structure; a conversion unit that uses the 3D structure of the structure to convert time series data of the first position information into time series data of second position information that represents the position of the indication point on the structure; an identification unit that identifies a target area on the structure to be photographed based on the time series data of the second position information; and a movement control unit that performs movement control to move a mobile body to the target area.

18. A mobile body system including: an indicator capable of indicating a point on a structure; a distance measurement unit that measures the distance to an indicated point, which is a point on the structure indicated by the indicator; a position calculation unit that uses the distance to calculate position information representing the position of the indicated point on the structure; an identification unit that identifies a target area on the structure to be photographed based on time series data of the position information; and a movement control unit that performs movement control to move a mobile body to the target area.

19. A mobile body system including: an indicator capable of indicating a point on a structure; an identification unit that identifies a target area on the structure to be photographed based on time series data of position information of an indication point, which is a point on the structure indicated by the indicator; and a movement control unit that performs movement control to move a mobile body to the target area.

Images